Information extraction method and device, storage medium and electronic equipment

Abstract

The invention discloses an information extraction method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining a to-be-extracted picture; determining the category of the to-be-extracted picture; determining an information extraction strategy corresponding to the category; and performing information extraction on the to-be-extracted picture according to the information extraction strategy. According to the method and the device, the flexibility of extracting the information of the picture is better.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and in particular, relates to an information extraction method, device, computer-readable storage medium and electronic device. Background Art

[0002] With the development of electronic devices such as smartphones and tablets, people are using electronic devices more and more frequently. When users use electronic devices, they usually need to extract corresponding information from a picture, such as text information. For example, when using an instant messaging application installed on an electronic device for communication, you want to transmit text information in a picture; or when using a text editing application installed on an electronic device for writing and other article creation work, such as news writing, you need to use text information in a picture taken on site.

[0003] In the related art, when electronic devices extract corresponding information from pictures, they usually extract information from different pictures according to a fixed information extraction strategy, and the flexibility of extracting information from pictures is relatively poor. Summary of the invention

[0004] The embodiments of the present application provide an information extraction method, device, storage medium and electronic device, which can achieve better flexibility in extracting information from images.

[0005] In a first aspect, an embodiment of the present application provides an information extraction method, comprising:

[0006] Get the image to be extracted;

[0007] Determine the category of the picture to be extracted;

[0008] determining an information extraction strategy corresponding to the category;

[0009] Information is extracted from the image to be extracted according to the information extraction strategy.

[0010] In a second aspect, an embodiment of the present application provides an information extraction device, including:

[0011] Image acquisition module, used to obtain the image to be extracted;

[0012] A category determination module, used to determine the category of the image to be extracted;

[0013] A strategy determination module, used to determine the information extraction strategy corresponding to the category;

[0014] The information extraction module is used to extract information from the image to be extracted according to the information extraction strategy.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the information extraction method provided by the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute the information extraction method provided in the embodiment of the present application by calling a computer program stored in the memory.

[0017] In an embodiment of the present application, by obtaining a picture to be extracted and determining the category of the picture to be extracted; determining an information extraction strategy corresponding to the category; and extracting information from the picture to be extracted according to the information extraction strategy. Since pictures of different categories include different content formats, different categories of pictures are suitable for different information extraction strategies. Therefore, the present application extracts information from pictures of different categories by setting corresponding information extraction strategies for pictures of different categories. Compared with the scheme in the related art that uses a fixed information extraction strategy to extract information from all categories of pictures, the present application has better flexibility in extracting information from pictures. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0019] Figure 1 It is a flowchart of the information extraction method provided in the embodiment of the present application.

[0020] Figure 2 It is a schematic diagram of a picture of category C1 provided in an embodiment of the present application.

[0021] Figure 3 It is a schematic diagram of a picture of category C2 provided in an embodiment of the present application.

[0022] Figure 4 It is a schematic diagram of a picture of category C3 provided in an embodiment of the present application.

[0023] Figure 5 It is a schematic diagram of the layout area and text area in the image M6 to be extracted provided in an embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of the text area in the layout area P11 provided in an embodiment of the present application.

[0025] Figure 7 This is a first schematic diagram of the corresponding text in the output layout area P11 provided in an embodiment of the present application.

[0026] Figure 8This is a second schematic diagram of the corresponding text in the output layout area P11 provided in an embodiment of the present application.

[0027] Fig. 9 It is a schematic diagram of the text layout format of the layout area P12 in the image to be extracted provided in an embodiment of the present application.

[0028] Fig.10 It is a schematic diagram of the corresponding text in the output layout area P12 provided in an embodiment of the present application.

[0029] Fig.11 It is a schematic diagram of the image M7 to be extracted provided in an embodiment of the present application.

[0030] Fig.12 It is a schematic diagram of text in the display layout areas P21 and P22 of the interface S1 provided in an embodiment of the present application.

[0031] Fig.13 It is a schematic diagram of the image M8 to be extracted provided in an embodiment of the present application.

[0032] Fig.14 The embodiment of this application provides Fig.13 The diagram shows the text in the layout areas P42 and P44 and the picture in the sub-picture area A after modification.

[0033] Fig.15 It is a schematic diagram of the image M9 to be extracted provided in an embodiment of the present application.

[0034] Fig.16 It is a schematic diagram of the layout area in the image M10 to be extracted provided in an embodiment of the present application.

[0035] Fig.17 It is a schematic diagram of the combined layout area in the image M10 to be extracted provided in an embodiment of the present application.

[0036] Fig.18 It is a schematic diagram of layout area labels corresponding to sample images of category C2 provided in an embodiment of the present application.

[0037] Fig.19 It is a schematic diagram of layout area labels corresponding to sample images of category C1 provided in an embodiment of the present application.

[0038] Fig. 20 It is a structural diagram of the information extraction device provided in an embodiment of the present application.

[0039] Fig.21 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] It should be noted that the terms "first", "second" and "third" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products or devices.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] The embodiments of the present application provide an information extraction method, an information extraction device, a storage medium, and an electronic device. The execution subject of the information extraction method may be the information extraction device provided in the embodiments of the present application, or an electronic device integrating the information extraction device, wherein the information extraction device may be implemented in hardware or software. The electronic device may be a device such as a smart phone, a tablet computer, a PDA, a laptop computer, etc. that is equipped with a processor and has information extraction capabilities.

[0043] See also Figure 1 , Figure 1 : is a flow chart of the information extraction method provided in the embodiment of the present application, and the flow chart may include:

[0044] In 101, a picture to be extracted is obtained.

[0045] The picture to be extracted may be any picture stored in the electronic device, may be a picture taken by the electronic device, or may be a picture acquired by the electronic device from other electronic devices.

[0046] In 102, the category of the picture to be extracted is determined.

[0047] For example, the categories of pictures can be divided based on the layout of the information in the pictures, such as classifying pictures with the same or similar layouts into the same category of pictures.

[0048] For example, pictures of articles and information can be determined as the same category, such as pictures of category C1, pictures of block cards can be determined as the same category, such as pictures of category C2, and pictures other than pictures of categories C1 and C2 can be determined as the same category, such as pictures of category C3. Among them, pictures of category C1 can include only text or a combination of text and pictures. Pictures of category C2 are modular pictures composed of text titles, subtitles, graphics or pictures, and are pictures with integrated content in a regular block form.

[0049] For example, Figure 2 The pictures M1 and M2 shown are pictures of category C1. Figure 3 The pictures M3 and M4 shown are pictures of category C2. Figure 4 The picture M5 shown is a picture of category C3.

[0050] Before obtaining the image to be extracted, the categories of multiple sample images can be labeled based on the above classification rules to obtain the category label corresponding to each sample image; then, the classification model is trained according to the multiple sample images and the category label corresponding to each sample image to obtain the trained classification model for subsequent determination of the category of the image to be extracted.

[0051] In an optional embodiment, when the category of the picture to be extracted cannot be determined, a plurality of optional categories may be provided for the user to select, so that the user can select the category of the picture to be extracted.

[0052] In an optional embodiment, a plurality of optional categories may be directly provided for the user to select, so that the user can select the category of the picture to be extracted.

[0053] In 103, an information extraction strategy corresponding to the category is determined.

[0054] For example, different information extraction strategies may be pre-set for different categories. For example, category C1 may correspond to information extraction strategy S1, category C2 may correspond to information extraction strategy S2, and category C3 may correspond to information extraction strategy S3.

[0055] In an optional embodiment, the information extraction strategies corresponding to some categories may be the same. For example, categories C1 and C2 correspond to information extraction strategy S1, and category C3 corresponds to information extraction strategy S3.

[0056] In this embodiment, after determining the category of the image to be extracted, the electronic device may determine the information extraction strategy corresponding to the category.

[0057] For example, assuming that the category of the image to be extracted is category C1, the information extraction strategy corresponding to the category can be determined to be information extraction strategy S1; assuming that the category of the image to be extracted is category C3, the information extraction strategy corresponding to the category can be determined to be information extraction strategy S3.

[0058] In 104, information extraction is performed on the image to be extracted according to the information extraction strategy.

[0059] In this embodiment, after the information extraction strategy corresponding to the category of the image to be extracted is determined, the electronic device can extract information from the image to be extracted according to the information extraction strategy.

[0060] For example, assuming that the information extraction strategy is information extraction strategy S1, information extraction strategy S1 is: divide the image to be extracted into at least one layout area, and extract the corresponding text in the layout area, then the electronic device can perform layout analysis on the image to be extracted through a pre-trained layout analysis model to divide the image to be extracted into at least one layout area, and then perform text recognition on the corresponding layout area, such as one of the at least one layout area, to obtain the corresponding text in the corresponding layout area, so as to realize the extraction of the corresponding text in the layout area.

[0061] It should be noted that the corresponding layout area can be one of the at least one layout area, a part of the at least one layout area, or all of the at least one layout area. The corresponding layout area can be specified by the user or determined by the electronic device based on certain rules.

[0062] For another example, assuming that the information extraction strategy is information extraction strategy S3, information extraction strategy S3 is: extract the corresponding text in the picture to be extracted, then the electronic device can perform text recognition on the picture to be extracted to obtain the corresponding text in the picture to be extracted, so as to realize the extraction of the text in the picture to be extracted.

[0063] In the present embodiment, by obtaining a picture to be extracted and determining the category of the picture to be extracted; determining an information extraction strategy corresponding to the category; and extracting information from the picture to be extracted according to the information extraction strategy. Since pictures of different categories include different content formats, different categories of pictures are suitable for different information extraction strategies. Therefore, the present application extracts information from pictures of different categories by formulating corresponding information extraction strategies for pictures of different categories. Compared with the scheme in the related art that uses a fixed information extraction strategy to extract information from all categories of pictures, the present application has better flexibility in extracting information from pictures.

[0064] Furthermore, since the information in pictures of the same category is relatively similar, and the information in pictures of different categories is relatively different, setting different information extraction strategies for pictures of different categories can improve the efficiency of extracting information from pictures of different categories.

[0065] In an optional embodiment, determining an information extraction strategy corresponding to a category includes:

[0066] If the category is the target category, the information extraction strategy is determined as follows: extracting the corresponding text in the image to be extracted;

[0067] If the category is not the target category, the information extraction strategy is determined as follows: dividing the image to be extracted into at least one layout area, and extracting the corresponding text in the layout area.

[0068] For example, considering that pictures belonging to category C3 usually do not have a fixed layout, it is not very meaningful to perform layout analysis on them, while pictures belonging to categories C1 and C2 usually have a fixed layout, and it is more meaningful to perform layout analysis on them. Therefore, in this embodiment, category C3 can be determined as the target category, and categories C1 and C2 can be determined as non-target categories. Then, when the category of the picture to be extracted is category C3, the information extraction strategy can be determined as: extracting the corresponding text in the picture to be extracted; when the category of the picture to be extracted is category C1 or C2, the information extraction strategy can be determined as: dividing the picture to be extracted into at least one layout area, and extracting the corresponding text in the layout area.

[0069] In an optional embodiment, information extraction is performed on the image to be extracted according to the information extraction strategy, including:

[0070] If the category is not the target category, then obtain the layout analysis model corresponding to the category;

[0071] Performing layout analysis on the image to be extracted by using a layout analysis model to divide the image to be extracted into at least one layout area;

[0072] Extract the corresponding text in the layout area.

[0073] Since pictures of different categories have different layouts, if the same layout analysis model is used to perform layout analysis on pictures of different categories, the accuracy of the layout analysis will be low. Based on this, in this embodiment, layout analysis models corresponding to different categories are trained for pictures of different categories. The layout analysis of each category is performed using the layout analysis model corresponding to each category to obtain at least one layout area in each category of pictures.

[0074] The layout area is a certain area in the image. When extracting text based on the layout area in the image, when the user only needs to obtain the corresponding layout area, such as the corresponding text in a certain layout area or a part of the layout area, only the text in the corresponding layout area can be provided to the user.

[0075] In an optional embodiment, extracting corresponding text in the layout area includes:

[0076] Perform text recognition on the image to be extracted to obtain at least one text and the text area where each text is located in the image to be extracted;

[0077] A target text area in the layout area is determined from the text area, and characters in the text corresponding to the target text area are determined as corresponding characters in the layout area.

[0078] When using a corresponding text recognition algorithm to perform text recognition on the image to be extracted, the coordinates of at least one text and the text box corresponding to each text in the pixel coordinate system established based on the image to be extracted are usually obtained, such as the first coordinate of the upper left corner of the text box in the pixel coordinate system and the second coordinate of the lower right corner of the text box in the pixel coordinate system. When the layout analysis model is used to perform layout analysis on the image to be extracted, the layout analysis model will also output the third coordinate of the upper left corner of each layout area in the pixel coordinate system and the fourth coordinate of the lower right corner in the pixel coordinate system.

[0079] Based on the first coordinate and the second coordinate corresponding to each text box, the text area of ​​each text box in the picture to be extracted can be determined. Based on the third coordinate and the fourth coordinate corresponding to each layout area, the area of ​​each layout area in the picture to be extracted can be determined, so that the text area in each layout area can be further determined.

[0080] For example, the positions of the text areas T111, T112, T113, T121, T122, and the layout areas P11, P12 in the image to be extracted M6 can be as follows: Figure 5 As shown, it can be determined that text areas T111, T112 and T113 are in layout area P11, and text areas T121 and T122 are in layout area P12. Assuming that the corresponding text in layout area P11 needs to be extracted, text areas T111, T112 and T113 can be determined as target text areas. Assuming that the text in text areas T111, T112 and T113 is as follows Figure 6 As shown, the corresponding text in the layout area P11 can be determined as "We are Chinese, I love my motherland, and my motherland is my home."

[0081] In an optional embodiment, the target text area in the layout area can also be determined by calculating the intersection and union ratio of the text area and the layout area. If the intersection and union ratio of a certain layout area and a certain text area is greater than or equal to the intersection and union ratio threshold, it is determined that the text area is in the layout area. The intersection and union ratio threshold can be set by the user or by the electronic device based on certain rules.

[0082] For example, assuming that the corresponding text in all layout areas needs to be extracted, when the intersection-and-union ratio of a certain text area with any layout area is less than the intersection-and-union ratio threshold, the text area is considered to be redundant information elimination. If the intersection-and-union ratio of a certain text area with only a certain layout area is greater than or equal to the intersection-and-union ratio threshold, then the text area is determined to be the target text area in the layout area. If the intersection-and-union ratios of a certain text area with multiple layout areas are greater than or equal to the intersection-and-union ratio threshold, then the corresponding layout area with the largest intersection-and-union ratio is determined from the multiple layout areas, and the text area is determined as the target text area in the layout area with the largest corresponding intersection-and-union ratio.

[0083] In an optional embodiment, after determining the characters in the text corresponding to the target text area as the corresponding characters in the layout area, the method further includes:

[0084] If there are multiple target text regions, the corresponding text in the layout region is output according to the relative position of each target text region and other target text regions in the layout region.

[0085] The relative positions of the characters in each target text area outputted correspond to the relative positions of each target text area. Figure 6 As shown, assuming that the corresponding layout area is layout area P11, and the target text areas are text areas T111, T112, and T113, it can be determined that the target text areas T111 are adjacent to T112, and the target text area T111 is on the left side of the target text area T112, and the target text area T113 is below the target text areas T111 and T112, and is directly below the target text area T111. Then, the following can be used to determine the target text area T111: Figure 7 The text in the text areas T111, T112 and T113 is output in the form shown, that is, the corresponding text in the layout area P11 is output. That is to say, in the output text, the text in the target text area T111 is adjacent to the text in T112, and the text in the target text area T111 is on the left side of the text in the target text area T112, while the text in the target text area T113 is below the text in the target text areas T111 and T112, and directly below the text in the target text area T111.

[0086] The corresponding text in each layout area can be output in the form of editable text, such as output to a word document or output to a memo; an editable interface can also be provided, and the text in each layout area can be output to the editable interface, so that the user can modify, copy, paste and other editing operations on the corresponding text in the layout area P11. The corresponding text in each layout area can also be output to an instant messaging application and sent to the corresponding chat partner at the user's choice. As for whether to output as a word document, output to a memo, output to an editable interface, or output to an instant messaging application, it can be specified by the user.

[0087] In an optional embodiment, outputting corresponding text in the layout area according to the relative position of each target text area and other target text areas in the layout area includes:

[0088] According to the relative position of each target text area and other target text areas in the layout area, the corresponding text in the layout area is output according to the target text typesetting format corresponding to the layout area.

[0089] The text layout format may include the font size, the font shape, the spacing between two adjacent words, the spacing between two lines of text in a paragraph, the spacing between two paragraphs, and so on.

[0090] For example, a corresponding text typesetting format can be set in advance and used as the target text typesetting format corresponding to a certain layout area. Then, based on the position of each target text area and other target text areas in the layout area, the corresponding text in the layout area can be output according to the target text typesetting format corresponding to the layout area.

[0091] For example, Figure 6 As shown, assuming that a certain layout area is layout area P11, the target text area is text areas T111, T112 and T113, the characters in text areas T111, T112 and T113 belong to the same paragraph, and the target text typesetting format is that the spacing between two lines of text in a paragraph is spacing d1, and the spacing between two adjacent characters is spacing d2, then the corresponding characters in the output layout area P11 can be as follows Figure 8 shown.

[0092] In an optional embodiment, according to the relative position of each target text area and other target text areas in the layout area, according to the target text typesetting format corresponding to the layout area, before outputting the corresponding text in the layout area, the method further includes:

[0093] The text layout format of the layout area in the image to be extracted is determined as the target text layout format corresponding to the layout area.

[0094] Considering that the text layout format desired by the user may be the layout format of the text in the image to be extracted, the text layout format of the layout area in the image to be extracted can be determined as the target text layout format corresponding to the layout area.

[0095] For example, suppose the text layout format of the layout area P12 in the image to be extracted is as follows: Fig. 9 As shown in FIG. 1 , if the spacing between two lines of text in a paragraph is spacing d3, and the spacing between two adjacent texts is spacing d4, then the text in the layout area P12 can be output as follows: Fig.10 shown.

[0096] In an optional embodiment, according to the relative position of each target text area and other target text areas in the layout area, and in accordance with the target text typesetting format corresponding to the layout area, outputting the text in the layout area includes:

[0097] If there are multiple layout areas, the text in multiple layout areas is output on the same interface according to the relative position of each target text area and other target text areas in the corresponding layout area, according to the target text typesetting format corresponding to the layout area and the typesetting formats of multiple layout areas.

[0098] For example, based on Fig.11 The picture M7 to be extracted is shown in the same interface, such as the text in the layout area P21 and P22 displayed in the interface S1 can be as follows Fig.12 The text displayed in the interface S1 can be edited, such as modified, copied, and pasted.

[0099] In an optional embodiment, if Fig.13 As shown, if the image to be extracted M8 contains layout areas P41, P42, P43, P44 and sub-image area A, it can be displayed in the corresponding editable interface Fig.13 The corresponding texts in the layout areas P41, P42, P43, and P44 and the pictures in the sub-picture area A are shown to restore the picture M8 to be extracted. Fig.14 As shown, the user can modify the text displayed on the editable interface, such as the editable interface S2, such as replacing all the text in the first area of ​​the editable interface S2, replacing part of the text in the second area of ​​the editable interface S2, and the user can also replace the picture displayed on the editable interface S2, wherein the first area and the Fig.13 The second area corresponds to the layout area P42 shown in FIG. Fig.13 The layout area P44 shown corresponds to this.

[0100] In an optional embodiment, extracting corresponding text in the layout area includes:

[0101] Segment the publication area from the image to be extracted;

[0102] Perform text recognition on the layout area to obtain the corresponding text in the layout area.

[0103] For example, in order to more conveniently obtain the text in a certain layout area, the layout area can be segmented from the image to be extracted, such as by cutting the layout area out from the image to be extracted, and then performing text recognition on the layout area to obtain at least one text corresponding to the layout area, and the text area where each text is located in the layout area, and the text in the at least one text corresponding to the layout area is determined as the corresponding text in the layout area.

[0104] It should be noted that, when there are multiple layout areas, the user can specify which layout area or which layout areas to cut to perform text recognition on each layout area.

[0105] In an optional embodiment, after obtaining at least one layout area, the corresponding layout area can be cut out based on the user's selection, and the corresponding cut layout area can be output, such as saving the cut layout area as a new picture in a picture library, or the cut layout area can be output as a new picture to a target application selected by the user for user use, such as outputting it to an instant messaging application and sending it to a corresponding chat object at the user's selection.

[0106] In an optional embodiment, after determining the characters in at least one text corresponding to the layout area as the corresponding characters in the layout area, the method further includes:

[0107] If there are multiple text areas, the corresponding text in the layout area is output according to the relative position of each text area and other text areas in the layout area.

[0108] It should be noted that the specific implementation of “if there are multiple text areas, then outputting the corresponding text in the layout area according to the relative positions of each text area and other text areas in the layout area” can be found in other embodiments and will not be repeated here.

[0109] In an optional embodiment, after performing layout analysis on the image to be extracted by using the layout analysis model to divide the image to be extracted into at least one layout area, the method further includes:

[0110] Determine a target layout area that meets a preset retention condition from at least one layout area;

[0111] Extract the corresponding text in the layout area, including:

[0112] Extract the corresponding text in the target layout area.

[0113] Taking into account that the layout analysis model will detect some layout areas that do not meet the preset retention conditions, and these layout areas often do not have any actual information or information that users need, in order to improve the speed of information extraction, these layout areas can be identified as layout areas that do not meet the preset retention conditions, and only the text in the target layout areas that meet the preset retention conditions can be extracted.

[0114] In an optional embodiment, determining a target layout area satisfying a preset retention condition from at least one layout area includes:

[0115] A layout area in which a geometric parameter of at least one layout area is greater than a parameter threshold is determined as a target layout area that meets a preset retention condition.

[0116] The geometric parameters include size, width, height, etc. The preset retention condition may be at least one of the following conditions: the size is greater than a size threshold, the width is greater than a width threshold, and the height is greater than a height threshold.

[0117] For example, layout areas whose sizes are less than or equal to a size threshold may be removed, and only text in layout areas whose sizes are greater than the size threshold may be extracted.

[0118] For example, the layout areas whose size is less than or equal to the size threshold and whose width is less than or equal to the width threshold can be removed, and only the text in the layout areas whose size is greater than the size threshold and whose width is greater than the width threshold can be extracted.

[0119] For example, you can remove the layout areas whose size is less than or equal to the size threshold, whose width is less than or equal to the width threshold, and whose height is less than or equal to the height threshold, and only extract the text in the layout areas whose size is greater than the size threshold, whose width is greater than the width threshold, and whose height is greater than the height threshold.

[0120] The size threshold, height threshold and width threshold may be set by a user or by the electronic device based on certain rules.

[0121] In an optional embodiment, determining a target layout area satisfying a preset retention condition from at least one layout area includes:

[0122] Determine the layout area with a larger confidence level corresponding to two layout areas intersecting at least one layout area as a candidate layout area;

[0123] A layout area that is not used to determine the candidate layout area and the candidate layout area in at least one layout area are determined as a target layout area that meets a preset retention condition.

[0124] For example, the layout areas with smaller confidences corresponding to two intersecting layout areas may be removed, and only the corresponding texts in the remaining layout areas may be extracted.

[0125] Among them, the confidence corresponding to each layout area can be output by the layout detection model.

[0126] In an optional embodiment, determining a target layout area satisfying a preset retention condition from at least one layout area includes:

[0127] Filter out at least one first candidate layout area whose geometric parameter is greater than a parameter threshold from at least one of the layout areas;

[0128] Determine a second candidate layout area from at least one of the first candidate layout areas, wherein the second candidate layout area is a first candidate layout area with a larger confidence value corresponding to two first candidate layout areas intersecting the at least one first candidate layout area;

[0129] Determine a third candidate layout area from at least one of the first candidate layout areas, wherein the third candidate layout area is a first candidate layout area in at least one of the first candidate layout areas that is not used to determine the second candidate layout area;

[0130] The second candidate layout area and the third candidate layout area are determined as the target layout areas that meet the preset retention conditions.

[0131] For example, the layout areas whose sizes are less than or equal to the size threshold can be removed to obtain at least one first candidate layout area, and then the first candidate layout areas with smaller confidences corresponding to the two intersecting first candidate layout areas can be removed, and only the text in the remaining candidate layout areas can be extracted.

[0132] For example, Fig.15 As shown, the image M9 to be extracted includes layout areas P31, P32, P33, P34, P35, P36, P37, P38, P39 and P40, wherein the sizes of layout areas P39 and P40 are smaller than the size threshold, the sizes of layout areas P31, P32, P33, P34, P35, P36, P37 and P38 are larger than the size threshold, only layout areas P31 and P38 intersect, and the confidence corresponding to layout area P31 is greater than the confidence corresponding to layout area P38, then layout areas P31, P32, P33, P34, P35, P36 and P37 can be determined as target layout areas that meet the preset retention conditions.

[0133] In an optional embodiment, after determining the target layout area satisfying the preset retention condition from at least one layout area, the method further includes:

[0134] Combine target layout areas that meet preset combination conditions to obtain combined layout areas;

[0135] The combined layout area and the target layout area that does not meet the preset combination conditions are determined as the area to be extracted;

[0136] Extract the corresponding text in the target layout area, including:

[0137] Extract the corresponding text in the area to be extracted.

[0138] For example, the target layout areas located in the same row or column may be determined as layout areas that meet preset combination conditions, and these target layout areas may be combined to obtain a combined layout area.

[0139] Among them, when the target layout area has the same upper left corner ordinate and the same lower right corner ordinate in the pixel coordinate system established based on the image to be extracted, it is determined that the target layout area is located in the same row. When the target layout area has the same upper left corner ordinate and the same lower right corner ordinate in the pixel coordinate system established based on the image to be extracted, it is determined that the target layout area is located in the same column.

[0140] For example, Fig.16 As shown, the image M10 to be extracted includes target layout areas P31, P32, P33, P34, P35, P36 and P37, wherein layout areas P31 and P32 are in the same row, target layout areas P33 and P34 are in the same row, and target layout areas P35 and P36 are in the same row. Fig.17 As shown, the target layout areas P31 and P32 can be combined to obtain the combined area P312, the target layout areas P33 and P34 can be combined to obtain the combined area P334, and the target layout areas P35 and P36 can be combined to obtain the combined area P356; then, the combined area P312, the combined area P334, the combined area P356 and the target layout area P37 can be determined as the areas to be extracted, and the text in the corresponding areas to be extracted can be extracted.

[0141] For another example, considering that there may be a situation where the layout analysis model identifies a layout area as two layout areas, and these layout areas are often located close to each other and have basically the same width or height, the target layout areas that are located close to each other and have basically the same width or height can be determined as layout areas that meet the preset combination conditions, and these target layout areas are combined to obtain the combined layout area.

[0142] Among them, when the difference between the coordinates of the lower left corner of a target layout area in the pixel coordinate system established based on the picture to be extracted and the coordinates of the upper left corner of another target layout area in the pixel coordinate system established based on the picture to be extracted is less than the coordinate difference threshold, the difference between the coordinates of the lower right corner of the target layout area in the pixel coordinate system established based on the picture to be extracted and the coordinates of the upper right corner of the other target layout area in the pixel coordinate system established based on the picture to be extracted is less than the coordinate difference threshold, and the difference between the width of the target layout area and the width of another target layout area is less than the width difference threshold, then the target layout area and the other target layout area are combined to obtain a combined layout area.

[0143] When the difference between the coordinates of the upper right corner of a target layout area in the pixel coordinate system established based on the image to be extracted and the coordinates of the upper left corner of another target layout area in the pixel coordinate system established based on the image to be extracted is less than the coordinate difference threshold, the difference between the coordinates of the lower right corner of the target layout area in the pixel coordinate system established based on the image to be extracted and the coordinates of the lower left corner of the other target layout area in the pixel coordinate system established based on the image to be extracted is less than the coordinate difference threshold, and the difference between the width of the target layout area and the height of the other target layout area is less than the height difference threshold, then the target layout area and the other target layout area are combined to obtain a combined layout area.

[0144] It should be noted that the specific implementation of "extracting the corresponding text in the area to be extracted" can be referred to other embodiments, which will not be repeated here.

[0145] In an optional embodiment, before obtaining the layout analysis model corresponding to the category, the method further includes:

[0146] Obtain multiple sample images and a layout area label corresponding to each sample image, and determine the category of each sample image;

[0147] The object detection model is trained according to sample images of the same category and their corresponding layout area labels to obtain a layout analysis model corresponding to each category.

[0148] It is understandable that the target detection model can be trained based on sample images of the same category and their corresponding layout area labels to obtain a layout analysis model corresponding to each category. After the category of the image to be extracted is determined, the layout analysis of the image to be extracted can be performed using the layout analysis model corresponding to the category.

[0149] Among them, the ways of obtaining multiple sample images include but are not limited to: downloading from the Internet, photographing through electronic devices such as smart phones or tablet computers, and obtaining by taking screenshots of corresponding interfaces.

[0150] When labeling the layout area of ​​sample images, the labeling can be performed based on actual needs, so as to obtain the layout area label corresponding to each sample image.

[0151] For example, Fig.18 The annotations of the sample images of the C2 category shown can be: for the blocks with obvious boundary lines as shown in the sample image M11, they can be annotated as a whole, so as to obtain the layout area labels P41, P42, P43 and P44 corresponding to the sample image M11. For the visually consistent modules as shown in the sample image M12, they can be annotated as a whole, so as to obtain the layout area labels P45 and P46 corresponding to the sample image M12. For the fixed layout information group as shown in the sample image M13, it can be annotated as a whole, so as to obtain the layout area labels P47, P48 and P49 corresponding to the sample image M13.

[0152] For example, Fig.19 The annotations of the sample images of the C1 category shown in the figure can be: for a pure text image such as the sample image M14, the title and a paragraph can be annotated as a whole, thereby obtaining the layout area labels P51, P52, P53 and P54 corresponding to the sample image M14. For a picture with a combination of text and images such as the sample image M15, a paragraph and a sub-image can be annotated as a whole, thereby obtaining the layout area labels P55 and P56 corresponding to the sample image M15.

[0153] After labeling other sample images in the same or corresponding manner as described above, the target detection model can be trained based on the sample images of each category and their corresponding layout area labels to obtain a layout analysis model corresponding to each category.

[0154] The training process and structure of the layout analysis model corresponding to each category are the same, and the only difference is that the categories of the sample images used for training are different.

[0155] For example, the target detection model E1 can be trained based on sample images of category C1 and their corresponding layout area labels to obtain a layout analysis model D1 corresponding to category C1. The target detection model E2 can be trained based on sample images of category C2 and their corresponding layout area labels to obtain a layout analysis model D2 corresponding to category C2. Subsequently, when it is determined that the category of the image to be extracted is category C2, the layout analysis model D1 can be used to perform layout analysis on the image to be extracted to divide the image to be extracted into at least one layout area; when it is determined that the category of the image to be extracted is category C2, the layout analysis model D2 can be used to perform layout analysis on the image to be extracted to divide the image to be extracted into at least one layout area.

[0156] See also Fig. 20 , Fig. 20 The information extraction device 200 includes: a picture acquisition module 201 , a category determination module 202 , a strategy determination module 203 and an information extraction module 204 .

[0157] The picture acquisition module 201 is used to acquire the picture to be extracted.

[0158] The category determination module 202 is used to determine the category of the image to be extracted.

[0159] The strategy determination module 203 is used to determine the information extraction strategy corresponding to the category.

[0160] The information extraction module 204 is used to extract information from the image to be extracted according to the information extraction strategy.

[0161] In an optional embodiment, the strategy determination module 203 can be used to: if the category is a target category, determine the information extraction strategy as: extract the corresponding text in the image to be extracted; if the category is not a target category, determine the information extraction strategy as: divide the image to be extracted into at least one layout area, and extract the corresponding text in the layout area.

[0162] In an optional embodiment, the information extraction module 203 can be used to: if the category is not the target category, obtain a layout analysis model corresponding to the category; perform layout analysis on the image to be extracted through the layout analysis model to divide the image to be extracted into at least one layout area; and extract the corresponding text in the layout area.

[0163] In an optional embodiment, the strategy determination module 203 can be used to: perform text recognition on the image to be extracted to obtain at least one text and a text area where each text is located in the image to be extracted; determine a target text area in the layout area from the text area, and determine the text in the text corresponding to the target text area as the corresponding text in the layout area.

[0164] In an optional embodiment, the information extraction device 200 may also include a text output module, which may be used to: if there are multiple target text areas, output the corresponding text in the layout area according to the relative position of each target text area and other target text areas in the layout area.

[0165] In an optional embodiment, the text output module can be used to: output the corresponding text in the layout area according to the relative position of each target text area and other target text areas in the layout area and according to the target text typesetting format corresponding to the layout area.

[0166] In an optional embodiment, the text output module may be used to: determine the text layout format of the layout area in the image to be extracted as the target text layout format corresponding to the layout area.

[0167] In an optional embodiment, the text output module can be used to: segment the layout area from the image to be extracted; perform text recognition on the layout area to obtain corresponding text in the layout area.

[0168] In an optional embodiment, the text output module can be used to: determine a target layout area that meets a preset retention condition from at least one of the layout areas; and extract corresponding text from the target layout area.

[0169] In an optional embodiment, the text output module can be used to: screen out at least one first candidate layout area whose geometric parameters are greater than a parameter threshold from at least one of the layout areas; determine a second candidate layout area from at least one of the first candidate layout areas, the second candidate layout area being a first candidate layout area with a larger confidence level corresponding to two intersecting first candidate layout areas in at least one of the first candidate layout areas; determine a third candidate layout area from at least one of the first candidate layout areas, the third candidate layout area being a first candidate layout area in at least one of the first candidate layout areas that is not used to determine the second candidate layout area; and determine the second candidate layout area and the third candidate layout area as the target layout areas that meet preset retention conditions.

[0170] In an optional embodiment, the text output module can be used to: combine the target layout areas that meet the preset combination conditions to obtain a combined layout area; determine the combined layout area and the target layout area that does not meet the preset combination conditions as areas to be extracted; and extract the corresponding text in the areas to be extracted.

[0171] In an optional embodiment, the strategy determination module 203 can be used to: obtain multiple sample images, and the layout area label corresponding to each of the sample images, and determine the category of each of the sample images; train the target detection model according to the sample images of the same category and their corresponding layout area labels to obtain a layout analysis model corresponding to each category.

[0172] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the information extraction method provided in the embodiment.

[0173] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the information extraction method provided in the present embodiment by calling a computer program stored in the memory.

[0174] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.21 , the electronic device 300 may include components such as a processor 301, a memory 302, etc. Those skilled in the art will appreciate that Fig.21 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the electronic device 300 may also include a screen.

[0175] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 302 and calling data stored in the memory 302, thereby monitoring the electronic device as a whole.

[0176] The memory 302 may be used to store applications and data. The applications stored in the memory 302 include executable codes. The applications may be composed of various functional modules. The processor 301 executes various functional applications and data processing by running the applications stored in the memory 302.

[0177] In this embodiment, the processor 301 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby achieving:

[0178] Get the image to be extracted;

[0179] Determine the category of the picture to be extracted;

[0180] determining an information extraction strategy corresponding to the category;

[0181] Information is extracted from the image to be extracted according to the information extraction strategy.

[0182] In an optional embodiment, when the processor 301 executes the information extraction strategy corresponding to the category, it can execute: if the category is a target category, then the information extraction strategy is determined to be: extract the corresponding text in the picture to be extracted; if the category is not a target category, then the information extraction strategy is determined to be: divide the picture to be extracted into at least one layout area, and extract the corresponding text in the layout area.

[0183] In an optional embodiment, when the processor 301 executes the information extraction for the image to be extracted according to the information extraction strategy, it can execute: if the category is not the target category, obtaining the layout analysis model corresponding to the category; performing layout analysis on the image to be extracted through the layout analysis model to divide the image to be extracted into at least one layout area; and extracting the corresponding text in the layout area.

[0184] In an optional embodiment, when the processor 301 executes the extraction of the corresponding text in the layout area, it can execute: performing text recognition on the image to be extracted to obtain at least one text and a text area where each text is located in the image to be extracted; determining a target text area in the layout area from the text area, and determining the text in the text corresponding to the target text area as the corresponding text in the layout area.

[0185] In an optional embodiment, after the processor 301 executes the step of determining the characters in the text corresponding to the target text area as the corresponding characters in the layout area, it may also execute: if there are multiple target text areas, then according to the relative positions of each target text area and other target text areas in the layout area, output the corresponding characters in the layout area.

[0186] In an optional embodiment, when the processor 301 executes the output of the corresponding text in the layout area according to the relative positions of each target text area and other target text areas in the layout area, it can execute: according to the relative positions of each target text area and other target text areas in the layout area, according to the target text typesetting format corresponding to the layout area, output the corresponding text in the layout area.

[0187] In an optional embodiment, the processor 301 may also execute the following steps before outputting the corresponding text in the layout area according to the target text layout format corresponding to the layout area based on the positions of each target text area and other target text areas in the layout area: determining the text layout format of the layout area in the image to be extracted as the target text layout format corresponding to the layout area.

[0188] In an optional embodiment, when the processor 301 executes the extraction of the corresponding text in the layout area, it can execute: segmenting the layout area from the image to be extracted; performing text recognition on the layout area to obtain the corresponding text in the layout area.

[0189] In an optional embodiment, after the processor 301 executes the layout analysis of the image to be extracted through the layout analysis model to divide the image to be extracted into at least one layout area, it can also execute: determining a target layout area that meets a preset retention condition from at least one of the layout areas; when the processor 301 executes the extraction of corresponding text in the layout area, it can execute: extracting the corresponding text in the target layout area.

[0190] In an optional embodiment, when the processor 301 executes the method of determining the target layout area that meets the preset retention condition from at least one of the layout areas, it can execute: screening out at least one first candidate layout area whose geometric parameters are greater than a parameter threshold from at least one of the layout areas; determining a second candidate layout area from at least one of the first candidate layout areas, the second candidate layout area being a first candidate layout area with a larger confidence level corresponding to two intersecting first candidate layout areas of at least one of the first candidate layout areas; determining a third candidate layout area from at least one of the first candidate layout areas, the third candidate layout area being a first candidate layout area in at least one of the first candidate layout areas that is not used to determine the second candidate layout area; and determining the second candidate layout area and the third candidate layout area as the target layout area domain that meets the preset retention condition.

[0191] In an optional embodiment, after the processor 301 executes the method of determining the target layout area that meets the preset retention condition from at least one of the layout areas, it can also execute: combining the target layout areas that meet the preset combination condition to obtain a combined layout area; determining the combined layout area and the target layout area that does not meet the preset combination condition as the area to be extracted; when the processor 301 executes the method of extracting the corresponding text in the target layout area, it can execute: extracting the corresponding text in the area to be extracted.

[0192] In an optional embodiment, before the processor 301 executes the acquisition of the layout analysis model corresponding to the category, it may also execute: acquiring multiple sample images and the layout area label corresponding to each of the sample images, and determining the category of each of the sample images; training the target detection model according to the sample images of the same category and their corresponding layout area labels to obtain the layout analysis model corresponding to each category.

[0193] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the information extraction method above, and will not be repeated here.

[0194] The information extraction device provided in the embodiment of the present application belongs to the same concept as the information extraction method in the above embodiment. Any method provided in the information extraction method embodiment can be run on the information extraction device. The specific implementation process is detailed in the information extraction method embodiment, which will not be repeated here.

[0195] It should be noted that, for the information extraction method of the embodiment of the present application, a person of ordinary skill in the art can understand that the whole or part of the process of implementing the information extraction method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the information extraction method. Among them, the computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0196] It is understandable that in the specific implementation of this application, user information is involved, such as application usage behavior data, logs and other related data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0197] For the information extraction device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0198] The above is a detailed introduction to an information extraction method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An information extraction method, characterized in that: include: Get the image to be extracted; Determine the category of the picture to be extracted; determining an information extraction strategy corresponding to the category; Information is extracted from the image to be extracted according to the information extraction strategy.

2. The information extraction method according to claim 1, characterized in that: The determining of the information extraction strategy corresponding to the category includes: If the category is the target category, the information extraction strategy is determined as: extracting the corresponding text in the image to be extracted; If the category is not the target category, the information extraction strategy is determined as: dividing the image to be extracted into at least one layout area, and extracting corresponding text in the layout area.

3. The information extraction method according to claim 2, characterized in that: The extracting information from the image to be extracted according to the information extraction strategy includes: If the category is not the target category, obtaining a layout analysis model corresponding to the category; Performing layout analysis on the image to be extracted by using the layout analysis model to divide the image to be extracted into at least one layout area; Extract the corresponding text in the layout area.

4. The information extraction method according to claim 3, characterized in that: The extracting corresponding text in the layout area includes: Performing text recognition on the image to be extracted to obtain at least one text and a text region where each text is located in the image to be extracted; A target text area in the layout area is determined from the text area, and characters in the text corresponding to the target text area are determined as corresponding characters in the layout area.

5. The information extraction method according to claim 4, characterized in that: After determining the characters in the text corresponding to the target text area as the corresponding characters in the layout area, the method further includes: If there are multiple target text areas, the corresponding text in the layout area is output according to the relative position of each target text area and other target text areas in the layout area.

6. The information extraction method according to claim 5, characterized in that: Outputting the corresponding text in the layout area according to the relative position of each target text area and other target text areas in the layout area includes: According to the relative position of each target text area and other target text areas in the layout area, the corresponding text in the layout area is output according to the target text typesetting format corresponding to the layout area.

7. The information extraction method according to claim 6, characterized in that: Before outputting the corresponding text in the layout area according to the position of each target text area and other target text areas in the layout area and the target text typesetting format corresponding to the layout area, the method further includes: The text layout format of the layout area in the image to be extracted is determined as the target text layout format corresponding to the layout area.

8. The information extraction method according to claim 3, characterized in that: The extracting corresponding text in the layout area includes: Segmenting the layout area from the picture to be extracted; Perform text recognition on the layout area to obtain corresponding text in the layout area.

9. The information extraction method according to claim 3, characterized in that: After performing layout analysis on the image to be extracted by using the layout analysis model to divide the image to be extracted into at least one layout area, the method further includes: Determining a target layout area that meets a preset retention condition from at least one of the layout areas; The extracting corresponding text in the layout area includes: Extract the corresponding text in the target layout area.

10. The information extraction method according to claim 9, characterized in that: The step of determining a target layout area satisfying a preset retention condition from at least one of the layout areas comprises: Filter out at least one first candidate layout area whose geometric parameter is greater than a parameter threshold from at least one of the layout areas; Determine a second candidate layout area from at least one of the first candidate layout areas, wherein the second candidate layout area is a first candidate layout area with a larger confidence value corresponding to two first candidate layout areas intersecting at least one of the first candidate layout areas; Determine a third candidate layout area from at least one of the first candidate layout areas, wherein the third candidate layout area is the first candidate layout area that is not used to determine the second candidate layout area in at least one of the first candidate layout areas; The second candidate layout area and the third candidate layout area are determined as the target layout areas that meet the preset retention conditions.

11. The information extraction method according to claim 9, characterized in that: After determining the target layout area satisfying the preset retention condition from at least one of the layout areas, the method further includes: Combining the target layout areas that meet the preset combination conditions to obtain a combined layout area; Determine the combined layout area and the target layout area that does not meet the preset combination condition as the area to be extracted; The extracting corresponding text in the target layout area includes: Extract the corresponding text in the area to be extracted.

12. The information extraction method according to claim 3, characterized in that: Before obtaining the layout analysis model corresponding to the category, the method further includes: Obtaining multiple sample images and a layout area label corresponding to each of the sample images, and determining a category of each of the sample images; The target detection model is trained according to the sample images of the same category and their corresponding layout area labels to obtain a layout analysis model corresponding to each category.

13. An information extraction device, characterized in that: include: Image acquisition module, used to obtain the image to be extracted; A category determination module, used to determine the category of the image to be extracted; A strategy determination module, used to determine the information extraction strategy corresponding to the category; The information extraction module is used to extract information from the image to be extracted according to the information extraction strategy.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the information extraction method according to any one of claims 1 to 12.

15. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the information extraction method according to any one of claims 1 to 12 by calling the computer program stored in the memory.

Images