Element extraction method, electronic equipment and storage medium

Abstract

The invention discloses an element extraction method, electronic equipment and a storage medium, belongs to the technical field of text recognition, and is used for improving the efficiency of text element extraction. The method comprises the following steps: identifying a to-be-identified target picture to obtain a cell text in cells of the target picture; the cell text is input into an element extraction model, target elements are extracted from the cell text, the element extraction model is trained through a neural network, and the element extraction model comprises layout information of the target elements in multiple templates.

Description

Technical Field

[0001] The present application belongs to the field of text recognition, and specifically relates to an element extraction method, an electronic device and a storage medium. Background Art

[0002] Extracting key elements from document images is a technical field that combines advanced technologies such as image processing, computer vision, and natural language processing (NLP). It aims to automatically identify and extract important information such as text, cells, charts, and images in images to facilitate the digitization, indexing, analysis, or archiving of information. It mainly involves technologies such as optical character recognition, layout analysis and structured information extraction, and cell recognition and processing.

[0003] When extracting key information, the commonly used extraction method relies on building specific recognition templates for different types. Each recognition type requires a separate standard image and corresponding text box definition, as well as the determination of the correspondence between element attributes and key values. This results in a lot of repetitive work when processing new types, resulting in low efficiency in feature extraction. Summary of the invention

[0004] The embodiments of the present application provide an element extraction method, an electronic device, and a storage medium, which can solve the problem that processing new types requires a lot of repetitive work, resulting in low element extraction efficiency.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting an element, the method comprising: identifying a target image to be identified, obtaining a cell text in a cell of the target image; inputting the cell text into an element extraction model, extracting a target element from the cell text, wherein the element extraction model is trained by a neural network, and the element extraction model includes layout information of target elements in multiple templates. In a second aspect, an embodiment of the present application provides an element extraction device, which includes: a processing module, used to identify a target image to be identified, and obtain cell text in the cells of the target image; an extraction module, used to input the cell text into an element extraction model, and extract target elements from the cell text, wherein the element extraction model is trained by a neural network, and the element extraction model includes layout information of target elements in multiple templates.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0008] In an embodiment of the present application, a target image to be identified is identified to obtain cell text in a cell of the target image; the cell text is input into a feature extraction model to extract target elements from the cell text, wherein the feature extraction model is trained by a neural network, and the feature extraction model includes layout information of target elements in multiple templates, and can extract target elements of texts corresponding to multiple templates, thereby improving feature extraction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application; Figure 2 It is a flowchart of an element extraction method provided in an embodiment of the present application; Figure 3 It is a flowchart of text and cell matching provided in an embodiment of the present application; Figure 4 This is a schematic diagram of correspondence between text and cells provided in an embodiment of the present application; Figure 5 It is a schematic diagram of a model application provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of a factor extraction device provided in an embodiment of the present application; Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0011] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0012] In the following, in combination with the accompanying drawings, a factor extraction method, an electronic device and a storage medium provided in an embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0013] The system architecture of the embodiment of the present application mainly involves two parts: the client and the server. Figure 1 As shown, the client includes a picture / image uploading module, a request encapsulation module and a result parsing module, and the server includes a picture correction module, a seal detection and removal module, a cell detection module, an OCR recognition module, a feature extraction module, a result error correction module and a return module.

[0014] Client: Image upload module: upload the image / picture to be identified. Request encapsulation module: the request encapsulation module encapsulates the image with base64 encoding and http request header. Result parsing module: the result parsing module receives the result returned by the server.

[0015] Server: Image correction module: parses the image passed in by the client, extracts the message body of the http request, decodes the base64 image and calls the correction model to straighten the image; Seal detection and removal module: calls the seal detection model to detect the number and position of seals, and then eliminates the impact of red seals; Cell detection module: calls the cell line detection model and algorithm to detect and reconstruct cells. OCR recognition module: calls the OCR model to identify the text content and text box position coordinates of the entire image, and then distinguishes the text of each cell according to the cell unit block. Key element extraction module based on VIE model: uses the improved GeoLayoutLM model trained on a private dataset to extract key elements. Result error correction module: retrieves and compares the feature extraction results with the database content to determine the correct result. Return module: returns the recognition result to the client.

[0016] Figure 2 An element extraction method provided by an embodiment of the present application is shown, and the method can be executed by a server. In other words, the method can be executed by software or hardware installed on the server, and the method includes the following steps: Step 201: Identify a target image to be identified, and obtain cell text in a cell of the target image.

[0017] In an embodiment of the present application, it is necessary to identify the target image to be identified uploaded by the client, and identify the cell text in the target image. In an embodiment of the present application, the target image can be a bill image, and of course it can also be other images. The target image is not specifically limited here.

[0018] In one implementation, before the target image to be identified is identified, it also includes: when the target image is offset, correcting the target image according to the offset angle of the target image; when it is detected that the target image includes an official seal, removing the official seal in the target image.

[0019] Specifically, sometimes there will be an offset in the target image, such as an offset of 90 degrees, 180 degrees, or 270 degrees. In order to reduce the complexity of subsequent text extraction and improve the accuracy of recognition, the target image needs to be rotated for recognition. In an embodiment of the present application, about 1,000 images are pre-labeled to train the scrfd model, and the offset angle of the target image is recognized by the scrfd model. According to the recognized offset angle a, the target image is rotated and corrected by 360-angle a using opencv.

[0020] After correcting the target image, the trained official seal detection model can be used to detect official seals on the image. If the target image includes official seals, the number of seals is identified and then the official seals in the target image are removed.

[0021] In one implementation, the step of identifying a target image to be identified and obtaining a cell text in a cell of the target image includes: The method of identifying a target image to be identified and obtaining cell text in a cell of the target image includes: obtaining pixel points belonging to a cell in the target image; determining the cell coordinates of each cell according to the pixel points; obtaining text in the target image; and determining the cell text corresponding to each cell according to the text and the cell coordinates.

[0022] In an embodiment of the present application, OCR recognition can be performed on the entire image to obtain all texts. Optical Character Recognition (OCR) is the core technology for extracting text from images. OCR technology realizes the conversion from image to text through a deep learning model, especially the combination of a convolutional neural network (CNN) and a recurrent neural network (RNN). In an embodiment of the present application, the line extraction model can be trained by pre-labeling about 1,000 data of settlement cell lines based on the Unet semantic segmentation algorithm using labelme. By processing the target image with the line extraction model, the pixel points of all horizontal and vertical lines belonging to the cells in the target image can be obtained. Then, the cell coordinates of each cell are determined based on the pixel points, and then the cell text corresponding to each cell is determined based on the text and the cell coordinates.

[0023] Step 204: Input the cell text into a feature extraction model to extract target features from the cell text.

[0024] Specifically, the feature extraction model can be a GeoLayoutLM model trained by a neural network, and the feature extraction model includes layout information of target features in multiple templates. In this way, the feature extraction model extracts features from cell texts of different layouts and different types according to the layout information of target features in multiple templates.

[0025] The element extraction method provided in the embodiment of the present application identifies the target image to be identified, obtains the cell text in the cell of the target image; inputs the cell text into the element extraction model, and extracts the target element from the cell text, wherein the element extraction model is trained by a neural network, and the element extraction model includes the layout information of the target elements in a plurality of templates, and can extract the target elements of the text corresponding to the plurality of templates, thereby improving the efficiency of element extraction, realizing the unified recognition of texts of different types and layouts, avoiding the construction of specific templates for each type of text, greatly simplifying the extraction process of key elements, and improving the extraction efficiency and accuracy. This general model based on deep learning can process bills of different formats without the need for a large number of customized templates, thereby improving the flexibility and scalability of the system.

[0026] In one implementation, determining the cell coordinates of each cell based on the pixel points includes: classifying the pixel points to determine the cell to which each pixel point belongs; and determining the cell coordinates of each cell by calculating the circumscribed matrix of the pixel points in the cell.

[0027] Specifically, after obtaining the pixel points of the cell, the pixel points can be classified, and then it can be calculated which cell each pixel point belongs to. By calculating the circumscribed matrix of the pixel points in the cell, the coordinates of the upper left corner and lower right corner of the rectangle are obtained, and the coordinates of the upper left corner and upper right corner of the cell are used as the starting coordinates of the cell line, and then the coordinates of each cell can be obtained. In this way, the coordinates of the cell are calculated by calculating the pixel points of the cell, which can improve the calculation efficiency. The applicability of cell bounding box selection, cell direction detection and perspective transformation is improved, and the accuracy and robustness of cell detection and image correction are improved.

[0028] In one implementation, determining the cell text corresponding to each cell based on the text and the cell coordinates includes: determining the cell text corresponding to each cell based on the center point of the text and the position of the text box of the text and the cell.

[0029] After obtaining the text and cell coordinates, it is necessary to confirm the text corresponding to each cell. The current method is to perform OCR recognition on each cell in the cell recognition, but when the number of cells is large, it will greatly increase the reasoning time of the model. Therefore, in order to improve the reasoning speed of the model, the cell text matching method provided in the embodiment of the present application is as follows: Figure 3 As shown, the target image is used to perform OCR recognition only once and the center point matching method is used to match the text box recognized by OCR with the cell. If there are multiple texts, they are first sorted up and down by position, and then sorted left and right to splice the content. Specifically, it is determined whether the center point of the cell text is in the cell. If it is, it is determined whether the edge of the cell text box exceeds the cell. If it does not exceed, it is sorted up and down by position, and then sorted left and right to splice the content. If it exceeds, ORC recognition is performed on the cell again, and then sorted up and down by position, and then sorted left and right to splice the content. In this way, OCR recognition is avoided for each cell, which can reduce the reasoning time and improve the matching efficiency of cell text and cell. It can effectively deal with scenes such as complex cell boundaries, overlapping cell lines and text, and abnormal internal layout of cells, greatly improving the robustness of cell detection.

[0030] In one implementation, the target element includes a first field and a second field corresponding to the first field, and before the cell text is input into the element extraction model, the method further includes: Preprocess the first training data to obtain second training data, wherein the second training data includes the first field, the second field, and a link relationship between the first field and the second field; obtain multiple templates that carry the target element; and train the element extraction model through the neural network, the multiple templates and the second training data.

[0031] Specifically, the application process of the feature extraction model is as follows: Figure 4 As shown, it includes data labeling, data preprocessing, model training, model reasoning and post-processing. In the embodiment of the present application, the first training data used to train the model is also a picture.

[0032] The target elements in the cell text usually include fields with corresponding relationships, such as the first field "Province:" and the second field "** Province". Since the FUNSD dataset required for model fine-tuning is relatively complex, X-AnyLabeling with built-in OCR function is usually used. After recognizing all the text on the image, compare the settlement form templates of each province and mark the first field as "answer", and its corresponding key, that is, the second field, as "question", for example Figure 5 As shown. In addition, if there are fields with separate lines, each line needs to be marked as "words", and then the text box containing all lines needs to be manually selected and marked as "answer" or "question". In order to keep the data consistent as much as possible during model training and inference, the data preprocessing part discards the manually annotated text results and only retains the link relationship between the first field and the second field, thus obtaining the second training data. At the same time, the cell text and text box coordinates in the image of the first training data are obtained, and the manually annotated categories and link relationships are filled in.

[0033] First, the data exported by the annotation software needs to be converted into the format of the FUNSD dataset. Each image corresponds to a json file, and each annotation field consists of id, text, box, linking, label, and words. Among them, id is an integer that increases from 0, text is the cell text content, box is the cell position, and consists of the coordinates of the upper left corner and the lower right corner, that is, [x0, y0, x1, y1]. Label can only be "question" / "answer" / "other", answer is the first field to be identified, question is the second field corresponding to the first field, and other is other irrelevant fields. For example, the label of "province" is question, and the label of "**province" is answer, and the fields in the cell are mostly two questions corresponding to one answer. Words means that if there is only one line in the field, the text and box in words are the same as the text and box above; if there are multiple lines, fill in the line unit, and box is still the upper left and lower right corner coordinates. Linking represents the binary id array of corresponding questions and answers, that is, if there are multiple questions corresponding to one answer, the linking of the answer should have multiple binary tuples. After the format conversion, the valid content of the annotation file is the "label" and "linking" fields, waiting for subsequent merging. After obtaining the cell text from the previous step, construct the annotation item based on the merged multi-line text, traverse the single-line text before the merge, and if the single-line text is part of a multi-line text, add it to the "words" of the annotation item, and finally determine the multi-line text box coordinates that surround all single-line texts in the cell based on the position of the text box in "words". Then construct the text outside the cell as an annotation item based on the single-line text. Find the annotation items that match the manually annotated data, fill in the corresponding categories and link relationships into the annotation items, and complete the merge of the manually annotated and OCR recognition data. Use a word segmenter to segment the text to obtain data that is truly suitable for model input.

[0034] In one implementation, the step of training the feature extraction model by using the neural network, the multiple templates and the second training data includes: Labeling the template data in each of the templates, the template data including the first field, the second field and the link relationship between the first field and the second field in the template; establishing a mapping relationship between the second training data and the template data; inputting the template data into a neural network, and acquiring the layout information of the target element through the neural network; training the element extraction model through the layout information of the target element and the second training data.

[0035] Specifically, when training the model, in addition to the training data constructed in the above steps, the data in the template library is also needed to assist in constructing the key field features of the template. Given the significant diversity of settlement form templates of operators in various regions, the same field may have very different expressions and layout positions in different templates, making automatic recognition and classification extremely complicated.

[0036] Directly using settlement form images containing various templates as training template data can easily confuse the model. To solve this problem, a graph neural network (GNN) is used here to capture and encode the key field layout information of these templates, thereby enhancing the model's understanding of template characteristics. GNN is a powerful deep learning model that excels at processing graph structured data. It learns the relationship between nodes through nodes and edges to extract information from the graph structure. In this way, it is equivalent to adding a "trigger" to the model to let it know what template layout should be used for the current image. The specific steps are as follows: 1. Key field annotation: For each settlement form template, annotate the template data, i.e. the first field "question" and the second field "answer", the text content of the field and the cell coordinates. This will provide template data for GNN to learn the key field layout patterns of different templates.

[0037] 2. Build a graph structure: Use the first and second fields as nodes and the key-value relationship as edges to build a directed graph. For example, use "province" and "**province" as nodes and connect them with edges to indicate that "province" corresponds to "**province".

[0038] 3. Template and training data mapping: Establish a one-to-one mapping relationship between template data and training data, that is, specify a corresponding template for each training image. This will ensure that GNN can learn the layout information of a specific template.

[0039] 4. GNN feature extraction: Input template data into GNN to learn the layout features of key fields. GNN learns the relationship between nodes through nodes and edges, thereby extracting information in the graph structure, such as the relative position relationship between the first field and the second field.

[0040] 5. Feature fusion: The layout information learned by GNN is combined with the text features extracted by the GeoLayoutLM model.

[0041] In the model inference stage, we first perform image correction and OCR recognition on the input image to obtain text content and cell location information. Then we construct FUNSD format data except for "label" and "linking" and perform the same word segmentation process. This information and the corresponding template information are input into the trained model to perform relationship link prediction.

[0042] The post-processing stage first parses the output of the feature extraction model, matches the identified "question" and "answer" according to the predicted link relationship, and ensures that each field has a corresponding answer. For many-to-one or one-to-many associations, the binary id array of the "linking" field is checked to correctly construct the association map between the question and the answer. Finally, the predicted "question" is classified and the corresponding "answer" is filled in the result to obtain the key element content. In one implementation, the target element is vectorized to obtain an element vector; the element vector is searched and compared with a reference vector in a knowledge base through a retrieval threshold to obtain a similarity between the element vector and the reference vector, wherein the retrieval threshold is determined based on the text complexity of the target element and the confidence output by the target element extraction model; the target element is corrected using the reference element corresponding to the reference vector with the highest similarity.

[0043] In terms of ensuring the accuracy of key element extraction, the embodiment of the present application introduces an adaptive threshold text error correction mechanism, which aims to optimize the OCR recognition results, especially the accuracy of core information such as provinces, cities, customer names and reasons. This mechanism not only improves the reliability of information extraction, but also achieves a dual improvement in error correction efficiency and accuracy by intelligently adapting to different text complexities and model trust.

[0044] Specifically, we first build a highly optimized knowledge base that contains key information items already in the system, such as geographical names of provinces, cities, and customer information. All of this information is converted through advanced text vectorization technology to form a dense and efficient search vector space. This conversion enables the model to quickly locate the most relevant and most likely correct text expression in the vast ocean of information.

[0045] During the retrieval stage, the field to be corrected (target element) is vectorized through the same model, and then searched and compared with the control vector in the database, and the new result is output after reordering. Traditional text correction methods often rely on fixed thresholds to determine whether to correct errors. The adaptive threshold algorithm of this application flexibly adjusts the retrieval threshold standard according to the complexity of the current text, such as text length, frequency of occurrence of special characters, complexity of grammatical structure and other factors, as well as the confidence of the output of the feature extraction model when processing the text. This means that when processing texts with simple structures and high model confidence, the error correction threshold can be set more strictly to ensure high accuracy; in the face of complex and changeable situations where the model judgment is uncertain, the threshold is appropriately relaxed to accommodate more possible correct options, thereby avoiding missing important information. The specific steps are: 1. Information vectorization: First, the target elements are vectorized through advanced vectorization technology and converted into high-density element vector representation. This step enables the system to quickly and accurately locate the most appropriate match in massive data.

[0046] 2. Similarity calculation and evaluation: Calculate the similarity between the generated feature vector and the reference vector of each entry in the pre-built knowledge base, which contains key entries such as geographical names such as provinces, cities, and customer information.

[0047] 3. Adaptive threshold screening: Different from the limitations of traditional fixed thresholds, this embodiment adopts an adaptive threshold strategy to dynamically adjust the retrieval threshold based on the complexity of the text (such as length, special character frequency, grammatical structure) and the confidence of the feature extraction model output. When the feature extraction model has a high confidence in the text processing, a higher threshold is set to achieve a stricter match. Conversely, when the model confidence is low or the text complexity is high, the threshold is appropriately relaxed to ensure that no potential important information is missed.

[0048] 4. Result re-ranking and confirmation: The selected candidate results will then go through a re-ranking process to ensure that the most likely correct option is at the top. Finally, the first-ranked result, that is, the control vector with the highest similarity to the element vector, is selected as the correct answer after correction.

[0049] The adaptive threshold text error correction mechanism can dynamically adjust according to the characteristics of the text, ensuring that in complex and changing documents, the extraction of core information such as province and city names is both fast and accurate, effectively balancing the strictness and inclusiveness of error correction, and greatly enhancing the reliability and efficiency of feature extraction.

[0050] It should be noted that the execution subject of the element extraction method provided in the embodiment of the present application may be an element extraction device, or a control module in the element extraction device for executing the element extraction method. In the embodiment of the present application, the element extraction device provided in the embodiment of the present application is described by taking the element extraction device executing the element extraction method as an example.

[0051] Figure 6 Schematic diagram of the structure of the element extraction device according to the embodiment of the present application. Figure 6 As shown, the element extraction device 600 includes: a processing module 610 and an extraction module 620.

[0052] The processing module 610 is used to identify the target image to be identified and obtain the cell text in the cell of the target image; the extraction module 620 is used to input the cell text into the element extraction model to extract the target element from the cell text, wherein the element extraction model is trained by a neural network and includes layout information of the target element in multiple templates. In one implementation, the first device is removed from the right, characterized in that it is combined into one section and copied here.

[0053] In one implementation, the processing module 610 is used to obtain pixel points belonging to cells in the target image; determine the cell coordinates of each cell based on the pixel points; obtain text in the target image; and determine the cell text corresponding to each cell based on the text and the cell coordinates.

[0054] In one implementation, the processing module 610 is used to classify the pixel points to determine the cell to which each pixel point belongs; and determine the cell coordinates of each cell by calculating the circumscribed matrix of the pixel points in the cell.

[0055] In one implementation, the processing module 610 is used to determine the cell text corresponding to each cell according to the center point of the text and the position of the text box and the cell of the text.

[0056] In one implementation, the target element includes a first field and a second field corresponding to the first field, and the processing module 610 is further used to preprocess the first training data to obtain second training data, wherein the second training data includes the first field, the second field, and a link relationship between the first field and the second field; obtain multiple templates that carry the target element; and train the element extraction model through the neural network, the multiple templates and the second training data.

[0057] In one implementation, the processing module 610 marks the template data in each of the templates, the template data including the first field, the second field and the link relationship between the first field and the second field in the template; establishes a mapping relationship between the second training data and the template data; inputs the template data into a neural network, and obtains the layout information of the target element through the neural network; and trains the element extraction model through the layout information of the target element and the second training data.

[0058] In one implementation, the extraction module 620 is also used to vectorize the target element to obtain an element vector; search and compare the element vector with a reference vector in a knowledge base through a retrieval threshold to obtain a similarity between the element vector and the reference vector, wherein the retrieval threshold is determined based on the text complexity of the target element and the confidence output by the target element extraction model; and correct the target element using the reference element corresponding to the reference vector with the highest similarity.

[0059] In one implementation, the processing module 610 is further used to correct the target image according to the offset angle of the target image when the target image is offset; and to remove the official seal from the target image when it is detected that the target image includes an official seal.

[0060] The element extraction device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0061] The element extraction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0062] The element extraction device provided in the embodiment of the present application can achieve Figures 2 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0063] like Figure 7 As shown, an embodiment of the present application further provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores a program or instruction that can be run on the processor 701, and when the program or instruction is executed by the processor 701, it implements: identifying a target image to be identified, and obtaining a cell text in a cell of the target image; inputting the cell text into a feature extraction model, and extracting a target feature from the cell text, wherein the feature extraction model is trained by a neural network, and the feature extraction model includes layout information of target features in multiple templates.

[0064] In one implementation, pixel points belonging to cells in the target image are obtained; cell coordinates of each cell are determined based on the pixel points; text in the target image is obtained; and cell text corresponding to each cell is determined based on the text and the cell coordinates.

[0065] In one implementation, the pixel points are classified to determine the cell to which each pixel point belongs; and the cell coordinates of each cell are determined by calculating the circumscribed matrix of the pixel points in the cell.

[0066] In one implementation, the cell text corresponding to each cell is determined based on the center point of the text and the position of the text box and the cell.

[0067] In one implementation, the target element includes a first field and a second field corresponding to the first field. Before the cell text is input into the element extraction model, the first training data is preprocessed to obtain second training data, wherein the second training data includes the first field, the second field, and a link relationship between the first field and the second field to obtain multiple templates that carry the target element; the element extraction model is trained by the neural network, the multiple templates and the second training data.

[0068] In one implementation, the template data in each of the templates is labeled, the template data including the first field, the second field and the link relationship between the first field and the second field in the template; a mapping relationship is established between the second training data and the template data; the template data is input into a neural network, and the layout information of the target element is obtained through the neural network; and the element extraction model is trained through the layout information of the target element and the second training data.

[0069] In one implementation, after extracting the target element from the cell text, the target element is vectorized to obtain an element vector; the element vector is searched and compared with a reference vector in a knowledge base through a retrieval threshold to obtain a similarity between the element vector and the reference vector, wherein the retrieval threshold is determined based on the text complexity of the target element and the confidence output by the target element extraction model; and the target element is corrected using the reference element corresponding to the reference vector with the highest similarity.

[0070] In one implementation, before the target image to be identified is identified, if the target image is offset, the target image is corrected according to the offset angle of the target image; if it is detected that the target image includes an official seal, the official seal in the target image is removed.

[0071] The specific execution steps can refer to the various steps of the above-mentioned element extraction method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0072] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals or other devices except terminals.

[0073] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0074] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0075] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0076] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned element extraction method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0077] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0078] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0080] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for extracting an element, characterized in that: include: Identify the target image to be identified, and obtain the cell text in the cell of the target image; The cell text is input into a feature extraction model to extract target features from the cell text, wherein the feature extraction model is trained by a neural network and includes layout information of target features in a plurality of templates.

2. The method according to claim 1, characterized in that The step of identifying the target image to be identified and obtaining the cell text in the cell of the target image includes: Obtaining pixel points belonging to cells in the target image; Determine the cell coordinates of each cell according to the pixel point; Obtain the text in the target image; The cell text corresponding to each cell is determined according to the text and the cell coordinates.

3. The method according to claim 2, characterized in that Determining the cell coordinates of each cell according to the pixel point includes: Classifying the pixels to determine the cell to which each pixel belongs; The cell coordinates of each cell are determined by calculating the circumscribed matrix of the pixels within the cell.

4. The method according to claim 2, characterized in that: The determining the cell text corresponding to each cell according to the text and the cell coordinates includes: The cell text corresponding to each cell is determined according to the center point of the text and the position of the text box and the cell of the text.

5. The method according to claim 1, characterized in that The target element includes a first field and a second field corresponding to the first field, and before the cell text is input into the element extraction model, the method further includes: Preprocessing the first training data to obtain second training data, wherein the second training data includes the first field, the second field, and a link relationship between the first field and the second field; Acquire multiple templates carrying the target elements; The feature extraction model is trained by using the neural network, the multiple templates and the second training data.

6. The method according to claim 5, characterized in that The step of training the feature extraction model by using the neural network, the multiple templates and the second training data includes: marking the template data in each of the templates, wherein the template data includes the first field, the second field and the link relationship between the first field and the second field in the template; Establishing a mapping relationship between the second training data and the template data; Inputting the template data into a neural network, and obtaining layout information of the target element through the neural network; The feature extraction model is trained using the layout information of the target feature and the second training data.

7. The method according to claim 1, characterized in that After extracting the target element from the cell text, the method further includes: Vectorizing the target element to obtain an element vector; The element vector is searched and compared with a reference vector in a knowledge base by a search threshold to obtain a similarity between the element vector and the reference vector, wherein the search threshold is determined according to the text complexity of the target element and the confidence output by the target element extraction model; The target element is modified by the comparison element corresponding to the comparison vector with the highest similarity.

8. The method according to claim 1, characterized in that Before identifying the target image to be identified, the method further includes: In the case where the target image is offset, correcting the target image according to the offset angle of the target image; When it is detected that the target image includes an official seal, the official seal in the target image is removed.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the element extraction method as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the element extraction method according to any one of claims 1 to 8 are implemented.

Images