Information extraction method, information extraction system, electronic device and storage medium

Abstract

The present application provides an information extraction method, an information extraction system, an electronic device and a storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a target image to be extracted, extracting QR code information from the target image, and obtaining QR code information; performing image segmentation on the target image using an image segmentation model to obtain a target area image; performing text detection on the target area image using a text detection model to obtain at least one set of text row coordinate information; performing text recognition on the target area image using a text recognition model and each set of text row coordinate information to obtain target text information; extracting keyword information from each target text information to obtain multiple target key-value pair information; fusing the QR code information and multiple target key-value pair information to obtain structured data information. The present application can realize the extraction of text information of bills in the form of structured data information, thereby improving the accuracy of information extraction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information extraction method, an information extraction system, an electronic device and a storage medium. Background Art

[0002] At present, Optical Character Recognition (OCR) is widely used in invoice recognition. However, in actual applications, there are many interference factors, such as blurred invoice information, folded and distorted invoices, seal interference, overlap of machine-printed information and printed information, and serialization, which make it very difficult to extract the text information on the invoice, thus affecting the recognition efficiency of information extraction. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an information extraction method, an information extraction system, an electronic device and a storage medium, which can realize the extraction of text information of bills in the form of structured data information, thereby improving the accuracy of information extraction.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes an information extraction method, the method comprising:

[0005] Acquire the target image to be extracted;

[0006] Extracting the two-dimensional code information from the target image to obtain the two-dimensional code information;

[0007] Performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image;

[0008] Performing text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information;

[0009] Performing text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information;

[0010] Extracting keyword information from each target text information to obtain a plurality of target key-value pair information;

[0011] The two-dimensional code information and the multiple target key-value pair information are fused to obtain structured data information.

[0012] In some embodiments, extracting the two-dimensional code information from the target image to obtain the two-dimensional code information includes:

[0013] Performing two-dimensional code detection on the target image to obtain coordinate information of the two-dimensional code area in the target image;

[0014] The two-dimensional code information of the target image is extracted according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code information.

[0015] In some embodiments, extracting the two-dimensional code information from the target image according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code information includes:

[0016] Performing image interception on the target image according to the coordinate information of the two-dimensional code area to obtain a two-dimensional code image;

[0017] The two-dimensional code information is extracted from the two-dimensional code image to obtain the two-dimensional code information.

[0018] In some embodiments, the fusing the two-dimensional code information and the plurality of target key-value pair information to obtain structured data information includes:

[0019] Extract keyword information from each of the two-dimensional code information based on the LayoutLM algorithm to obtain multiple two-dimensional code key-value pair information;

[0020] The multiple target key-value pair information and the multiple two-dimensional code key-value pair information are fused to obtain structured data information.

[0021] In some embodiments, the step of performing text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information includes:

[0022] According to each set of text line coordinate information, the target area image is intercepted to obtain a text image;

[0023] The pre-trained text recognition model is used to perform text recognition on the plurality of text images to obtain target text information.

[0024] In some embodiments, the step of performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image includes:

[0025] Performing image segmentation on the target image using a pre-trained image segmentation model to obtain target area coordinates;

[0026] Based on a perspective transformation algorithm, image correction is performed on the area corresponding to the target area coordinates to obtain a target area image corresponding to the target image.

[0027] In some embodiments, the pre-trained image segmentation model is trained by the following method:

[0028] Constructing a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes an original sample image and a target area sample image;

[0029] Build an initial image segmentation model based on the DeepLab-V3 network structure;

[0030] Inputting a plurality of the original sample images into the initial image segmentation model for model training to obtain a target area training image;

[0031] Determine a loss value according to the target area training image and the target area sample image;

[0032] The model parameters of the initial image segmentation model are adjusted according to the target area training image and the target area sample image, and the adjusted model is continuously trained based on the training sample set until the loss value meets the preset training end condition to obtain the image segmentation model.

[0033] To achieve the above purpose, a second aspect of an embodiment of the present application provides an information extraction system, the system comprising:

[0034] A target image acquisition module is used to acquire the target image to be extracted;

[0035] A first information extraction module, used to extract the two-dimensional code information from the target image to obtain the two-dimensional code information;

[0036] An image segmentation module is used to perform image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image;

[0037] A text detection module, used to perform text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information;

[0038] A text recognition module, used to perform text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information;

[0039] A second information extraction module is used to extract keyword information from each target text information to obtain a plurality of target key-value pair information;

[0040] The information fusion module is used to fuse the two-dimensional code information and the multiple target key-value pair information to obtain structured data information.

[0041] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides an electronic device, including:

[0042] at least one memory;

[0043] at least one processor;

[0044] at least one computer program;

[0045] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement the information extraction method described in the first aspect above.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute an information extraction method described in the first aspect above.

[0047] The present application proposes an information extraction method, an information extraction system, an electronic device and a storage medium, which obtains the target image to be extracted and extracts the QR code information from the target image to obtain the QR code information. In order to improve the accuracy of text information extraction in the target image, the target image is segmented using a pre-trained image segmentation model to obtain a target area image. The target area image is subjected to text detection using a pre-trained text detection model to obtain at least one set of text row coordinate information, so that the text recognition of the target area can be more completely realized. The target area image is subjected to text recognition using a pre-trained text recognition model and each set of text row coordinate information to obtain the target text information; keyword information is extracted from each target text information to obtain multiple target key-value pair information. The QR code information and the multiple target key-value pair information are fused to obtain structured data information. The present application can realize the extraction of text information of bills in the form of structured data information, thereby improving the accuracy of information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of an information extraction method provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S120 in FIG.

[0050] Figure 3 yes Figure 2 Flow chart of step S220 in FIG.

[0051] Figure 4 yes Figure 1 Flowchart of step S170 in FIG.

[0052] Figure 5 yes Figure 1 Flow chart of step S150 in FIG.

[0053] Figure 6 yes Figure 1 Flow chart of step S130 in FIG.

[0054] Figure 7 is a training flow chart of the image segmentation model provided in the embodiment of the present application;

[0055] Figure 8 It is a structural schematic diagram of an information extraction system provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Differentiable Binarization Network (DBNet): is a segmentation-based text detection algorithm that can adapt to image scenes to accurately detect text areas. In terms of threshold selection, the DBNet algorithm uses the network to predict the threshold at each position of the image instead of using a fixed value, so that the background and foreground can be separated well. At the same time, the DBNet algorithm uses the FPN network structure for feature extraction, which can better realize text detection.

[0063] Convolutional Recurrent Neural Network (CRNN): It is a network structure that is mainly used for end-to-end recognition of text sequences of indefinite length. It does not require cutting individual characters first, but converts text recognition into a time-dependent sequence learning problem, that is, image-based sequence recognition.

[0064] Natural Language Processing (NLP): It is the use of computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0065] ZXing: is an open source Java library for parsing 1D / 2D barcodes in various formats. The goal is to be able to decode QR codes, DataMatrix, and UPC 1D barcodes. It provides clients for multiple platforms including: J2ME, J2SE, and Android.

[0066] Encoder-Decoder: It is a common model framework in deep learning. Many common applications are designed using the encoder-decoder framework. The Encoder and Decoder parts can be any text, voice, image, and video data. Various models can be designed based on Encoder-Decoder.

[0067] DeepLab-V3 network structure: It adopts the Encoder-Decoder architecture commonly used in current semantic segmentation networks. The Encoder network structure can be any commonly used network structure, and adopts the Atrous Spatial Pyramid Pooling (ASPP) structure with dilated convolution. ASPP can effectively capture multi-scale information, and uses multiple atrous rate dilated convolution layers in parallel to make the model perform better on multi-scale objects.

[0068] BERT (Bidirectional Encoder Representation from Transformers) model: used to further increase the generalization ability of the word vector model, fully describe the characteristics of character level, word level, sentence level and even inter-sentence relationship, and is built based on Transformer. There are three types of Embedding in BERT, namely Token Embedding, Segment Embedding, and Position Embedding; Token Embeddings is a word vector, and the first word is the CLS mark, which can be used for subsequent classification tasks; Segment Embeddings is used to distinguish between two sentences, because pre-training not only does LM but also does classification tasks with two sentences as input; Position Embeddings, the position word vector here is not the trigonometric function in Transfor, but is learned by BERT through training. However, BERT directly trains a Position Embedding to retain the position information, randomly initializes a vector for each position, adds model training, and finally obtains an Embedding containing position information. Finally, in the combination of this Position Embedding and word Embedding, BERT chooses direct splicing.

[0069] LayoutLM algorithm: It is an effective pre-training model. Based on the BERT model, the LayoutLM algorithm adds two additional input embedding vectors to the model: (1) a 2D position embedding vector, which is used to represent the relative position mark within the document; (2) an image embedding vector within the document, which is used for scanning marks.

[0070] At present, Optical Character Recognition (OCR) is widely used in invoice recognition. However, in actual applications, there are many interference factors, such as blurred invoice information, folded and distorted invoices, seal interference, overlap of machine-printed information and printed information, and serialization, which make it very difficult to extract the text information on the invoice, thus affecting the recognition efficiency of information extraction.

[0071] Based on this, the embodiments of the present application provide an information extraction method, an information extraction system, an electronic device and a storage medium, which can realize the extraction of text information of bills in the form of structured data information, thereby improving the accuracy of information extraction.

[0072] An information extraction method, an information extraction system, an electronic device and a storage medium provided in an embodiment of the present application are specifically illustrated through the following embodiments. First, an information extraction method in an embodiment of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] An information extraction method provided in an embodiment of the present application relates to the field of artificial intelligence technology. An information extraction method provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the information extraction method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] See also Figure 1 , Figure 1 is an optional flowchart of an information extraction method provided in an embodiment of the present application. Figure 1 The method may specifically include but is not limited to steps S110 to S170.

[0078] Step S110, obtaining a target image to be extracted;

[0079] Step S120, extracting the two-dimensional code information from the target image to obtain the two-dimensional code information;

[0080] Step S130, performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image;

[0081] Step S140, performing text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information;

[0082] Step S150, performing text recognition on the target area image using the pre-trained text recognition model and each set of text line coordinate information to obtain target text information;

[0083] Step S160, extracting keyword information from each target text information to obtain a plurality of target key-value pair information;

[0084] Step S170, fusing the two-dimensional code information and multiple target key-value pair information to obtain structured data information.

[0085] In steps S110 to S170 of some embodiments, the target image to be extracted is obtained, and the QR code information is extracted from the target image to obtain the QR code information. In order to improve the accuracy of text information extraction in the target image, the target image is segmented using a pre-trained image segmentation model to obtain a target area image. The target area image is subjected to text detection using a pre-trained text detection model to obtain at least one set of text row coordinate information, so that the text recognition of the target area can be more completely realized. The target area image is subjected to text recognition using a pre-trained text recognition model and each set of text row coordinate information to obtain target text information; keyword information is extracted from each target text information to obtain multiple target key-value pair information. The QR code information and the multiple target key-value pair information are fused to obtain structured data information. The present application can realize text information extraction of bills in the form of structured data information, thereby improving the accuracy of information extraction.

[0086] In step S110 of some embodiments, due to various interference factors in actual applications, such as blurred ticket information, folded and distorted invoices, seal interference, overlap of machine-printed information and printed information, serialization, etc., it is very difficult to extract the text information on the invoice. In order to better extract the text information of the invoice with seal interference, poor copy quality, etc., first, obtain the target image to be extracted. Specifically, the image with the ticket information to be identified can be captured by an image acquisition device, such as a camera, an image scanner, etc., to obtain the target image. For example, when it is necessary to extract information from the uploaded invoice, the target image can be first obtained by using an image acquisition device such as a camera.

[0087] It should be noted that the target image obtained also includes the target image directly obtained from the user's upload, which is not limited here.

[0088] It should be noted that after obtaining the initial image with interference through the image acquisition device, since the initial image may contain background information that may affect information extraction, the initial image can be firstly intercepted to obtain the target image to be extracted.

[0089] In step S120 of some embodiments, in order to improve the completeness of information extraction from the target image, two-dimensional code information is extracted from the target image to obtain the two-dimensional code information. Specifically, the two-dimensional code information is extracted from the target image using a ZXing extraction tool to obtain the two-dimensional code information.

[0090] In step S130 of some embodiments, in order to avoid interference of the background part in the target image on text detection, the target image is segmented using a pre-trained image segmentation model to obtain a target area image, and then text detection and recognition are performed based on the obtained target area image, thereby improving the accuracy of information extraction.

[0091] In step S140 of some embodiments, in order to effectively extract the text information of the bill with interference, the pre-trained text detection model is used to perform text detection on the target area image to obtain at least one set of text line coordinate information. Specifically, the text detection model is used to perform text detection on the target area image to obtain at least one text box containing text information, and at least one set of text line coordinate information corresponding to the target area image is obtained based on the position information of each text box in the target area image.

[0092] It should be noted that, in some specific embodiments, in order to improve the detection accuracy of images, a text detection model based on the DBNet algorithm is adopted to realize text detection of target area images.

[0093] In step S150 of some embodiments, when a text box list consisting of at least one set of text line coordinate information is obtained, text recognition is performed on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information. Specifically, text recognition is performed on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain all text information in the text box corresponding to each set of text line coordinate information. After all text boxes corresponding to the text line coordinate information are subjected to text recognition, all target text information in the target area image is obtained.

[0094] It should be noted that in order to improve the text recognition accuracy of the image corresponding to each set of text row coordinate information, a text recognition model based on the CRNN network structure is used to perform text recognition on the target area image. Specifically, the text recognition model CRNN includes a convolution layer, a circulation layer and a transcription layer. The convolution layer is used to extract features of the corresponding area of ​​each set of text row coordinate information in the target area image to obtain a convolution feature vector, thereby achieving feature dimension reduction and reduction of the amount of calculation for the corresponding area in the target area image. The convolution feature vector is label predicted by the circulation layer to obtain a label distribution. The label distribution is feature integrated by the transcription layer to obtain the target text information corresponding to each set of text row coordinate information. For example, multiple sets of text row coordinate information and target area images are input into a pre-trained text recognition model, and multiple target text information is output, thereby improving the accuracy of information extraction.

[0095] In step S160 of some embodiments, in order to achieve data structuring of the target image, keyword information is extracted from each target text information to obtain multiple target key-value pair information. Specifically, an NLP extraction algorithm can be used to extract keyword information from each target text information to obtain multiple target key-value pair information. For example, the target key-value pair information can be in the form of "name-XXX", where XXX can represent keyword information in the target image. The present application can achieve text information extraction of bills in the form of structured data information, thereby improving the accuracy of information extraction.

[0096] In step S170 of some embodiments, in order to improve the integrity of information extraction, the QR code information and multiple target key-value pair information are fused to obtain structured data information. The present application can improve the accuracy of extracting text information from bills in the form of structured data information and combine it with the QR code information on the bill, thereby realizing the function of automatically entering structured data information into the system.

[0097] It should be noted that the information extraction method of the embodiment of the present application can be applied to different fields. For example, in the insurance field of the financial industry, when it is necessary to extract text information from some bill materials with paper bending and deformation, seal interference, poor copy quality, such as insurance applications, underwriting materials, claims documents, etc. and store them in a structured manner, the information extraction method of the embodiment of the present application can be used. In the form of structured data information and combined with the QR code information on the bill, the ability to extract text information from the bill is improved, thereby improving the accuracy of information extraction.

[0098] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0099] See also Figure 2 , Figure 2 is a flowchart of a specific method of step S120 in some embodiments of the present application. In some embodiments of the present application, step S120 specifically includes but is not limited to step S210 and step S220. Figure 2 These two steps are introduced in detail.

[0100] Step S210, performing a two-dimensional code detection on the target image to obtain coordinate information of the two-dimensional code area in the target image;

[0101] Step S220, extracting the two-dimensional code information from the target image according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code information.

[0102] In step S210 of some embodiments, in order to improve the integrity of information extraction in the target image, the present application extracts the two-dimensional code information from the target image by combining the two-dimensional code information on the bill to obtain structured data information, and obtains the two-dimensional code information. Specifically, the target image is input into a two-dimensional code detector for two-dimensional code detection to determine whether there is a two-dimensional code image in the target image. When there is a two-dimensional code image in the target image, the coordinate information of the two-dimensional code area in the target image is obtained.

[0103] In step S220 of some embodiments, when there is a two-dimensional code image in the target image, the two-dimensional code information is extracted from the target image, specifically, the two-dimensional code information is extracted from the target image according to the coordinate information of the two-dimensional code area to obtain the corresponding two-dimensional code information. For example, the two-dimensional code information is extracted from the two-dimensional code area in the target area using the extraction tool ZXing to obtain the corresponding two-dimensional code information.

[0104] See also Figure 3 , Figure 3 is a flowchart of a specific method of step S220 in some embodiments of the present application. In some embodiments of the present application, step S220 specifically includes but is not limited to step S310 and step S320. Figure 3 These two steps are introduced in detail.

[0105] Step S310, capturing the target image according to the coordinate information of the two-dimensional code area to obtain a two-dimensional code image;

[0106] Step S320, extracting the two-dimensional code information from the two-dimensional code image to obtain the two-dimensional code information.

[0107] In step S310 and step S320 of some embodiments, when there is a two-dimensional code image in the target image, in order to improve the accuracy of extracting the two-dimensional code information in the target image, the target image is intercepted according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code image. Specifically, the image segmentation algorithm based on the DeepLab-V3 network structure is used to intercept the target image to obtain the coordinate information of the two-dimensional code area in the target image, that is, the two-dimensional code image in the target image can be obtained according to the four vertex coordinates of the two-dimensional code area. And the two-dimensional code information is extracted from the two-dimensional code image to obtain the two-dimensional code information corresponding to the target area.

[0108] See also Figure 4 , Figure 4 is a flowchart of a specific method of step S170 in some embodiments of the present application. In some embodiments of the present application, step S170 specifically includes but is not limited to step S410 and step S420. Figure 4 These two steps are introduced in detail.

[0109] Step S410, extracting keyword information from each QR code information based on the LayoutLM algorithm to obtain multiple QR code key-value pair information;

[0110] Step S420: fuse the multiple target key-value pair information and the multiple QR code key-value pair information to obtain structured data information.

[0111] In step S410 and step S420 of some embodiments, in order to improve the integrity of information extraction, the two-dimensional code information and multiple target key-value pair information are fused to obtain structured data information. The keyword information of each two-dimensional code information is extracted using the LayoutLM algorithm to obtain multiple two-dimensional code key-value pair information. Multiple target key-value pair information and multiple two-dimensional code key-value pair information are fused, that is, all extracted target key-value pair information and all two-dimensional code key-value pair information are merged to obtain structured data information corresponding to the target image. The present application can improve the extraction of text information of bills in the form of structured data information and in combination with the two-dimensional code information on the bill, thereby improving the accuracy of information extraction and realizing the function of automatically entering structured data information into the system.

[0112] It should be noted that the QR code key-value pair information may also be in the form of "total amount-YYY", where YYY may represent the corresponding keyword information in the QR code image. The present application can extract text information of bills in the form of structured data information, thereby improving the accuracy of information extraction.

[0113] See also Figure 5 , Figure 5 is a flowchart of a specific method of step S150 in some embodiments of the present application. In some embodiments of the present application, step S150 specifically includes but is not limited to step S510 and step S520. Figure 5 These two steps are introduced in detail.

[0114] Step S510, performing image interception on the target area image according to each set of text line coordinate information to obtain a text image;

[0115] Step S520: Perform text recognition on multiple text images using a pre-trained text recognition model to obtain target text information.

[0116] In step S510 and step S520 of some embodiments, in order to improve the text recognition accuracy of the text recognition model for the target area image, the target area image is intercepted according to each set of text line coordinate information to obtain a text image. Then, the pre-trained text recognition model is used to perform text recognition on multiple text images to obtain multiple target text information corresponding to each text image.

[0117] See also Figure 6 , Figure 6 is a flowchart of a specific method of step S130 in some embodiments of the present application. In some embodiments of the present application, step S130 specifically includes but is not limited to step S610 and step S620. Figure 6 These two steps are introduced in detail.

[0118] Step S610, using a pre-trained image segmentation model to segment the target image to obtain target area coordinates;

[0119] Step S620, performing image correction on the area corresponding to the target area coordinates based on a perspective transformation algorithm to obtain a target area image corresponding to the target image.

[0120] In step S610 and step S620 of some embodiments, in order to avoid the influence of the background part of the target area in the target image on the text detection, the target image is segmented using a pre-trained image segmentation model to obtain the target area coordinates. Specifically, for example, when the target image is an invoice image, the target image is segmented using an image segmentation model to obtain the target area coordinates, that is, the four vertex coordinates corresponding to the invoice area in the invoice image. In order to avoid the influence of the background part of the invoice area in the invoice image on the text detection, the area corresponding to the invoice area coordinates is image-corrected based on the perspective transformation algorithm to obtain the invoice area image corresponding to the invoice image. And the text detection model is used to perform text detection on the target area image to obtain at least one set of text row coordinate information. The text recognition model and each set of text row coordinate information are used to perform text recognition on the target area image to obtain all the target text information. The present application can improve the extraction of text information of the bill by using structured data information and combining the perspective transformation algorithm to perform image correction on the area corresponding to the target area coordinates, thereby improving the accuracy of information extraction.

[0121] See also Figure 7 , Figure 7 is a training flow chart of the pre-trained image segmentation model of the present application. In some embodiments of the present application, the training process of the image segmentation model specifically includes but is not limited to steps S710 to S750. Figure 7 These five steps are introduced in detail.

[0122] Step S710, constructing a training sample set, the training sample set including a plurality of training samples, each training sample including an original sample image and a target area sample image;

[0123] Step S720, constructing an initial image segmentation model based on the DeepLab-V3 network structure;

[0124] Step S730, inputting a plurality of original sample images into an initial image segmentation model for model training to obtain a target area training image;

[0125] Step S740, determining a loss value according to the target area training image and the target area sample image;

[0126] Step S750, adjust the model parameters of the initial image segmentation model according to the target area training image and the target area sample image, and continue to train the adjusted model based on the training sample set until the loss value meets the preset training end condition to obtain the image segmentation model.

[0127] In step S710 of some embodiments, in order to accurately and effectively obtain the target area image for text recognition in the target image, the target image is segmented using a pre-trained image segmentation model to obtain the target area image. Specifically, a training sample set is first constructed, the training sample set includes a plurality of training samples, each training sample includes an original sample image and a target area sample image, and a large number of training samples are sent to the initial image segmentation model for training, so that the obtained image segmentation model can more accurately achieve image segmentation of the target image.

[0128] In steps S720 to S750 of some embodiments, in order to obtain a more accurate image segmentation model for image segmentation, an initial image segmentation model is constructed based on the DeepLab-V3 network structure, and multiple original sample images are input into the initial image segmentation model for model training to obtain a target area training image. The loss value is determined according to the target area training image and the target area sample image, and the model parameters of the initial image segmentation model are adjusted according to the target area training image and the target area sample image. The adjusted model is continued to be trained based on the training sample set until the loss value meets the preset training end condition to obtain an image segmentation model. The present application can obtain a more accurate target area image in the form of structured data information, combined with the image segmentation model obtained by training based on the DeepLab-V3 network structure, thereby improving the accuracy of information extraction.

[0129] An information extraction method provided by an embodiment of the present application obtains a target image to be extracted, and in order to improve the integrity of information extraction in the target image, performs two-dimensional code detection on the target image to obtain the coordinate information of the two-dimensional code area in the target image. And the target image is intercepted according to the coordinate information of the two-dimensional code area to obtain a two-dimensional code image. The two-dimensional code information is extracted from the two-dimensional code image to obtain the corresponding two-dimensional code information. In order to avoid the influence of the background part of the target area in the target image on the text detection, in order to improve the accuracy of text information extraction in the target image, the target image is segmented using a pre-trained image segmentation model to obtain the target area coordinates, and the area corresponding to the target area coordinates is corrected based on the perspective transformation algorithm to obtain the target area image corresponding to the target image. The target area image is subjected to text detection using a pre-trained text detection model to obtain at least one set of text row coordinate information, so that the text recognition of the target area can be more completely realized. And the target area image is subjected to text recognition using a pre-trained text recognition model and each set of text row coordinate information to obtain target text information. In order to improve the integrity of information extraction, keyword information is extracted from each target text information to obtain multiple target key-value pair information. The keyword information of each QR code information is extracted based on the LayoutLM algorithm to obtain multiple QR code key-value pair information. In order to achieve data structuring of the target image, multiple target key-value pair information and multiple QR code key-value pair information are fused to obtain structured data information. This application can improve the text information extraction of the bill in the form of structured data information and combine it with the QR code information on the bill, improve the accuracy of information extraction, and realize the function of automatically entering structured data information into the system.

[0130] See also Figure 8 The embodiment of the present application also provides an information extraction system that can implement the above-mentioned information extraction method. The system includes a target image acquisition module 810, a first information extraction module 820, an image segmentation module 830, a text detection module 840, a text recognition module 850, a second information extraction module 860 and an information fusion module 870.

[0131] The target image acquisition module 810 is used to acquire the target image to be extracted;

[0132] The first information extraction module 820 is used to extract the two-dimensional code information from the target image to obtain the two-dimensional code information;

[0133] An image segmentation module 830 is used to segment the target image using a pre-trained image segmentation model to obtain a target area image;

[0134] A text detection module 840 is used to perform text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information;

[0135] A text recognition module 850 is used to perform text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information;

[0136] The second information extraction module 860 is used to extract keyword information from each target text information to obtain multiple target key-value pair information;

[0137] The information fusion module 870 is used to fuse the two-dimensional code information and multiple target key-value pair information to obtain structured data information.

[0138] It should be noted that an information extraction system in an embodiment of the present application is used to implement the above-mentioned information extraction method. The information extraction system in an embodiment of the present application corresponds to the aforementioned information extraction method. For the specific processing process, please refer to the aforementioned information extraction method, which will not be repeated here.

[0139] An information extraction system provided by an embodiment of the present application can implement the above-mentioned information extraction method, by acquiring a target image to be extracted, in order to improve the completeness of information extraction in the target image, performing two-dimensional code detection on the target image, and obtaining the coordinate information of the two-dimensional code area in the target image. And according to the coordinate information of the two-dimensional code area, the target image is intercepted to obtain a two-dimensional code image. The two-dimensional code information is extracted from the two-dimensional code image to obtain the corresponding two-dimensional code information. In order to avoid the influence of the background part of the target area in the target image on the text detection, in order to improve the accuracy of text information extraction in the target image, the target image is segmented using a pre-trained image segmentation model to obtain the target area coordinates, and the area corresponding to the target area coordinates is corrected based on the perspective transformation algorithm to obtain the target area image corresponding to the target image. The target area image is subjected to text detection using a pre-trained text detection model to obtain at least one set of text row coordinate information, so that the text recognition of the target area can be more completely realized. And the target area image is subjected to text recognition using a pre-trained text recognition model and each set of text row coordinate information to obtain the target text information. In order to improve the integrity of information extraction, keyword information is extracted for each target text information to obtain multiple target key-value pair information. The LayoutLM algorithm is used to extract keyword information for each QR code information to obtain multiple QR code key-value pair information. In order to achieve data structuring of the target image, multiple target key-value pair information and multiple QR code key-value pair information are fused to obtain structured data information. The present application can improve the text information extraction of the bill in the form of structured data information and combine it with the QR code information on the bill, thereby improving the accuracy of information extraction and realizing the function of automatically entering structured data information into the system.

[0140] The embodiment of the present application also provides an electronic device, which includes: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement any of the information extraction methods in the above embodiments. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0141] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0142] The processor 910 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0143] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes an information extraction method of the embodiment of this application;

[0144] Input / output interface 930, used to implement information input and output;

[0145] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0146] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0147] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0148] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute any one of the information extraction methods in the above embodiments.

[0149] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0151] It can be understood by those skilled in the art that Figures 1 to 7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0152] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0154] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0155] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0157] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0160] The preferred embodiments of the present invention are described with reference to the accompanying drawings, but the scope of the present invention is not limited by the above. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An information extraction method, characterized in that: The method comprises: Acquire a target image to be extracted; wherein the target image is derived from any one of an insurance application form, underwriting information, and a claim settlement document; Extracting the two-dimensional code information from the target image to obtain the two-dimensional code information; Performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image; wherein, performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image comprises: performing image segmentation on the target image using a pre-trained image segmentation model to obtain target area coordinates; performing image correction on an area corresponding to the target area coordinates based on a perspective transformation algorithm to obtain a target area image corresponding to the target image; Performing text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information; Performing text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information; Extracting keyword information from each target text information to obtain a plurality of target key-value pair information; The two-dimensional code information and the multiple target key-value pair information are fused to obtain structured data information; wherein, the two-dimensional code information and the multiple target key-value pair information are fused to obtain structured data information, including: extracting keyword information from each of the two-dimensional code information based on the LayoutLM algorithm to obtain multiple two-dimensional code key-value pair information; fusing the multiple target key-value pair information with the multiple two-dimensional code key-value pair information to obtain structured data information.

2. The information extraction method according to claim 1, characterized in that: The step of extracting the two-dimensional code information from the target image to obtain the two-dimensional code information includes: Performing two-dimensional code detection on the target image to obtain coordinate information of the two-dimensional code area in the target image; The two-dimensional code information of the target image is extracted according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code information.

3. An information extraction method according to claim 2, characterized in that: The step of extracting the two-dimensional code information from the target image according to the coordinate information of the two-dimensional code area to obtain the two-dimensional code information includes: Performing image interception on the target image according to the coordinate information of the two-dimensional code area to obtain a two-dimensional code image; The two-dimensional code information is extracted from the two-dimensional code image to obtain the two-dimensional code information.

4. The information extraction method according to claim 1, characterized in that: The method of performing text recognition on the target area image using the pre-trained text recognition model and each set of text line coordinate information to obtain target text information includes: According to each set of text line coordinate information, the target area image is intercepted to obtain a text image; The pre-trained text recognition model is used to perform text recognition on the plurality of text images to obtain target text information.

5. An information extraction method according to any one of claims 1 to 4, characterized in that: The pre-trained image segmentation model is trained by the following method: Constructing a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes an original sample image and a target area sample image; Build an initial image segmentation model based on the DeepLab-V3 network structure; Inputting a plurality of the original sample images into the initial image segmentation model for model training to obtain a target area training image; Determine a loss value according to the target area training image and the target area sample image; The model parameters of the initial image segmentation model are adjusted according to the target area training image and the target area sample image, and the adjusted model is continuously trained based on the training sample set until the loss value meets the preset training end condition to obtain the image segmentation model.

6. An information extraction system, characterized in that: The system comprises: A target image acquisition module, used to acquire a target image to be extracted; wherein the target image is derived from any one of an insurance application form, underwriting information, and a claim settlement document; A first information extraction module, used to extract the two-dimensional code information from the target image to obtain the two-dimensional code information; An image segmentation module is used to perform image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image; wherein, performing image segmentation on the target image using a pre-trained image segmentation model to obtain a target area image comprises: performing image segmentation on the target image using a pre-trained image segmentation model to obtain target area coordinates; performing image correction on an area corresponding to the target area coordinates based on a perspective transformation algorithm to obtain a target area image corresponding to the target image; A text detection module, used to perform text detection on the target area image using a pre-trained text detection model to obtain at least one set of text line coordinate information; A text recognition module, used to perform text recognition on the target area image using a pre-trained text recognition model and each set of text line coordinate information to obtain target text information; A second information extraction module is used to extract keyword information from each target text information to obtain a plurality of target key-value pair information; An information fusion module is used to fuse the two-dimensional code information and the multiple target key-value pair information to obtain structured data information; wherein, the two-dimensional code information and the multiple target key-value pair information are fused to obtain structured data information, including: extracting keyword information from each of the two-dimensional code information based on the LayoutLM algorithm to obtain multiple two-dimensional code key-value pair information; fusing the multiple target key-value pair information with the multiple two-dimensional code key-value pair information to obtain structured data information.

7. An electronic device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: An information extraction method as claimed in any one of claims 1 to 5.

8. A storage medium, the storage medium being a computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: An information extraction method as claimed in any one of claims 1 to 5.

Images