Certificate information extraction method, device, equipment and medium

Abstract

The application provides a certificate information extraction method, device, equipment and medium. The method comprises: classifying an original image using a classification model to obtain classification information; if it is determined according to the classification information that the original image belongs to a certificate image, performing scaling processing on the original image based on the image size of the original image and a preset value to obtain a scaled image; detecting the scaled image using a text detection model to obtain first text box coordinates; taking a screenshot of the original image according to second text box coordinates corresponding to the first text box coordinates to obtain a local text image; inputting the local text image into a text recognition model to obtain text information; inputting the original image, a certificate type of the original image, the second text box coordinates and the text information into an information extraction model to obtain a field type; and filtering the text information according to the field type and the certificate type to obtain certificate information. Using the above method, certificate information can be effectively extracted, and the accuracy of certificate information extraction is improved.

Description

Technical Field

[0001] This application belongs to the field of computer science, specifically the field of image recognition technology, and in particular relates to a method, apparatus, device, and medium for extracting document information. Background Technology

[0002] With the increasing prevalence of online business, more and more offline verification processes are shifting online. For example, e-commerce platforms need to manage the risks of merchants, thus requiring verification of the authenticity of the merchants' identification information. Currently, the extraction of identification information has strict requirements regarding the size and type of the identification image. The extraction process is also limited by factors such as the standardized layout of the text information within the identification image, the completeness of the text information, and the accuracy of the recognition information obtained from the identification image. If the size, type, or layout of the identification image does not meet the model's extraction requirements, the text information is incomplete, or there are errors in the recognition information, the accuracy of the extracted identification information will be poor, or the identification information may be impossible to extract.

[0003] If the technical issues of inaccurate document information and inability to extract document information cannot be resolved, it will affect the certification of qualifications and make it impossible to ensure the normal operation of e-commerce platforms and stores. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, device, equipment and medium for extracting document information, which can solve the technical problems of poor accuracy or inability to extract document information.

[0005] On one hand, this application provides a method for extracting document information. The method includes: acquiring an original image and classifying the original image using a preset classification model to obtain classification information of the original image, wherein the classification information includes document type information, which indicates whether the original image is a document image, and the document type of the original image when the original image is a document image; if the original image is determined to be a document image based on the document type information, scaling the original image based on its image size and a preset value to obtain a scaled image, and performing text detection on the scaled image using a preset text detection model. The coordinates of the first text box corresponding to the scaled image are obtained by measurement. The coordinates of the first text box are adjusted to obtain the coordinates of the second text box. The original image is then cropped according to the coordinates of the second text box to obtain a partial text image. The partial text image is input into a preset text recognition model to obtain the text information in the partial text image. The original image, the document type, the coordinates of the second text box, and the text information are input into a preset information extraction model to obtain the field type of the text information. The text information is then filtered according to the field type and the document type to obtain the document information in the original image.

[0006] In some embodiments of this application, the preset value includes a first preset value and a second preset value. The scaling process of the original image based on its image size and the preset value to obtain a scaled image includes: determining the image size of the original image, wherein the image size includes an image width and an image height; determining a larger and a smaller value between the image width and the image height; calculating a first target value based on the image width and the image height; if the larger value is less than the first preset value, scaling the larger value based on the first preset value; calculating a second target value based on the first target value and the first preset value; and scaling the smaller value based on the second target value to obtain the scaled image; or, if the larger value is greater than the first preset value, scaling the larger value based on the second preset value; calculating a second target value based on the first target value and the second preset value; and scaling the smaller value based on the second target value to obtain the scaled image.

[0007] In some embodiments of this application, adjusting the coordinates of the first text box to obtain the coordinates of the second text box includes: inverse scaling the coordinates of the first text box to obtain inversely scaled text box coordinates, and adjusting the inversely scaled text box coordinates using a preset adjustment strategy to obtain the coordinates of the second text box.

[0008] In some embodiments of this application, when the text box is rectangular, there are multiple inverse scaling text box coordinates. Adjusting the inverse scaling text box coordinates using a preset adjustment strategy to obtain the second text box coordinates includes: calculating the text box width and height based on the multiple inverse scaling text box coordinates; if the text box width is greater than the text box height, and the document type belongs to a preset document type, calculating a target width based on a first preset configuration value and the text box height, and calculating a target height based on a second preset configuration value and the text box height; or, if the text box width is greater than the text box height, and the document type does not belong to the preset document type, calculating a target width based on a third preset configuration value and the text box height, and calculating a target height based on a fourth preset configuration value and the text box height. Calculate the target height; or, if the width of the text box is less than or equal to the height of the text box, and the document type belongs to the preset document type, calculate the target height based on the first preset configuration value and the width of the text box, and calculate the target width based on the second preset configuration value and the width of the text box; or, if the width of the text box is less than or equal to the height of the text box, and the document type does not belong to the preset document type, calculate the target height based on the third preset configuration value and the width of the text box, and calculate the target width based on the fourth preset configuration value and the width of the text box. Adjust the width of the text box based on the target width, and adjust the height of the text box based on the target height to obtain the target text box. Then, determine the coordinates of each inversely scaled text box in the target text box as the coordinates of the second text box.

[0009] In some embodiments of this application, the information extraction model includes a location embedding layer, a self-attention model, a normalization layer, an averaging layer, a convolutional neural network layer, and a linear layer. The step of inputting the original image, the document type, the coordinates of the second text box, and the text information into a preset information extraction model to obtain the field type of the text information includes: encoding and fusing the document type and the text information to obtain fused information; inputting the fused information into the self-attention model to obtain semantic feature information; encoding the coordinates of the second text box using the location embedding layer to obtain location encoding information; normalizing the semantic feature information and the location encoding information using the normalization layer to obtain a word embedding vector; inputting the word embedding vector into the self-attention model to obtain text context feature information; taking a screenshot of the original image using the coordinates of the second text box to obtain a screenshot image; extracting features from the screenshot image using the convolutional neural network layer to obtain visual context feature information; and calling the averaging layer, the linear layer, and the self-attention model to generate the field type based on the visual context information and the text context information.

[0010] In some embodiments of this application, the step of calling the averaging layer, the linear layer, and the self-attention model to generate the field type based on the visual context information and the text context information includes: calling the averaging layer to average the visual context information to obtain average visual context information; calling the linear layer to perform fully connected operations on the average visual context information and the text context information respectively to obtain visual feature information of the average visual context information and text feature information of the text context information; performing information fusion on the text feature information and the visual feature information to obtain multimodal context feature information; and using the self-attention model to aggregate the multimodal context feature information to obtain the field type.

[0011] In some embodiments of this application, the step of filtering the text information according to the field type and the document type to obtain the document information in the original image includes: classifying and extracting the text information according to the field type to obtain structured information; filtering the structured information according to the document type to obtain filtered structured information; verifying the target information in the filtered structured information according to preset rules; and when the target information is successfully verified, using the filtered structured information as the document information.

[0012] On the other hand, this application provides a document information extraction device, operating in an electronic device. The device includes: an acquisition unit for acquiring an original image; a classification unit for classifying the original image using a preset classification model to obtain classification information of the original image, wherein the classification information includes document type information, which indicates whether the original image is a document image, and, when the original image is a document image, the document type of the original image; a scaling unit for scaling the original image based on its image size and a preset value, if the original image is determined to be a document image according to the document type information, to obtain a scaled image; and a detection unit for detecting the scaled image using a preset text detection model. The system includes a measurement unit for obtaining the coordinates of a first text box corresponding to the scaled image, an adjustment unit for adjusting the coordinates of the first text box to obtain the coordinates of a second text box, a screenshot unit for taking a screenshot of the original image based on the coordinates of the second text box to obtain a partial text image, an input unit for inputting the partial text image into a preset text recognition model to obtain the text information in the partial text image, and the input unit for inputting the original image, the document type, the coordinates of the second text box, and the text information into a preset information extraction model to obtain the field type of the text information, and a filtering unit for filtering the text information based on the field type and the document type to obtain the document information in the original image.

[0013] On the other hand, this application provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing at least one instruction to implement the document information extraction method.

[0014] On the other hand, this application provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the document information extraction method.

[0015] Through the above implementation methods, it is possible to accurately determine whether the original image is an ID image based on the document type information. When the original image is not an ID image, no information extraction is performed on the original image, thus saving time. When the original image is determined to be an ID image, it is scaled based on a preset value and / or the document type of the original image. This allows for flexible adjustment of the image size, ensuring that the scaled image meets the size requirements of the text detection model and avoiding the problem of not being able to extract document information due to the size of the document image not meeting the model's extraction requirements. This improves the detection accuracy of the text detection model. Based on the coordinates of the second text box, the position of the text information in the original image can be accurately determined. By taking a screenshot based on the coordinates of the second text box, a local text image containing the text information can be accurately obtained, reducing interference from text layout issues. Since the local text image obtained by the screenshot process removes redundant parts, using a text recognition model to recognize the local text image can improve the recognition accuracy and speed of the text information. By inputting the original image, document type, second text box coordinates, and text information into the information extraction model, the accuracy of the field type can be improved due to the use of multiple information fusion extraction methods. Filtering text information based on field type and document type can remove erroneous information, reduce reliance on the accuracy and completeness of identification information, thereby effectively extracting document information and improving the accuracy of the filtered document information. Attached Figure Description

[0016] Figure 1 This is a structural diagram of an electronic device provided in an embodiment of this application.

[0017] Figure 2 This is a flowchart of a document information extraction method provided in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of a pixel coordinate system provided in an embodiment of this application.

[0019] Figure 4 This is a flowchart of a method for generating scaled images according to another embodiment of this application.

[0020] Figure 5 This is a flowchart of a method for generating the coordinates of a second text box according to an embodiment of this application.

[0021] Figure 6 This is a flowchart of a method for generating field types provided in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram of the workflow of an information extraction model provided in an embodiment of this application.

[0023] Figure 8 This is a flowchart of a method for generating document information according to an embodiment of this application.

[0024] Figure 9 This is a functional block diagram of a document information extraction device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0027] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0028] With the increasing growth of online business, more and more of the business that used to be conducted offline is now being moved online. For example, e-commerce platforms need to manage the risks of merchants, so they need to verify the authenticity of the merchant's credentials.

[0029] Currently, document information extraction is mainly based on keyword matching and Natural Language Processing (NLP) methods. However, these methods impose strict requirements on the size and type of the document image. The extraction of document information is easily limited by factors such as the standardized layout of the text information in the document image, the completeness of the text information, the clarity of the document image, and the accuracy of the recognition information obtained from the document image. If the size, type, or layout of the document image does not meet the model's extraction requirements, the text information is incomplete, or the recognition information contains errors, the accuracy of the extracted document information will be poor, or the document information may be impossible to extract.

[0030] If the technical issues of inaccurate document information and inability to extract document information cannot be resolved, it will affect the certification of qualifications and make it impossible to ensure the normal operation of e-commerce platforms and stores.

[0031] To address the aforementioned technical problems, this application provides a method, apparatus, device, and medium for extracting document information, which can effectively extract document information and improve the accuracy of document information extraction. The document information extraction method provided in this application can be applied to one or more electronic devices.

[0032] like Figure 1 The diagram shown is a structural diagram of an electronic device provided in one embodiment of this application. The electronic device 10 can be a mobile phone, tablet computer, laptop computer, self-moving device, or other electronic device. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0033] like Figure 1 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the input / output interface 104 via the bus 105.

[0034] Communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, frequency modulation (FM), near field communication (NFC), and infrared (IR).

[0035] Memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The RAM can be directly read and written by the processor 103, and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0036] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110. Non-volatile memory can include disk storage devices and flash memory.

[0037] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions that, when executed by the processor 103, enable a document information retrieval method to be executed on the electronic device 10.

[0038] In other embodiments, such as Figure 1 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.

[0039] Processor 103 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0040] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute computer programs stored in the memory 102 to implement the above-described document information extraction method.

[0041] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input / output devices, such as a mouse, keyboard, touch device, display screen, etc., so that users can enter information or visualize information.

[0042] Bus 105 is used at least to provide a channel for communication between communication modules 101, memory 102, processor 103, and input / output interface 104 in electronic device 10.

[0043] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] like Figure 2 The diagram shown is a flowchart of a document information extraction method according to an embodiment of this application. The order of steps in the flowchart can be adjusted according to different needs, and some steps can be omitted. The method is executed by an electronic device, for example... Figure 1 The electronic device 10 shown.

[0045] S10, acquire the original image.

[0046] In some embodiments, electronic devices can acquire original images in various ways. For example, an electronic device can identify an image transmitted from the front end to the middle platform of an e-commerce platform as the original image. The front end can be a graphical user interface (GUI), including, but not limited to, the website interface and application interface of the e-commerce platform. Through the GUI interface of the front end, users can upload images by operating relevant controls (e.g., image upload controls). The middle platform represents the intermediate layer between the front end and the back end of the e-commerce platform, responsible for handling business logic, data transmission, and interface calls. The front end can collect images uploaded by various merchants and other users, and send the collected images to the middle platform for storage or verification. The back end of the e-commerce platform can include servers, databases, etc., responsible for handling user requests, storing and managing data, and processing business logic.

[0047] In this embodiment, the process of acquiring the original image described above is for reference only, and the actual application is not limited to the example above.

[0048] S11. Classify the original image using a preset classification model to obtain classification information of the original image. The classification information includes document type information, which indicates whether the original image is a document image and the document type of the original image when the original image is a document image.

[0049] In some embodiments of this application, the classification model can be a multi-classification model, which can be a machine learning model capable of determining document angle, document type, and document detection. For example, classification models include, but are not limited to, the YOLOv5m model, the RetinaNet model, and the EfficientDet model; this application does not limit the classification model. Classification information includes, but is not limited to, document type, document angle, and document frame coordinates. The document angle indicates the tilt angle of the document in the original image, and includes, but is not limited to, 0 degrees, 90 degrees, 180 degrees, and 270 degrees. The document frame coordinates indicate the coordinates of the document frame corresponding to the document in the original image. The document frame includes all pixels corresponding to the document in the original image, and the document frame coordinates are determined by the shape of the document frame. For example, the document angle can be 270 degrees, and the document type can be an ID card or a business license. If the document frame is rectangular, the document frame coordinates can be represented by the coordinates of the four vertices (pixels) of the rectangle or by the coordinates of the two vertices on the diagonal of the rectangle.

[0050] In this embodiment, the coordinates of the vertices (pixels) are the coordinates in the pixel coordinate system corresponding to the original image. For example, as... Figure 3The diagram shown is a schematic representation of a pixel coordinate system provided in an embodiment of this application. The electronic device constructs a pixel coordinate system with the pixel point Oxy in the first row and first column of the original image as the origin, the parallel line containing the pixels in the first row as the x-axis, and the vertical line containing the pixels in the first column as the y-axis. The establishment of the above pixel coordinate system is merely illustrative and is not limited to this in practical applications.

[0051] In some embodiments of this application, before classifying the original image using a preset classification model, the electronic device can adjust (scale) the resolution of the original image according to the resolution requirements of the classification model for the input image. For example, when the classification model is the YOLOv5m model, the resolution of the original image can be adjusted to 640x640, and then the adjusted original image can be input into the classification model for classification, which can improve the clarity of the original image and thus improve the classification efficiency of the classification model.

[0052] In some embodiments of this application, the classification model described above can be obtained by training, testing, and validating the machine learning algorithm corresponding to the classification model. For example, if the classification model is a YOLOv5m model, the machine learning algorithm corresponding to the YOLOv5m model is the YOLOv5m algorithm.

[0053] In some embodiments of this application, in order to train, test and validate machine learning algorithms, electronic devices acquire datasets and divide the datasets into training sets, test sets and validation sets according to a preset ratio.

[0054] The dataset includes multiple ID card images and annotation information for each image. The method for obtaining the multiple ID card images in the dataset is basically the same as the method for obtaining the original images described above. The annotation information and classification information for each ID card image are basically the same, so this application will not repeat the description. The preset ratio for dividing the dataset can be set by the user, and this application does not impose any restrictions on it. For example, the preset ratio for dividing the dataset can be training set:validation set:test set = 7:1:2 or training set:validation set:test set = 6:2:2, etc. The training set, test set, and validation set all include ID card images of various ID card types. To ensure the generalization of the initial model, when dividing the dataset into training set, test set, and validation set, it is necessary to ensure that there are no duplicate ID card images between the training set and the test set (the training set and the test set are independent of each other), and / or, there are no duplicate ID card images between the training set and the validation set (the training set and the validation set are independent of each other).

[0055] In some embodiments of this application, the electronic device uses a training set to train the YOLOv5m algorithm until convergence to obtain an initial model, and uses a test set to test the initial model. When the initial model passes the test, a validation set is used to validate the initial model. When the initial model passes the validation, the initial model is determined as a classification model.

[0056] When testing and validating the initial model, the test accuracy of the initial model on the test set and the validation accuracy of the initial model on the validation set can be used as evaluation metrics. The test accuracy is the percentage of the number of document images in the test set that are correctly classified when the initial model classifies the test set. The validation accuracy is the percentage of the number of document images in the validation set that are correctly classified when the initial model classifies the validation set.

[0057] For example, if the initial model's accuracy on the test set is greater than or equal to a preset accuracy threshold, the initial model is determined to have passed the test; if the initial model's accuracy on the test set is less than the preset accuracy threshold, the initial model is determined to have failed the test. Alternatively, if the initial model's accuracy on the validation set is greater than or equal to a preset accuracy threshold, the initial model is determined to have passed validation; if the initial model's accuracy on the validation set is less than the preset accuracy threshold, the initial model is determined to have failed validation. For example, the preset accuracy threshold can be set to 0.8, 0.9, etc., and is not limited to these in practical applications.

[0058] In other embodiments of this application, the area under the receiver operating characteristic curve (AUC) and the Kolmogorov-Smirnov distance (KS) can also be used as evaluation metrics to assess the performance of the initial model and determine whether the initial model has passed the test and validation.

[0059] In this embodiment, since there are no duplicate ID images between the training set and the test set, and / or there are no duplicate ID images between the training set and the validation set, when the initial model passes the validation on the test set, and / or when the initial model passes the validation on the validation set, it means that the initial model has high generalization and robustness, and that the initial model can exhibit similar performance on different datasets and can adapt to different data distributions and changes.

[0060] S12, determine whether the original image belongs to an ID image based on the ID type information.

[0061] In some embodiments of this application, when it is determined that the original image does not belong to the document image, the electronic device performs step S13, and when it is determined that the original image belongs to the document image, the electronic device performs step S14.

[0062] In some embodiments of this application, since the document type information can indicate whether the original image is a document image (whether the original image includes a document) and the document type of the original image when the original image is a document image, the document type information can be used to directly determine whether the original image is a document image.

[0063] For example, if the document type information of the original image is "ID card," then the electronic device determines that the original image is an document image, and the document type of the original image is an ID card. Alternatively, if the document type information of the original image is "Original image is not a document image," then the electronic device determines that the original image is not a document image. Or, if the document type information of the original image is "Business license," then the electronic device determines that the original image is a document image, and the document type of the original image is a business license.

[0064] S13, output preset prompt information, which is used to indicate that the original image does not belong to the document image.

[0065] In some embodiments of this application, the preset prompt message can be set independently, and this application does not impose any restrictions on it. For example, the preset prompt message could be: This image is not an ID card image, please confirm.

[0066] In this embodiment, when it is determined that the original image does not belong to the document image, the electronic device does not scale the original image, which can save time.

[0067] S14, scale the original image based on its original image size and preset values ​​to obtain a scaled image.

[0068] In some embodiments of this application, if the classification information also includes the document angle and document frame coordinates, the electronic device can preprocess the original image before scaling it to obtain a scaled image. Preprocessing includes, but is not limited to, cropping and rotation.

[0069] Specifically, the preprocessing of the original image by the electronic device includes: cropping the original image according to the coordinates of the document frame, and rotating the cropped original image according to the document angle to obtain the preprocessed original image.

[0070] In some embodiments of this application, the cropped original image retains the document in the original image, removes redundant parts that do not contain the document, and rotates the cropped original image by the angle of the document to correct the original image, thereby improving the text detection capability of the original image.

[0071] In other embodiments of this application, the above preprocessing process is an optional step. For ease of description, the processing of the preprocessed original image is basically the same as the processing of the original image in the following text. For ease of discussion, only the processing of the original image will be described in the following text.

[0072] In some embodiments of this application, the preset value can be a preset scaling ratio used when scaling the original image. The preset value includes a first preset value and a second preset value. The second preset value can be greater than the first preset value. The first and second preset values ​​can be set by the user, and this application does not limit them. For example, the first preset value can be 960, and the second preset value can be 1200.

[0073] In other embodiments of this application, the electronic device can identify smaller documents such as ID cards and driver's licenses as preset document types, and larger documents such as business licenses and operating permits as non-preset document types. Furthermore, the electronic device can establish a correspondence between a first preset value and preset document types, and a correspondence between a second preset value and non-preset document types. This allows the electronic device, after determining the document type of the original image, to directly compare the image size of the original image with the preset value corresponding to the document type, and scale the original image based on the comparison result and the preset value corresponding to the document type, without needing to simultaneously compare the image size of the original image with both the first and second preset values.

[0074] The preset document type may also include other types of documents. The above examples are only examples and do not constitute a limitation on the preset document type. Non-preset document types are other document types besides the preset document types.

[0075] In some embodiments of this application, the step of scaling the original image by the electronic device based on the comparison result and the preset value corresponding to the document type can be referred to steps S144-S145 below.

[0076] In this embodiment, by scaling the original image according to preset values ​​and / or document type, the image size can be flexibly adjusted, thereby ensuring that the scaled image meets the size requirements of the text detection model and avoiding the problem of being unable to extract document information due to the document image size not meeting the model extraction requirements.

[0077] S15, use a preset text detection model to detect the scaled image and obtain the coordinates of the first text box corresponding to the scaled image.

[0078] In some embodiments of this application, the text detection model includes, but is not limited to, the Mask R-CNN model and the DBNet model, etc. This application does not limit the text detection model. A text box refers to a text box that includes text information in a scaled image. The shape of the text box can be a rectangle, a polygon, or other shapes. The number of text boxes can be one or more. For example, a text box can be the smallest bounding rectangle corresponding to each line of text information in the scaled image.

[0079] In some embodiments of this application, the first text box coordinates refer to the coordinates of the text box. Each text box may have one or more first text box coordinates, and each first text box coordinate includes an abscissa and a ordinate. For example, if the text box is the smallest bounding rectangle corresponding to each line of text information in a scaled image, the first text box coordinates can be the coordinates of the four vertices (pixels) of the smallest bounding rectangle, or the coordinates of the two vertices (pixels) on the diagonal of the smallest bounding rectangle.

[0080] In this embodiment, the method for detecting scaled images using a text detection model can refer to the methods for object detection in related technologies.

[0081] S16, adjust the coordinates of the first text box to obtain the coordinates of the second text box, and perform a screenshot of the original image based on the coordinates of the second text box to obtain a partial text image.

[0082] In some embodiments of this application, the electronic device adjusts the coordinates of the first text box to obtain the coordinates of the second text box by: the electronic device inversely scaling the coordinates of the first text box to obtain the inversely scaled text box coordinates, and adjusting the inversely scaled text box coordinates using a preset adjustment strategy to obtain the coordinates of the second text box.

[0083] The inverse scaling process is the reverse of the scaling process described above, and will not be described further in this embodiment.

[0084] In some embodiments of this application, since the first text box coordinates of each text box may include one or more, the second text box coordinates of each text box may also include one or more.

[0085] In this embodiment, the coordinates of the first text box are inversely scaled so that the inversely scaled coordinates can be restored to the coordinates in the original image or the preprocessed original image. Since redundant parts that do not contain text information are removed from the local text image and the detection area is reduced, the interference of text layout problems can be reduced, and the text detection speed and accuracy of the local text image can be improved.

[0086] S17, input the local text image into the preset text recognition model to obtain the text information in the local text image.

[0087] In some embodiments of this application, the text recognition model includes, but is not limited to, a convolutional recurrent neural network (CRNN) model and a natural scene text detection and recognition model (AttentionOptical character recognition, AttentionOCR). This application does not limit the text recognition model.

[0088] In some embodiments of this application, if the text recognition model is a CRNN model, the CRNN model includes a convolutional neural network layer, a recurrent neural network layer, a linear layer, a transcription layer, and a decoding layer. The electronic device inputs a local text image into a preset text recognition model to obtain text information from the local text image, including: the electronic device using a convolutional neural network layer to extract features from the local text image to obtain a first feature sequence; further, the electronic device using a recurrent neural network layer to learn the first feature sequence to obtain a second feature sequence corresponding to the first feature sequence, the second feature sequence including contextual information between multiple features in the first feature sequence; calling a linear layer to perform a fully connected operation on the second feature sequence to obtain a feature vector; calling a transcription layer to map the feature vector to a character sequence of text; and calling a decoding layer to decode the character sequence to obtain the text information.

[0089] The convolutional neural network layer includes multiple convolutional layers and pooling layers, while the recurrent neural network layer can be a bi-long short-term memory network (Bi-LSTM). The transcription layer can use the connectionist temporal classification (CTC) loss function, and the decoding layer can use a beamsearch algorithm based on a language model.

[0090] S18: Input the original image, the document type of the original image, the coordinates of the second text box, and the text information into the preset information extraction model to obtain the field type of the text information.

[0091] In some embodiments of this application, the information extraction model may include a position embedding layer, a self-attention model (transformer), a normalization layer (LayerNorm), an average layer, a convolutional neural network (CNN) layer, and a linear layer. The linear layer may be replaced by a fully connected layer or an equivalent fully connected layer.

[0092] In some embodiments of this application, field types are used to distinguish different text information and to indicate the information type of the text information. For example, in "ID number: 44abc678900007m", 44abc678900007m is text information, and ID number is the field type corresponding to the text information.

[0093] In this embodiment, the accuracy of field types is improved by fusing and extracting multiple information, including the coordinates of the second text box, document type, and text information. Filtering text information based on field type and document type removes erroneous information, reduces reliance on the accuracy and completeness of the identified information, and thus improves the accuracy of the filtered document information.

[0094] S19. Based on the field type and document type, the text information is filtered to obtain the document information in the original image.

[0095] In some embodiments of this application, the document information can be structured information, which refers to information that can be decomposed into multiple interconnected pieces of information with a clear hierarchical structure. The electronic device can perform multiple filters on the text information according to the field type and document type to remove incorrect information from the text information, and verify the filtered text information to determine whether the filtered text information is correct.

[0096] Through the above implementation methods, it is possible to accurately determine whether the original image is an ID image based on the document type information. When the original image is not an ID image, no information extraction is performed on the original image, thus saving time. When the original image is determined to be an ID image, it is scaled based on a preset value and / or the document type of the original image. This allows for flexible adjustment of the image size, ensuring that the scaled image meets the size requirements of the text detection model and avoiding the problem of not being able to extract document information due to the size of the document image not meeting the model's extraction requirements. This improves the detection accuracy of the text detection model. Based on the coordinates of the second text box, the position of the text information in the original image can be accurately determined. By taking a screenshot based on the coordinates of the second text box, a local text image containing the text information can be accurately obtained, reducing interference from text layout issues. Since the local text image obtained by the screenshot process removes redundant parts, using a text recognition model to recognize the local text image can improve the recognition accuracy and speed of the text information. By inputting the original image, document type, second text box coordinates, and text information into the information extraction model, the accuracy of the field type can be improved due to the use of multiple information fusion extraction methods. Filtering text information based on field type and document type can remove erroneous information, reduce reliance on the accuracy and completeness of identification information, thereby effectively extracting document information and improving the accuracy of the filtered document information.

[0097] In some embodiments of this application, the original image is scaled based on its original image size and a preset value to obtain a scaled image. For example... Figure 4 The diagram shown is a flowchart of a method for generating a scaled image according to another embodiment of this application, including the following steps:

[0098] S141, determine the image size of the original image, where the image size includes the image width and the image height.

[0099] In some embodiments of this application, the image width can be the number of pixels in a row (horizontal direction) of the original image, and the image height can be the number of pixels in a column (vertical direction) of the original image. For example, if the number of pixels in a row (horizontal direction) of the original image is 640 and the number of pixels in a column (vertical direction) is 360, the image width of the original image is 640 and the image height of the original image is 360.

[0100] S142, determine the larger and smaller values ​​between the image width and the image height, and calculate the first target value based on the image width and the image height.

[0101] In some embodiments of this application, the image width and image height are typically not the same (image width is not equal to image height). If the image width is greater than the image height, the larger value is the image width and the smaller value is the image height; or, if the image width is less than the image height, the larger value is the image height and the smaller value is the image width.

[0102] In some embodiments of this application, the electronic device may determine the ratio between the image width and the image height as a first target value, or the electronic device may determine the ratio between the image height and the image width as a first target value.

[0103] S143, compare the larger value with the first preset value, and determine whether the larger value is less than the first preset value.

[0104] In some embodiments of this application, the first preset value can be set independently, and this application does not impose any restrictions on it. For example, the first preset value can be 960.

[0105] In this embodiment, if the larger value is less than the first preset value, the electronic device executes step S144; or, if the larger value is greater than or equal to the first preset value, the electronic device executes step S145.

[0106] In other embodiments of this application, when the image width and image height are equal, or when both the image width and image height are less than a first preset value, the electronic device adjusts the image width and image height to the first preset value.

[0107] S144, the larger value is scaled according to the first preset value, the second target value is calculated according to the first target value and the first preset value, and the smaller value is scaled according to the second target value to obtain a scaled image.

[0108] In some embodiments of this application, when the larger value is the image width and the first target value is the ratio between the image width and the image height, and when the larger value is the image height and the first target value is the ratio between the image height and the image width, the electronic device can determine the ratio between the first preset value and the first target value as the second target value. Alternatively, when the larger value is the image height and the first target value is the ratio between the image width and the image height, and when the larger value is the image width and the first target value is the ratio between the image height and the image width, the electronic device calculates the product between the first preset value and the first target value to determine the second target value.

[0109] In some embodiments of this application, the electronic device adjusts a larger value to a first preset value and a smaller value to a second target value to obtain a scaled image.

[0110] For example, if the maximum value of the original image is the image width of 640, the minimum value of the original image is the image height of 320, the first target value is the image width of 640 / the image height of 320 = 2, the first preset value is 960, and the second preset value is 1200. Since the maximum value of 640 is less than the first preset value of 960, the second target value is 480 = the first preset value of 960 / the first target value of 2. The electronic device adjusts the image width in the original image from 640 to 960 and the image height in the original image from 320 to 480 to obtain a scaled image.

[0111] S145, determine whether the larger value is equal to the first preset value.

[0112] In this embodiment, if the larger value is equal to the first preset value, the electronic device executes step S146; or, if the larger value is greater than the first preset value, the electronic device executes step S147.

[0113] S146, Do not adjust the image width and image height of the original image.

[0114] In this embodiment, since the adjustment of the image width and image height of the original image is based on the larger value between the first preset value and the first preset value, there is no need to adjust the image width and image height of the original image when the larger value is equal to the first preset value.

[0115] S147, determine whether the larger value is equal to the second preset value.

[0116] In this embodiment, if the larger value is equal to the second preset value, the electronic device executes step S146; or, if the larger value is greater than the first preset value, the electronic device executes step S148.

[0117] S148, the larger value is scaled according to the second preset value, the second target value is calculated according to the first target value and the second preset value, and the smaller value is scaled according to the second target value to obtain a scaled image.

[0118] In some embodiments of this application, the second preset value may be greater than the first preset value, and the second preset value can be set by the user; this application does not impose any restrictions on this. For example, when the first preset value is 960, the second preset value may be 1200.

[0119] In some embodiments of this application, when the larger value is the image width and the first target value is the ratio between the image width and the image height, and when the larger value is the image height and the first target value is the ratio between the image height and the image width, the electronic device can determine the ratio between the second preset value and the first target value as the second target value. Alternatively, when the larger value is the image height and the first target value is the ratio between the image width and the image height, and when the larger value is the image width and the first target value is the ratio between the image height and the image width, the electronic device calculates the product between the second preset value and the first target value to determine the second target value.

[0120] In some embodiments of this application, the electronic device adjusts a larger value to a second preset value and a smaller value to a second target value to obtain a scaled image.

[0121] For example, if the larger value of the original image is the image width of 1000, the smaller value of the original image is the image height of 720, the first preset value is 960, the second preset value is 1200, and the first target value is image width 1000 / image height 720 = 25 / 18, and the image width of 1000 is greater than the first preset value of 960, then the second target value is 864 = second preset value 1200 / first target value 25 / 18. The electronic device adjusts the image width in the original image from 1000 to 1200 and the image height in the original image from 720 to 864 to obtain a scaled image.

[0122] In this embodiment, adaptive scaling based on the original image size allows for flexible adjustment of the original image's size to suit the input requirements of different models. Furthermore, since the first target value is the original ratio between the image height and width of the original image, this embodiment scales a smaller value based on this original ratio, ensuring that the ratio between the image height and width of the scaled image remains the same as the original ratio, thereby ensuring the accuracy of the scaled image.

[0123] In other embodiments of this application, if the image width is equal to the image height, and both the image width and image height are greater than a first preset value, or if the image width is equal to the image height, and both the image width and image height are greater than a second preset value, the electronic device adjusts the image width and image height to the second preset value.

[0124] In some embodiments of this application, when the text box is rectangular, the inverse scaling coordinates of the text box are multiple, such as... Figure 5 The diagram shown is a flowchart of a method for generating the coordinates of a second text box according to an embodiment of this application, including the following steps:

[0125] S161, calculate the text box width and text box height based on the coordinates of multiple inverse scaling text boxes of the text box.

[0126] In some embodiments of this application, if the text box is a rectangle, and the coordinates of multiple inversely scaled text boxes are the coordinates of four vertices (top left vertex, bottom left vertex, top right vertex, and bottom right vertex) in the rectangle, and the coordinates of each vertex include a vertical coordinate and a horizontal coordinate, the electronic device can determine the difference between the horizontal coordinate of the top right vertex and the horizontal coordinate of the top left vertex as the width of the text box, and determine the difference between the vertical coordinate of the bottom right vertex and the vertical coordinate of the bottom left vertex as the height of the text box.

[0127] In other embodiments of this application, if the text box has a different shape, the width and height of the text box can be calculated in other ways.

[0128] S162, compare the width of the text box with the height of the text box to determine whether the width of the text box is greater than the height of the text box.

[0129] In this embodiment, if the width of the text box is greater than the height of the text box, the electronic device executes step S163; if the width of the text box is less than or equal to the height of the text box, the electronic device executes step S166.

[0130] S163, determine whether the document type of the original image belongs to the preset document type.

[0131] In some embodiments of this application, if the width of the text box is greater than the height of the text box, and the document type of the original image belongs to a preset document type, the electronic device executes step S164; or, if the width of the text box is greater than the height of the text box, and the document type of the original image does not belong to a preset document type (the document type of the original image belongs to a non-preset document type), the electronic device executes step S165.

[0132] In this embodiment, the electronic device can identify smaller documents such as ID cards and driver's licenses as preset document types, and larger documents such as business licenses and operating permits as non-preset document types. The preset document types may also include other types of documents. The examples above are merely illustrations and do not constitute a limitation on the preset document types. Non-preset document types refer to any document types other than the preset document types.

[0133] S164, calculate the target width based on the first preset configuration value and the text box height, and calculate the target height based on the second preset configuration value and the text box height.

[0134] In some embodiments of this application, the first preset configuration value and the second preset configuration value can be set independently, and this application does not impose any restrictions on this. For example, the first preset configuration value can be 1 / 4, and the second preset configuration value can be 1 / 40.

[0135] In some embodiments of this application, the electronic device may determine the target width by multiplying the first preset configuration value and the height of the text box, and determine the target height by multiplying the second preset configuration value and the height of the text box.

[0136] In other embodiments of this application, other methods may also be used, such as calculating the target width by subtracting, adding, or dividing the first preset configuration value from the text box height, and calculating the target height by subtracting, adding, or dividing the second preset configuration value from the text box height. This application does not limit the calculation method.

[0137] S165, calculate the target width based on the third preset configuration value and the text box height, and calculate the target height based on the fourth preset configuration value and the text box height.

[0138] In some embodiments of this application, the third and fourth preset configuration values ​​can be set independently, and this application does not impose any restrictions on them. For example, the third preset configuration value can be 1 / 8, and the fourth preset configuration value can be 1 / 10.

[0139] In some embodiments of this application, the electronic device may determine the target width by multiplying the third preset configuration value by the height of the text box, and determine the target height by multiplying the fourth preset configuration value by the height of the text box.

[0140] S166, determine whether the document type of the original image belongs to the preset document type.

[0141] If the width of the text box is less than or equal to the height of the text box, and the document type of the original image belongs to the preset document type, the electronic device executes step S167. Alternatively, if the width of the text box is less than or equal to the height of the text box, and the document type of the original image does not belong to the preset document type (the document type of the original image belongs to a non-preset document type), the electronic device executes step S168.

[0142] In this embodiment, smaller documents such as ID cards and driver's licenses can be designated as preset document types, while larger documents such as business licenses and operating permits can be designated as non-preset document types. The preset document types may also include other types of documents. The examples above are merely illustrations and do not constitute a limitation on the preset document types. Non-preset document types refer to any document types other than the preset document types.

[0143] S167, calculate the target height based on the first preset configuration value and the width of the text box, and calculate the target width based on the second preset configuration value and the width of the text box.

[0144] In some embodiments of this application, the electronic device may determine the target height by multiplying the first preset configuration value by the width of the text box, and determine the target width by multiplying the second preset configuration value by the width of the text box.

[0145] S168, calculate the target height based on the third preset configuration value and the width of the text box, and calculate the target width based on the fourth preset configuration value and the width of the text box.

[0146] In some embodiments of this application, the electronic device may determine the target height by multiplying the third preset configuration value by the width of the text box, and determine the target width by multiplying the fourth preset configuration value by the width of the text box.

[0147] S169, adjust the width of the text box according to the target width, adjust the height of the text box according to the target height, obtain the target text box, and determine the coordinates of each inverse scaling text box in the target text box as the coordinates of the second text box.

[0148] In some embodiments of this application, the electronic device adjusts the width of the text box to the target width and the height of the text box to the target height to obtain the target text box. If the target text box is rectangular, the electronic device can determine the coordinates of the four vertices of the target text box as the coordinates of the second text box, or it can determine the coordinates of the two vertices on the diagonal of the target text box as the coordinates of the second text box.

[0149] In this embodiment, since the coordinates of the second text box are used to perform screenshot processing on the original image, the coordinates of the second text box are obtained by adjusting the coordinates of the first text box. This allows for flexible adjustment of the size of the screenshotted local text image based on the size of the original image, making the local text image more accurate.

[0150] In some embodiments of this application, the information extraction model may include a location embedding layer, a self-attention model, a normalization layer, an averaging layer, a convolutional neural network layer, and a linear layer. For example... Figure 6 The diagram shown is a flowchart of a field type generation method provided in an embodiment of this application, including the following steps:

[0151] S181, the document type and text information are encoded and fused to obtain fused information, and the fused information is input into the self-attention model to obtain semantic feature information.

[0152] In some embodiments of this application, the electronic device can encode the document type and text information into vectors, and concatenate the vector corresponding to the document type and the vector corresponding to the text information to obtain fused information. Alternatively, the electronic device can directly combine the document type and text information to obtain fused information.

[0153] In some embodiments of this application, the self-attention model includes an encoder capable of learning contextual information from all layers within the self-attention model. The process by which the attention mechanism layer network processes the fused information to obtain semantic feature information can be referenced from related technologies regarding the process of calling the encoder for mapping.

[0154] S182, use the position embedding layer to encode the coordinates of the second text box to obtain position encoding information.

[0155] In some embodiments of this application, the electronic device calls a location embedding layer to normalize the coordinates of the second text box, and calculates a location index based on the normalized second text box coordinates and the dimensions of a preset location embedding matrix. The electronic device then selects the corresponding location information from the location embedding matrix as location encoding information based on the location index. The dimensions of the location embedding matrix include the maximum length of the sequence. The electronic device calculates the location index based on the normalized second text box coordinates and the maximum length of the sequence. For example, if the normalized second text box coordinates are (0.5, 0.6) and the maximum length of the sequence is 100, then the location index (49.5, 59.4) = (0.5, 0.6) * (100 - 1).

[0156] S183 uses a normalization layer to normalize the semantic feature information and the positional encoding information to obtain the word embedding vector (textualembedding). The word embedding vector is then input into the self-attention model to obtain the textual context features.

[0157] In some embodiments of this application, the process of the self-attention model processing word embedding vectors to obtain text context feature information can be referred to in step S181.

[0158] S184: The original image is cropped using the coordinates of the second text box to obtain a cropped image. A convolutional neural network layer is then used to extract features from the cropped image to obtain visual contextual feature information.

[0159] In some embodiments of this application, the convolutional neural network layer can be ResNet34, ResNet50, etc., and this application does not limit the convolutional neural network layer.

[0160] In this embodiment, the electronic device locates and extracts the text contained in the original image based on the coordinates of the second text box, thereby removing non-text information from the screenshot image and making the screenshot image more accurate.

[0161] S185 invokes the average layer, linear layer, and self-attention model to generate field types based on visual and textual context information.

[0162] In some embodiments of this application, the electronic device invokes an averaging layer, a linear layer, and a self-attention model to generate field types based on visual context information and text context information. This includes: the electronic device invokes an averaging layer to average the visual context information to obtain average visual context information, and invokes a linear layer to perform fully connected operations on the average visual context information and the text context information to obtain visual feature information of the average visual context information and text feature information of the text context information. Further, the electronic device fuses the text feature information and visual feature information to obtain multimodal context feature information, and uses a self-attention model to aggregate the multimodal context feature information to obtain field types.

[0163] In this embodiment, both visual context information and textual context information can be multiple. The averaging layer is called to average the visual context information to obtain the average visual context information. This involves summing all the visual context information and dividing the sum by the total number of all visual context information. This reduces the number of parameters in the information extraction model and decreases the risk of overfitting.

[0164] The fully connected operation on the average visual context information and textual context information includes: mapping the average visual context information and textual context information to the same dimension, and then performing a weighted operation on the mapped average visual context information and textual context information to obtain multimodal contextual feature information. This process is called adaptive multimodal contextual feature fusion. The weights of the weighting operation can be set by the user, and this application does not impose any restrictions on this. Feature fusion can improve the accuracy of field types.

[0165] In some embodiments of this application, such as Figure 7 The diagram shown is a flowchart of an information extraction model provided in one embodiment of this application. Figure 7 As can be seen, the information extraction model includes a location embedding layer, a self-attention model, a normalization layer, an averaging layer, a convolutional neural network layer, and a linear layer. Appropriate processing can be performed using the corresponding network layers. Figure 7 In this context, α and β are the weights used when performing weighted operations on the linear layer.

[0166] In some embodiments of this application, such as Figure 8 The diagram shown is a flowchart of a method for generating document information according to an embodiment of this application, including the following steps:

[0167] S191, classify and extract text information according to field type to obtain structured information.

[0168] In some embodiments of this application, an electronic device can select text information corresponding to field types from text information as structured information.

[0169] S192, filter the structured information according to the document type to obtain the filtered structured information.

[0170] In some embodiments of this application, the electronic device filters out information that does not belong to the document type from the structured information and retains the information that does belong to the document type.

[0171] For example, if the document type is an ID card, but the structured information includes text information such as a business license, then the text information such as the business license in the structured information needs to be deleted.

[0172] S193, verify the target information in the filtered structured information according to preset rules.

[0173] In some embodiments of this application, the target information can be set independently, and this application does not impose any restrictions on it. For example, when the original image is an ID card image, the target information can be the ID card number; or, when the original image is a business license image, the target information can be the business license number.

[0174] In some embodiments of this application, the preset rules are verification rules from related technologies. For example, if the target information is an ID card number or a business license number, the preset rules can be formulas or rules for verifying ID card numbers and business license numbers. Based on the preset rules, it can be determined whether the ID card number and business license number are normal (e.g., whether the number of digits meets the requirements).

[0175] S194, determine whether the target information has passed the verification.

[0176] In some embodiments of this application, the electronic device determines that the target information verification passes when the target information conforms to preset rules, or determines that the target information verification fails when the target information does not conform to preset rules. In this embodiment, if the target information verification passes, the electronic device executes step S195, or if the target information verification fails, the electronic device executes step S196.

[0177] S195, the filtered structured information is used as document information.

[0178] In this embodiment, the electronic device will only use the filtered structured information as document information when the target information passes the verification, thus improving the accuracy of the document information.

[0179] S196 outputs a message indicating a verification error.

[0180] In some embodiments of this application, the above-mentioned prompt information can be set independently, and this application does not impose any restrictions on it. For example, when the target information is an ID card number, the prompt information could be: "ID card number is abnormal, please confirm."

[0181] In this embodiment, when the target information fails verification, the electronic device will not use the filtered structured information as the document information, but will instead provide a prompt message to remind the user to check in a timely manner, thereby quickly identifying the problem and extracting the correct document information.

[0182] like Figure 9 The diagram shown is a functional block diagram of a document information extraction device according to an embodiment of this application. The document information extraction device 11 includes an acquisition unit 110, a classification unit 111, a scaling unit 112, a detection unit 113, an adjustment unit 114, a screenshot unit 115, an input unit 116, and a filtering unit 117. The module / unit referred to in this application refers to a module / unit capable of being... Figure 1 The processor 103 in the middle acquires a series of computer-readable instruction segments that are capable of performing a fixed function, and these segments are stored in Figure 1 The memory 102 is used for this purpose. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0183] Acquisition unit 110 is used to acquire the original image.

[0184] The classification unit 111 is used to classify the original image using a preset classification model to obtain classification information of the original image. The classification information includes information such as document type information, which indicates whether the original image is a document image and the document type of the original image when the original image is a document image.

[0185] The scaling unit 112 is used to scale the original image based on the image size of the original image and a preset value if the original image is determined to be an ID image according to the ID type information, so as to obtain a scaled image.

[0186] The detection unit 113 is used to detect the scaled image using a preset text detection model to obtain the coordinates of the first text box corresponding to the scaled image.

[0187] The adjustment unit 114 is used to adjust the coordinates of the first text box to obtain the coordinates of the second text box.

[0188] The screenshot unit 115 is used to perform screenshot processing on the original image according to the coordinates of the second text box to obtain a partial text image;

[0189] The input unit 116 is used to input a local text image into a preset text recognition model to obtain text information in the local text image.

[0190] The input unit 116 is also used to input the original image, document type, second text box coordinates and text information into a preset information extraction model to obtain the field type of the text information.

[0191] The filtering unit 117 is used to filter text information according to field type and document type to obtain document information in the original image.

[0192] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can refer to the methods in the above embodiments of this application.

[0193] The computer-readable storage medium can be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. Alternatively, the computer-readable storage medium can be an external storage device of the electronic device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0194] In some embodiments, a computer-readable storage medium may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the electronic device, etc.

[0195] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0196] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for extracting document information, characterized in that, The method includes: The original image is acquired and classified using a preset classification model to obtain classification information of the original image. The classification information includes document type information, which is used to indicate whether the original image is a document image and, when the original image is a document image, the document type of the original image. If the original image is determined to be an ID image based on the ID type information, the original image is scaled based on its image size and a preset value to obtain a scaled image. This includes: determining the larger and smaller values ​​between the image width and image height of the original image; calculating a first target value based on the image width and image height; if the larger value is less than a first preset value, scaling the larger value based on the first preset value; calculating a second target value based on the first target value and the first preset value; and scaling the smaller value based on the second target value to obtain the scaled image; or if the larger value is greater than the first preset value, scaling the larger value based on the second preset value; calculating a second target value based on the first target value and the second preset value; and scaling the smaller value based on the second target value to obtain the scaled image. The scaled image is detected using a preset text detection model to obtain the coordinates of the first text box corresponding to the scaled image. The coordinates of the first text box are adjusted to obtain the coordinates of the second text box, and the original image is cropped based on the coordinates of the second text box to obtain a partial text image; The local text image is input into a preset text recognition model to obtain the text information in the local text image; The original image, the document type, the coordinates of the second text box, and the text information are input into a preset information extraction model to obtain the field type of the text information. The information extraction model includes a position embedding layer, a self-attention model, a normalization layer, an averaging layer, a convolutional neural network layer, and a linear layer. The process includes: encoding and fusing the document type and the text information to obtain fused information; inputting the fused information into the self-attention model to obtain semantic feature information; encoding the coordinates of the second text box using the position embedding layer to obtain position encoding information; normalizing the semantic feature information and the position encoding information using the normalization layer to obtain a word embedding vector; inputting the word embedding vector into the self-attention model to obtain text context feature information; taking a screenshot of the original image using the coordinates of the second text box to obtain a screenshot image; extracting features from the screenshot image using the convolutional neural network layer to obtain visual context feature information; and calling the averaging layer, the linear layer, and the self-attention model to generate the field type based on the visual context information and the text context information. Based on the field type and the document type, the text information is filtered to obtain the document information in the original image.

2. The document information extraction method as described in claim 1, characterized in that, The step of adjusting the coordinates of the first text box to obtain the coordinates of the second text box includes: The coordinates of the first text box are inversely scaled to obtain the inversely scaled text box coordinates. The coordinates of the inverse scaling text box are adjusted using a preset adjustment strategy to obtain the coordinates of the second text box.

3. The document information extraction method as described in claim 2, characterized in that, When the text box is rectangular, there are multiple inverse scaling text box coordinates. Adjusting the inverse scaling text box coordinates using a preset adjustment strategy to obtain the second text box coordinates includes: The text box width and height are calculated based on the coordinates of the multiple inversely scaled text boxes of the text box; If the width of the text box is greater than the height of the text box, and the document type belongs to a preset document type, calculate the target width based on the first preset configuration value and the height of the text box, and calculate the target height based on the second preset configuration value and the height of the text box; or If the width of the text box is greater than the height of the text box, and the document type does not belong to the preset document type, calculate the target width based on the third preset configuration value and the height of the text box, and calculate the target height based on the fourth preset configuration value and the height of the text box; or If the width of the text box is less than or equal to the height of the text box, and the document type belongs to the preset document type, calculate the target height based on the first preset configuration value and the width of the text box, and calculate the target width based on the second preset configuration value and the width of the text box; or If the width of the text box is less than or equal to the height of the text box, and the document type does not belong to the preset document type, calculate the target height based on the third preset configuration value and the width of the text box, and calculate the target width based on the fourth preset configuration value and the width of the text box; Adjust the width of the text box according to the target width, adjust the height of the text box according to the target height, and obtain the target text box of the text box. Then, determine the coordinates of each inverse scaling text box as the coordinates of the second text box in the target text box.

4. The document information extraction method as described in claim 1, characterized in that, The step of calling the averaging layer, the linear layer, and the self-attention model to generate the field type based on the visual context information and the text context information includes: The averaging layer is invoked to average the visual context information to obtain average visual context information. The linear layer is invoked to perform fully connected operations on the average visual context information and the text context information respectively to obtain the visual feature information of the average visual context information and the text feature information of the text context information. The textual and visual feature information is fused to obtain multimodal contextual feature information, and the self-attention model is used to aggregate the multimodal contextual feature information to obtain the field type.

5. The document information extraction method as described in claim 1, characterized in that, The step of filtering the text information according to the field type and the document type to obtain the document information in the original image includes: The text information is categorized and extracted according to the field type to obtain structured information; The structured information is filtered according to the document type to obtain filtered structured information. The target information in the filtered structured information is verified according to preset rules. When the target information is successfully verified, the filtered structured information is used as the document information.

6. A document information extraction device, operating in an electronic device, the device being used to implement the document information extraction method as described in any one of claims 1 to 5, characterized in that, The device includes: The acquisition unit is used to acquire the original image; A classification unit is used to classify the original image using a preset classification model to obtain classification information of the original image. The classification information includes document type information, which indicates whether the original image is a document image and, when the original image is a document image, the document type of the original image. A scaling unit is used to scale the original image based on the image size of the original image and a preset value if the original image is determined to be an ID image according to the ID type information, so as to obtain a scaled image. The detection unit is used to detect the scaled image using a preset text detection model to obtain the coordinates of the first text box corresponding to the scaled image. An adjustment unit is used to adjust the coordinates of the first text box to obtain the coordinates of the second text box; The screenshot unit is used to perform screenshot processing on the original image according to the coordinates of the second text box to obtain a partial text image; An input unit is used to input the local text image into a preset text recognition model to obtain the text information in the local text image; The input unit is further configured to input the original image, the document type, the coordinates of the second text box, and the text information into a preset information extraction model to obtain the field type of the text information; The filtering unit is used to filter the text information according to the field type and the document type to obtain the document information in the original image.

7. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the document information extraction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which, when executed by a processor in an electronic device, implements the document information extraction method as described in any one of claims 1 to 5.

Images