A key information extraction method and device, a storage medium and an electronic device

Abstract

The present disclosure relates to a key information extraction method, device, storage medium and electronic equipment, and belongs to the technical field of natural language processing. The method comprises the following steps: extracting visual features and text semantic features of a plurality of initial text boxes in a to-be-processed image; obtaining node features of the plurality of initial text boxes according to the visual features and the text semantic features; clustering the node features of the plurality of initial text boxes based on a graph convolution model to obtain categories corresponding to the plurality of initial text boxes respectively, wherein the graph convolution model adopts spectral domain convolution for clustering; and determining a target text box having necessary key information from the plurality of initial text boxes according to the categories corresponding to the plurality of initial text boxes respectively. By using the key information extraction method proposed in the present disclosure, necessary key information can be extracted from the to-be-processed image, so that the staff only needs to audit the necessary key information, and does not need to audit irrelevant information, thereby improving the auditing efficiency of the staff.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a key information extraction method, device, storage medium, and electronic device. Background Art

[0002] In recent years, with the rapid development of the knowledge economy, the education market has shown a good growth trend. At the same time, it has also brought greater work pressure to teachers in major schools.

[0003] For example, when universities and colleges are recruiting teachers, they need recruiters to review applicants' teaching certificates, qualification certificates and other personal documents. However, when there are a large number of applicants, the number of personal documents such as teaching certificates and qualification certificates that recruiters need to review will be large. Manually reviewing a large number of personal documents is time-consuming and labor-intensive, and there is a problem of low efficiency. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a key information extraction method, device, storage medium and electronic device to solve the above technical problems.

[0005] In order to achieve the above-mentioned object, a first aspect of an embodiment of the present disclosure provides a key information extraction method, the method comprising:

[0006] Extracting visual features and text semantic features of multiple initial text boxes in the image to be processed;

[0007] Obtaining node features of multiple initial text boxes based on the visual features and text semantic features;

[0008] Based on a graph convolution model, clustering the node features of the multiple initial text boxes to obtain categories corresponding to the multiple initial text boxes, wherein the graph convolution model uses spectral domain convolution for clustering;

[0009] According to the categories corresponding to the multiple initial text boxes, a target text box having necessary key information is determined from the multiple initial text boxes.

[0010] Optionally, extracting visual features of a plurality of initial text boxes in the image to be processed includes:

[0011] Based on the HRNet model, visual features and text semantic features of the multiple initial text boxes are extracted.

[0012] Optionally, before extracting the visual features and text semantic features of the initial text box in the image to be processed, the method includes:

[0013] Identifying the image to be processed, and obtaining a plurality of text boxes of the image to be processed;

[0014] Labeling text boxes with key information among the multiple text boxes to obtain an initial text box with a first preset label;

[0015] Based on the graph convolutional model, an initial text box with the first preset label is extracted.

[0016] Optionally, the graph convolution model is trained by the following steps:

[0017] The initial model is trained using Chinese and English as training samples to obtain the graph convolutional model.

[0018] Optionally, the image to be processed is obtained by the following steps:

[0019] enhancing the image quality of the input image based on a generative adversarial network to obtain a first image;

[0020] Correcting the first image to obtain the image to be processed.

[0021] Optionally, correcting the first image to obtain the image to be processed includes:

[0022] The background area of ​​the first image is removed to obtain the image to be processed.

[0023] Optionally, correcting the first image to obtain the image to be processed includes:

[0024] Fitting multiple vertices of the first image according to multiple edges of the first image to obtain multiple vertices of the first image;

[0025] The image to be processed is obtained according to a plurality of vertices of the first image and a plurality of edges of the first image.

[0026] According to a second aspect of an embodiment of the present disclosure, a key information extraction device is provided, the device comprising:

[0027] An extraction module, used to extract visual features and text semantic features of multiple initial text boxes in the image to be processed;

[0028] A node feature determination module, configured to obtain node features of a plurality of initial text frames based on the visual features and text semantic features;

[0029] A clustering module, configured to cluster the node features of the multiple initial text boxes based on a graph convolution model to obtain categories corresponding to the multiple initial text boxes, wherein the graph convolution model uses spectral domain convolution for clustering;

[0030] The key information extraction module is used to determine a target text box with necessary key information from multiple initial text boxes according to the type of the image to be processed.

[0031] According to the third aspect of the embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the key information extraction method provided in any one of the first aspects of the embodiment of the present disclosure are implemented.

[0032] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0033] a memory having a computer program stored thereon;

[0034] A processor is used to execute the computer program in the memory to implement the steps of the key information extraction method provided in any one of the first aspects of the embodiments of the present disclosure.

[0035] Through the above technical solution, a target text box is obtained from multiple initial text boxes, so that the key information in the obtained target text box is necessary key information, and the necessary key information is necessary key information related to confirming the card holder and the type of certificate. Therefore, the staff only needs to review the necessary key information to determine the card type and holder information of the image to be processed, without having to review information that is not related to the card type and holder information, thereby improving the review efficiency of the staff.

[0036] In addition, the convolution model can use spectral domain convolution for clustering. Since spectral domain convolution can predict the category of the initial text box to be predicted through the initial text boxes of any number of neighbors, rather than taking a fixed number of neighbors to predict the category of the initial text box, the category of the initial text box can be predicted through a smaller number of neighbors, which improves the speed of initial text box category prediction. It can also predict the category of the initial text box through a larger number of neighbors, which improves the accuracy of initial text box category prediction.

[0037] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0039] Figure 1 is a flowchart of a key information extraction method according to an exemplary embodiment of the present disclosure;

[0040] Figure 2is a schematic diagram of one type of teacher qualification certificate shown in an exemplary embodiment of the present disclosure;

[0041] Figure 3 is a schematic diagram of an initial text box with key information in a teacher qualification certificate according to an exemplary embodiment of the present disclosure;

[0042] Figure 4 is a schematic diagram of a target text box with necessary key information according to an exemplary embodiment of the present disclosure;

[0043] Figure 5 is a schematic diagram of an HRNet network shown in an exemplary embodiment of the present disclosure;

[0044] Figure 6 is a schematic diagram of nodes and convolution kernels shown in an exemplary embodiment of the present disclosure;

[0045] Figure 7 is a schematic diagram illustrating an exemplary embodiment of the present disclosure for determining vertices of an image to be processed;

[0046] Figure 8 is a block diagram of a key information extraction device shown in an exemplary embodiment of the present disclosure;

[0047] Figure 9 FIG. 4 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0049] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0050] See also Figure 1 A key information extraction method is shown, which includes the following steps:

[0051] Step S11: extracting visual features and text semantic features of multiple initial text boxes in the image to be processed.

[0052] In this step, after image recognition is performed on the image to be processed, multiple text boxes of the image to be processed are obtained, and the initial text box refers to the initial text box with the first preset label among the multiple text boxes.

[0053] Specifically, the initial text box is obtained through the following steps:

[0054] Sub-step A1: Identify the image to be processed and obtain multiple text boxes of the image to be processed.

[0055] Among them, the image to be processed can be a card data image such as a teacher qualification certificate or a certificate of qualification. When identifying the image to be processed, OCR (Optical Character Recognition) can be used to detect and identify the text lines in the image to be processed, and provide the spatial position relationship and text recognition results of each text box in the image to be processed.

[0056] For example, after the teacher qualification certificate image to be processed is recognized, the following is obtained: Figure 2 The spatial position relationship and text recognition results of the text boxes such as "Certificate Holder", "Zhang San", "Gender", "Female", etc. are shown.

[0057] Sub-step A2: Label the text boxes containing key information among the multiple text boxes to obtain an initial text box with a first preset label.

[0058] Among them, the text recognition results of multiple text boxes can be key information or non-key information. Key information refers to information that can indicate the function of the image to be processed, and non-key information refers to information that cannot indicate the function of the image to be processed.

[0059] For example, Figure 2 As shown, the image to be processed is a teacher's qualification certificate. The text recognition results such as Zhang San, female, ethnicity X, and senior high school teacher's qualification certificate in the image to be processed can indicate that the image to be processed is Zhang San's senior high school teacher's qualification certificate. Therefore, these text recognition results are key information. However, the text recognition results such as according to the regulations of XXX, certificate holder, gender, and date of birth in the image to be processed are irrelevant to indicating that the image to be processed is Zhang San's teacher's qualification certificate. Therefore, these text recognition results are non-key information.

[0060] When labeling multiple text boxes, the text boxes with key information can be labeled with a first preset label, while the text boxes with non-key information can be labeled with a second preset label. In this way, an initial text box with the first preset label and a non-initial text box with the second preset label can be obtained.

[0061] Specifically, since the key information in each image to be processed is different, the first preset label of each text box with key information needs to be marked one by one; and the non-key information in each image to be processed is roughly the same, so the text box of non-key information can be marked with a second preset label based on the preset template.

[0062] For example, for multiple teacher qualification certificates, non-critical information such as "according to XXX regulations", certificate holder, gender, and date of birth are displayed in the same text information in each different teacher qualification certificate. Therefore, when annotating text boxes with non-critical information in the image to be processed of the teacher qualification certificate type, the text boxes with non-critical information can be annotated based on a preset template.

[0063] Among them, different preset templates are used to mark the text boxes of non-critical information in different types of images to be processed. For example, preset template A is used to mark the non-initial text boxes of non-critical information in the teacher's qualification certificate, preset template B is used to mark the non-initial text boxes of non-critical information in the architect's qualification certificate, and preset template C is used to mark the non-initial text boxes with non-critical information in the agent's qualification certificate.

[0064] Specifically, when a preset template is used for annotation, the preset template includes multiple preset text boxes, each of which contains its own non-critical information, and each of which is labeled with a second preset label. Therefore, the non-critical information on the preset template can be compared to determine whether it is consistent with the non-critical information on the image to be processed. If they are consistent, the second preset label of the preset text box on the preset template is labeled on the text box in the image to be processed, thereby resulting in an initial text box on the image to be processed with the second preset label.

[0065] Sub-step A3: Based on the graph convolution model, extract the initial text box with the first preset label.

[0066] After obtaining the initial text frame with the first preset label and the non-initial text frame with the second preset label, the non-initial text frame with the second preset label can be filtered out by the graph convolutional network (GCN model) to extract the initial text frame with the first preset label from the multiple text frames. Figure 2 Filter out the multiple initial text boxes shown as follows Figure 3 Multiple initial text boxes with first preset labels are shown.

[0067] Specifically, the key information in multiple training images and the first preset labels corresponding to the key information can be used as training samples to train the initial model; then the non-key information in the training images and the second preset labels corresponding to the non-key information can be used as training samples to train the initial model to obtain a graph convolution model.

[0068] After receiving an image to be processed, the trained graph convolutional model can identify initial text boxes corresponding to key information in the image to be processed based on the first preset label, and can also identify non-initial text boxes corresponding to non-key information in the image to be processed based on the second preset label. After distinguishing the initial text boxes from the non-initial text boxes in the image to be processed, the initial text boxes with the first preset label are extracted.

[0069] Among them, in the process of training the graph convolution model, it is not limited to inputting Chinese text information into the graph convolution model for training. English can also be used as training samples to train the initial model, so that the obtained graph convolution model can not only extract the initial text box of the image to be processed in Chinese, but also extract the initial text box of the image to be processed in English, thereby improving the versatility of the graph convolution model.

[0070] In this step, the visual features of the initial text box include the spatial location information of the initial text box, and the text semantic features of the initial text box include the semantics of the key information. The visual features and text semantic features of the initial text box can be extracted through the HRNet model:

[0071] In the related technology, the UNet model is used to output the feature map of the image to be processed; then the text area coordinates of the initial text box are mapped to each text area in the feature map to obtain the visual features and text semantic features of each initial text box.

[0072] Specifically, the UNet model is divided into an encoding part and a decoding part. The UNet model obtains the feature map of the image to be processed through the following steps: the encoding part downsamples the image to be processed. The initial input size of the image to be processed is 224x224. After four downsamplings, four images of different sizes, 112x112, 56x56, 28x28, and 14x14, are obtained; the decoding part is used for image splicing, convolution, and upsampling. For example, the 14x14 feature map is upsampled to obtain a 28x28 feature map, and then the 28x28 feature map is spliced ​​with the 28x28 feature map obtained by downsampling on the channel, and then the spliced ​​image is convolved and upsampled to obtain a 56x56 feature map, and then the 56x56 feature map is spliced, convolved, and upsampled. After four upsamplings, a feature map with the same size as the image to be processed can be output.

[0073] In this process, the UNet model has the following defects:

[0074] 1. The encoding part of the UNet model is used to obtain the context information of each text region in the image to be processed and determine the text semantic features of each text region based on the context information; the decoding part is used to locate the spatial position of each text region in the image to be processed to obtain the visual features of each text region.

[0075] The more encoding and decoding parts there are in the UNet model, the lower the accuracy of spatial positioning of the text area will be; the fewer encoding and decoding parts there are, the less contextual information is obtained and the lower the accuracy of the obtained text semantic features. It can be seen that the UNet model cannot obtain accurate visual features and text semantic features at the same time.

[0076] 2. During the downsampling and upsampling process of the UNet network, the spatial position information of the initial text box will be lost, resulting in a decrease in the ability to locate the initial text box.

[0077] The present disclosure is based on the HRNet model, outputs a feature map of the image to be processed, maps the text area coordinates of the initial text box to each text area in the feature map, and obtains the visual features and text semantic features of each initial text box; wherein, the feature map has different visual features.

[0078] Specifically, see Figure 5 As shown, Figure 5 Each block in represents each feature map. The larger the size of the feature map, the higher the resolution, and the smaller the size of the feature map, the lower the resolution.

[0079] from Figure 5 As can be seen from the figure, the high-resolution network ( Figure 5 The subnetwork in the first row and first column of the image is used as the initial network, and the multi-resolution subnetworks are connected in parallel for multi-scale fusion, ultimately resulting in the rightmost subnetwork. In the last column of feature maps in the rightmost subnetwork, each feature map receives information from the multiple subnetworks connected in parallel. This allows the rightmost feature map to obtain both rich high-resolution and rich low-resolution information, making the visual features and text semantic features of the text region of the processed image more accurate. Accordingly, after mapping the text region coordinates of the initial text box to the text region, the visual features and text semantic features of each initial text box are also richer.

[0080] In the initial text frame, the high-resolution feature map represents more text semantic information of the text area; the low-resolution feature map represents more spatial position information of the text area. After the rightmost feature map is obtained based on the high-resolution network, since the rightmost feature map contains both rich high-resolution features and rich low-resolution features, the visual features and text semantic features of the feature map of the image to be processed can be made more accurate.

[0081] By using the HRNet model to extract visual features and text semantic features from multiple text regions in the image to be processed, each text region obtained can have rich high-resolution information and rich low-resolution information. After mapping each initial text frame to a different text region, the key information within each initial text frame can also have rich high-resolution information and rich low-resolution information. Since more high-resolution information means more text semantic information for the initial text frame, and more low-resolution information means more visual features for the initial text frame, the visual features and text semantic features of the multiple initial text frames obtained can be made richer, and the initial text frames represented by richer visual features and text semantic features are also more accurate.

[0082] Step S12: obtaining node features of a plurality of initial text boxes according to the visual features and text semantic features.

[0083] In this step, the visual features of the initial text box and the text semantic features are fused through the Kronecker product to obtain the fused features, and the fused features are the node features.

[0084] After obtaining the node features, it is also necessary to use a multimodal graph inference model to obtain the structural features of each initial text box. The structural features refer to the weights of the edges between each initial text box. The weights are used to represent the relative spatial position distance between each text box (including relative horizontal distance and relative vertical distance).

[0085] Step S13: Based on a graph convolution model, clustering the node features of the multiple initial text boxes to obtain categories corresponding to the multiple initial text boxes. The graph convolution model uses spectral domain convolution for clustering.

[0086] Among them, after the image to be processed is input into the graph convolution model, the graph convolution model not only needs to filter out the initial text box with the first preset label from multiple text boxes, but also needs to cluster the multiple initial text boxes to obtain the categories corresponding to the multiple initial text boxes.

[0087] Specifically, when clustering multiple initial text boxes, not only the node features of the initial text boxes but also their structural features must be considered. Therefore, the node features and structural features of each initial text box need to be used as input to the graph convolutional model, allowing the graph convolutional model to learn the relationship between each initial text box and then classify each initial text box.

[0088] In related technologies, the graph convolution model uses time-domain convolution (also called spatial convolution) for clustering. When using time-domain convolution clustering, the category of the current initial text box is determined based on multiple initial text boxes adjacent to the initial text box to be predicted.

[0089] For example, see Figure 6 As shown, the node corresponding to each small grid in the image is equivalent to each initial text box in the image to be processed, and has a fixed domain size when convolving the nodes (for example, Figure 6 The 3x3 convolution kernel on the right is Figure 6 The convolution kernel is used to perform convolution from the upper left corner to the lower right corner of the A node in the 8-neighborhood.

[0090] In this process, it is necessary to first build the neighborhood of node A, select a fixed number of neighboring nodes based on the expected probability of the neighboring nodes being selected; then sort the neighboring nodes according to the expected probability of the neighboring nodes being selected; finally, convolve the sorted neighboring nodes with node A through the convolution kernel to obtain the category of node A.

[0091] However, this approach to constructing neighborhoods relies on a fixed number of neighbors, e.g. Figure 6 As shown, node A must rely on 8 neighbors to obtain the category of node A, and its prediction speed is slow.

[0092] This disclosure proposes that the graph convolution model uses spectral domain convolution for clustering. Spectral domain convolution clustering can perform convolution on any number of neighbors to predict the category of the node (initial text box) to be predicted.

[0093] For example, by using spectral domain convolution, the category of node A can be obtained with the help of 8 neighbors, or based on 4 neighbors, and its prediction speed is faster than that of time domain convolution.

[0094] Step S14: determining a target text box having necessary key information from the multiple initial text boxes according to the categories corresponding to the multiple initial text boxes.

[0095] In this step, after multiple initial text boxes are recognized through the graph convolution model, the categories of the multiple initial text boxes can be obtained. For example, the category of "Zhang San" is the certificate holder, the category of "female" is gender, and so on.

[0096] Among these obtained categories, there is also non-essential key information. Non-essential key information refers to repeated information or information that is irrelevant to confirming the cardholder information and the certificate type; necessary key information refers to information related to confirming the cardholder and the certificate type.

[0097] For example, see Figure 3 As shown in the figure, the initial text box of "Senior High School Teacher Qualification" appears repeatedly on the left and right sides of the image to be processed. However, for the user to confirm the type of certificate, only one of the "Senior High School Teacher Qualification" is needed. Therefore, it is necessary to compare the clarity of the key information displayed in these two initial text boxes. Figure 2 It can be seen that the clarity of the initial text box of "Senior High School Teacher Qualification" on the left is lower than that of the initial text box of "Senior High School Teacher Qualification" on the right, so the initial text box of "Senior High School Teacher Qualification" with lower clarity on the left is filtered out to obtain the target text box of "Senior High School Teacher Qualification" with higher clarity on the right.

[0098] Clarity refers to the clarity of the key information represented within the initial text box. If the clarity of the key information within the initial text boxes representing the same key information varies, the key information within the initial text box with lower clarity is considered non-essential, and the key information within the initial text box with higher clarity is considered essential. If the clarity of the key information within the initial text boxes representing the same key information is the same, one initial text box can be selected from the two initial text boxes as the target text box.

[0099] For example, see Figure 3 As shown, the initial text box of the "Official Seal of the Certification Agency" on the right is not related to confirming the certificate holder information and the certificate type. Therefore, the "Official Seal of the Certification Agency" on the right can be determined as non-critical information and can be filtered out from multiple initial text boxes.

[0100] By filtering out duplicate information and information that is irrelevant to confirming the holder's information and the type of certificate, Figure 3 The initial text boxes such as "Zhang San", "Senior High School Teacher Qualification", "Certification Agency" and "June 1, 2002" are filtered out to obtain the following Figure 4 Multiple target text boxes are shown.

[0101] Through the key information extraction method proposed in the present invention, on the one hand, based on the HRNet model, the obtained feature map will have rich high-resolution information and low-resolution information, so as to obtain rich visual features and text semantic features. After the visual features and text semantic features are increased, the initial text box can be expressed more accurately, and then the category of the initial text box can be accurately determined based on the accurate initial text box, solving the problem of inaccurate category prediction caused by inaccurate initial text boxes from the source; secondly, the graph convolution model can use spectral domain convolution for clustering, because spectral domain convolution can predict the category of the current initial text box to be predicted through the initial text boxes of any number of neighbors, rather than taking a fixed number of neighbors to predict the initial text box. categories, so the category of the initial text box can be predicted by a smaller number of neighbors, which improves the speed of initial text box category prediction, and the category of the initial text box can be predicted by a larger number of neighbors, which improves the accuracy of initial text box category prediction; thirdly, non-initial text boxes are preliminarily screened from multiple text boxes, so that the text information in the multiple initial text boxes is key information, and then the target text box is obtained from the multiple initial text boxes, and the key information in the target text box is further made necessary key information. Finally, the necessary key information related to confirming the certificate holder and the certificate type is presented, so that the staff only needs to review the necessary key information, without having to review useless information, which greatly improves the staff's review efficiency.

[0102] In a possible implementation, since the quality of the card images taken by the user varies and may interfere with image recognition, the card images taken by the user need to be further processed before image recognition to obtain an image to be processed.

[0103] Specifically, the method includes the following steps:

[0104] Step S21: Based on the generative adversarial network, enhance the image quality of the input image to obtain a first image.

[0105] In this step, through generative adversarial networks (GAN networks), the overall or local features of the image can be purposefully enhanced, converting the originally unclear card image into a clear image to be processed, or enhancing the features of interest in the card image, suppressing the features of no interest, and expanding the differences between different features in the card image, thereby improving the visual effect of the image.

[0106] Step S22: Correcting the first image to obtain the image to be processed.

[0107] In this step, the correction includes removing the background area of ​​the first image to obtain the image to be processed, and fitting the multiple vertices of the first image according to the multiple edges of the first image to obtain the multiple vertices of the first image; and obtaining the image to be processed according to the multiple vertices of the first image and the multiple edges of the first image.

[0108] The first image may be a card image such as a teacher qualification certificate or an agent qualification certificate.

[0109] Due to the different angles at which users take photos, the shape of the card image may vary, potentially being a parallelogram or a trapezoid. The card image may also be tilted or rotated. For parallelogram-shaped card images, a global perspective transformation can be used to remove the background. For trapezoidal-shaped card images, an affine transformation can be used to remove the background. For tilted or rotated card images, an affine transformation can also be used to process them to obtain a card image with the correct orientation.

[0110] Through global perspective transformation and affine transformation, the background of card images of different angles and shapes can be removed to obtain the image to be processed, avoiding the background of the card image taken by the user from interfering with image recognition and improving the accuracy of image recognition.

[0111] When two ID cards are placed on top of each other, there may be other ID card images below the ID card image that the user takes a photo of. Figure 7 As shown, the card image at the top is the image to be processed, and the card image at the bottom is the remaining image accidentally taken by the user. In order to avoid the lower image interfering with the upper image, it is also necessary to perform vertex fitting on the obtained card image through global perspective transformation to obtain the image to be processed.

[0112] Specifically, the four edges of the card image can be extended, and the four vertices where each edge intersects (such as Figure 7 The four vertices A, B, C, and D shown in the figure are used as the four vertices of the card image. Then, the image to be processed is obtained based on the four vertices and four edges, and the remaining images except the image to be processed are deleted to remove the interference of the image below.

[0113] Alternatively, the four sides of the card image may be extended, and the four vertices where each side intersects are used as the four vertices of the card image. The four vertices are then fitted using an image model to obtain the image to be processed.

[0114] For example, four vertices and images corresponding to the four vertices may be input into an image model for training, so that after receiving the four vertices, the image model can output an image to be processed by fitting the four vertices.

[0115] By removing the background of the first image, the background of the first image can be prevented from interfering with the image; by correcting the first image, the image below the first image can be prevented from interfering with the first image, and the occurrence of incorrect key information extraction caused by identifying the image below the first image during image recognition can be avoided.

[0116] See also Figure 8 As shown, an exemplary embodiment of the present disclosure shows a block diagram of a key information extraction device, where the device 1300 includes: an extraction module 1301 , a node feature determination module 1302 , a clustering module 1303 and a key information extraction module 1304 .

[0117] The extraction module 1301 is used to extract visual features and text semantic features of multiple initial text boxes in the image to be processed;

[0118] A node feature determination module 1302 is configured to obtain node features of a plurality of initial text frames based on the visual features and text semantic features;

[0119] A clustering module 1303 is configured to cluster the node features of the multiple initial text boxes based on a graph convolution model to obtain categories corresponding to the multiple initial text boxes, wherein the graph convolution model uses spectral domain convolution for clustering;

[0120] The key information extraction module 1304 is configured to determine a target text box having necessary key information from a plurality of initial text boxes according to the type of the image to be processed.

[0121] Optionally, the extraction module 1301 includes:

[0122] The first extraction module is used to extract visual features and text semantic features of the multiple initial text boxes based on the HRNet model.

[0123] Optionally, the apparatus 1300 further includes:

[0124] an identification module, configured to identify the image to be processed and obtain a plurality of text boxes of the image to be processed;

[0125] a labeling module, configured to label a text box having key information among the plurality of text boxes to obtain an initial text box having a first preset label;

[0126] The second extraction module is used to extract the initial text box with the first preset label based on the graph convolution model.

[0127] Optionally, the apparatus 1300 further includes:

[0128] The training module is used to train the initial model using Chinese and English as training samples to obtain the graph convolution model.

[0129] Optionally, the apparatus 1300 further includes:

[0130] A first image determination module is configured to enhance the image quality of an input image based on a generative adversarial network to obtain a first image;

[0131] A correction module is used to correct the first image to obtain the image to be processed.

[0132] Optionally, the correction module includes:

[0133] The first correction processing module is used to remove the background area of ​​the first image to obtain the image to be processed.

[0134] Optionally, the correction module further includes:

[0135] a fitting module, configured to fit the multiple vertices of the first image according to the multiple edges of the first image to obtain the multiple vertices of the first image;

[0136] The second correction processing module is configured to obtain the image to be processed according to the multiple vertices of the first image and the multiple edges of the first image.

[0137] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0138] The present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the key information extraction method provided by the present disclosure.

[0139] The present disclosure also provides an electronic device, comprising:

[0140] a memory having a computer program stored thereon;

[0141] A processor is used to execute the computer program in the memory to implement the steps of the key information extraction method provided by the present disclosure.

[0142] Figure 91 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 9 The electronic device 1900 includes a processor 1922, which may be one or more, and a memory 1932 for storing a computer program executable by the processor 1922. The computer program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 1922 may be configured to execute the computer program to perform the above-mentioned key information extraction method.

[0143] In addition, the electronic device 1900 may further include a power supply component 1926 and a communication component 1950. The power supply component 1926 may be configured to perform power management of the electronic device 1900, and the communication component 1950 may be configured to implement communication of the electronic device 1900, for example, wired or wireless communication. In addition, the electronic device 1900 may further include an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM , Linux TM etc.

[0144] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described key information extraction method. For example, the non-transitory computer-readable storage medium may be the aforementioned memory 1932 including the program instructions. The program instructions may be executed by the processor 1922 of the electronic device 1900 to perform the above-described key information extraction method.

[0145] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned key information extraction method when executed by the programmable device.

[0146] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0147] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0148] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A key information extraction method, characterized in that: The method comprises: Extracting visual features and text semantic features of multiple initial text boxes in the image to be processed; Obtaining node features of a plurality of initial text frames according to the visual features and the text semantic features; the node features are fusion features of the visual features and the text semantic features; Based on a graph convolution model, clustering the node features of the multiple initial text boxes to obtain categories corresponding to the multiple initial text boxes, wherein the graph convolution model uses spectral domain convolution for clustering; Determining a target text box having necessary key information from the multiple initial text boxes according to the categories corresponding to the multiple initial text boxes; Before extracting the visual features and text semantic features of multiple initial text boxes in the image to be processed, the method includes: identifying the image to be processed and obtaining multiple text boxes in the image to be processed; labeling the text boxes with key information in the multiple text boxes to obtain initial text boxes with a first preset label; based on the graph convolution model, screening out non-initial text boxes with a second preset label and extracting the initial text boxes with the first preset label; Among them, the text boxes with key information in multiple text boxes are marked to obtain an initial text box with a first preset label, including: comparing whether the non-key information on the preset template is consistent with the non-key information on the image to be processed, and if they are consistent, marking the second preset label of the preset text box on the preset template on the text box in the image to be processed; there are multiple preset text boxes on the preset template, each preset text box has its own non-key information, and each preset text box is marked with the second preset label.

2. The key information extraction method according to claim 1, characterized in that: The step of extracting visual features of a plurality of initial text boxes from the image to be processed includes: Based on the HRNet model, visual features and text semantic features of the multiple initial text boxes are extracted.

3. The key information extraction method according to claim 1, characterized in that: The graph convolution model is trained by the following steps: The initial model is trained using Chinese and English as training samples to obtain the graph convolutional model.

4. The key information extraction method according to claim 1, characterized in that: The image to be processed is obtained by the following steps: enhancing the image quality of the input image based on a generative adversarial network to obtain a first image; Correcting the first image to obtain the image to be processed.

5. The key information extraction method according to claim 4, characterized in that: Correcting the first image to obtain the image to be processed includes: The background area of ​​the first image is removed to obtain the image to be processed.

6. The key information extraction method according to claim 4, characterized in that: Correcting the first image to obtain the image to be processed includes: Fitting multiple vertices of the first image according to multiple edges of the first image to obtain multiple vertices of the first image; The image to be processed is obtained according to a plurality of vertices of the first image and a plurality of edges of the first image.

7. A key information extraction device, characterized in that: The device comprises: An extraction module, used to extract visual features and text semantic features of multiple initial text boxes in the image to be processed; A node feature determination module is used to obtain node features of multiple initial text frames based on the visual features and text semantic features; the node features are fusion features of the visual features and the text semantic features; A clustering module, configured to cluster the node features of the multiple initial text boxes based on a graph convolution model to obtain categories corresponding to the multiple initial text boxes, wherein the graph convolution model uses spectral domain convolution for clustering; A key information extraction module, configured to determine a target text box having necessary key information from a plurality of initial text boxes according to the type of the image to be processed; Before extracting the visual features and text semantic features of multiple initial text boxes in the image to be processed, the device is further configured to identify the image to be processed and obtain multiple text boxes in the image to be processed; annotate the text boxes with key information in the multiple text boxes to obtain initial text boxes with a first preset label; and based on the graph convolution model, screen out non-initial text boxes with a second preset label and extract the initial text boxes with the first preset label; Among them, the marking of text boxes with key information in multiple text boxes to obtain an initial text box with a first preset label includes: comparing whether the non-key information on the preset template is consistent with the non-key information on the image to be processed. If they are consistent, the second preset label of the preset text box on the preset template is marked on the text box in the image to be processed; the preset template has multiple preset text boxes, each preset text box has its own non-key information, and each preset text box is marked with the second preset label.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.

Images