Image information extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image information extraction method and device, electronic equipment and a storage medium, and belongs to the technical field of image recognition and artificial intelligence. The method comprises the following steps: performing text recognition on a sample image to obtain text information; performing key information labeling on the text information to obtain text labeling information; performing key information identification on the text information based on the extraction indication information and an image information extraction model to obtain predicted text information; adjusting the extraction indication information and the image information extraction model based on the text annotation information and the prediction text information to obtain target indication information and a target information extraction model; and obtaining a target image, and performing information extraction on the target image based on the target indication information and the target information extraction model. According to the embodiment of the invention, the accuracy of image information extraction can be improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition and artificial intelligence technology, and in particular to an image information extraction method and device, an electronic device and a storage medium. Background Art

[0002] Image information extraction refers to the extraction of specific information from an image. By extracting specific information from an image, it can help users quickly obtain the information they need from the image. For example, in the financial field, by extracting information such as the customer name from an insurance policy image, the customer information of the policy can be quickly obtained.

[0003] At present, the common image information extraction process is prone to image tilt problems, which leads to serial extraction errors when recognizing the text in the image. Therefore, how to improve the accuracy of image information extraction has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image information extraction method and device, an electronic device and a storage medium, aiming to improve the accuracy of image information extraction.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides an image information extraction method, the method comprising:

[0006] Get a sample image containing text content;

[0007] Performing text recognition on the sample image to obtain text information;

[0008] Performing key information annotation on the text information to obtain text annotation information;

[0009] Based on preset extraction instruction information and a preset image information extraction model, key information recognition is performed on the text information to obtain first predicted text information;

[0010] Based on the text annotation information and the first predicted text information, adjusting the extraction indication information to obtain target indication information;

[0011] Based on the target indication information and the image information extraction model, performing key information recognition on the text information to obtain second predicted text information;

[0012] Based on the text annotation information and the second predicted text information, adjusting the image information extraction model to obtain a target information extraction model;

[0013] A target image is acquired, and information is extracted from the target image based on the target indication information and the target information extraction model.

[0014] In some embodiments, acquiring a target image and extracting information from the target image based on the target indication information and the target information extraction model includes:

[0015] Acquire a target image, and perform text recognition on the target image to obtain a target text;

[0016] Based on the target information extraction model, encoding the target text to obtain a text encoding vector;

[0017] Performing hidden state calculation on the text encoding vector to obtain a text hidden state vector;

[0018] Based on the target indication information, performing attention weighting processing on the text hidden state vector to obtain an attention weight of the text hidden state vector;

[0019] Based on the attention weight, a target word vector is selected from the text hidden state vector;

[0020] The target word vectors are concatenated to obtain the predicted text information.

[0021] In some embodiments, performing text recognition on the text image to obtain text information includes:

[0022] Performing text detection on the sample image to obtain a text area;

[0023] Perform text extraction on the text area to obtain the text information.

[0024] In some embodiments, performing text detection on the sample image to obtain a text area includes:

[0025] Performing image preprocessing on the sample image to obtain a text standard image;

[0026] Performing convolution pooling processing on the text standard image to obtain a text image feature vector;

[0027] Based on the text image feature vector, generating a candidate text box;

[0028] Scoring the candidate text boxes to obtain text box score data;

[0029] The text area is screened out from the candidate text boxes based on the text box score data.

[0030] In some embodiments, extracting text from the text area to obtain the text information includes:

[0031] Serializing the text region to obtain a text sequence, wherein the text sequence includes a text word vector;

[0032] Performing context information capture processing on the text sequence to obtain a text feature sequence;

[0033] Based on the text feature sequence, calculating the phrase probability distribution of the text word vector;

[0034] Based on the phrase probability distribution, the text word vectors are sorted to obtain the text information.

[0035] In some embodiments, the step of adjusting the prompt word of the extraction indication information based on the text annotation information and the predicted text information to obtain the target indication information includes:

[0036] Comparing the text annotation information with the predicted text information to obtain an information proofreading result, wherein the information proofreading result indicates whether the text annotation information is the same as or different from the predicted text information;

[0037] If the information proofreading result indicates that the predicted text information is different from the text annotation information, selecting a target indication information phrase from a preset indication information word library;

[0038] Performing indicator phrase positioning on the extracted indicator information to obtain an indicator phrase position;

[0039] Based on the indication phrase position and the indication information phrase, the extracted indication information is phrase-replaced to obtain the target indication information.

[0040] In some embodiments, adjusting the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model includes:

[0041] Comparing the text annotation information with the second predicted text information to obtain a second information proofreading result, wherein the second information proofreading result indicates that the text annotation information is the same as or different from the second predicted text information;

[0042] If the second information proofreading result indicates that the predicted text information is different from the text annotation information, adding a network layer to the image information extraction model to obtain an initial information extraction model;

[0043] Based on the text information and the text annotation information, the initial information extraction model is trained to obtain the target information extraction model.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image information extraction device, the device comprising:

[0045] A text image acquisition module is used to acquire a sample image containing text content;

[0046] A text image recognition module, used to perform text recognition on the sample image to obtain text information;

[0047] A text information annotation module, used to annotate the text information with key information to obtain text annotation information;

[0048] A first information extraction module, configured to identify key information of the text information based on preset extraction instruction information and a preset image information extraction model, to obtain first predicted text information;

[0049] an indication information adjustment module, configured to adjust the extraction indication information based on the text annotation information and the first predicted text information to obtain target indication information;

[0050] A second information extraction module, configured to perform key information recognition on the text information based on the target indication information and the image information extraction model to obtain second predicted text information;

[0051] An extraction model adjustment module, used to adjust the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model;

[0052] The image information extraction module is used to obtain a target image and extract information from the target image based on the target indication information and the target information extraction model.

[0053] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0054] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0055] The image information extraction method and device, electronic device and storage medium proposed in the present application obtain text information by performing text recognition on a sample image, and can accurately recognize the text information in the image, and then instruct the image information extraction model to perform key information recognition on the text information according to the preset extraction instruction information to obtain the first predicted text information, which can improve the recognition of the extracted information in the text information, thereby improving the accuracy of image information extraction, and then adjust the extraction instruction information according to the text annotation information and the first predicted text information to obtain the target instruction information, which can improve the accuracy of the instruction information, thereby improving the accuracy of image information extraction, and then adjust the image information extraction model according to the text annotation information and the second predicted text information to obtain the target information extraction model, which can improve the accuracy of the image information extraction model, thereby improving the accuracy of image information extraction, and finally, extract information from the target image according to the target instruction information and the target information extraction model, which can improve the success rate of target image extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the image information extraction method provided in an embodiment of the present application;

[0057] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0058] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0059] Figure 4 yes Figure 2 Flow chart of step S202 in FIG.

[0060] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0061] Figure 6 yes Figure 1 Flow chart of step S107 in FIG.

[0062] Figure 7 yes Figure 1 Flow chart of step S108 in FIG.

[0063] Figure 8 is a schematic diagram of the structure of an image information extraction device provided in an embodiment of the present application;

[0064] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] Text recognition model: The text recognition model is a technology used to extract text from images. It uses optical character recognition technology to convert text in images into editable text. Text recognition models are widely used in computer vision and natural language processing, such as converting scanned documents into editable text files, automatically reading license plate numbers, processing handwritten text, etc. The development of text recognition technology is driven by the explosive growth of mobile Internet and the popularization of deep learning technology. Its application scope has penetrated into many fields such as image content extraction of advertising recommendation systems, review and filtering of image and video content, medical image recognition, certificate recognition, document recognition, and street view sign recognition.

[0070] Large language model: A large language model is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. The large language model uses a Transformer architecture and pre-training objectives similar to the small model. The difference from the small model is the increase in model size, training data, and computing resources.

[0071] Attention mechanism: The attention mechanism is a computational method that mimics human visual attention, which aims to enable the model to dynamically focus on the most relevant information when processing input data, thereby improving performance. Its benefit is that it can effectively capture long-distance dependencies and important features, significantly improving the accuracy and efficiency of tasks such as natural language processing, image recognition, and translation. By adaptively adjusting the focus of attention, the attention mechanism enhances the model's expressiveness and enables it to perform better in complex tasks.

[0072] Image information extraction refers to the extraction of specific information from an image. By extracting specific information from an image, it can help users quickly obtain the information they need from the image. For example, in the financial field, by extracting information such as the customer name from an insurance policy image, the customer information of the policy can be quickly obtained.

[0073] At present, the common image information extraction process is prone to image tilt problems, which leads to serial extraction errors when recognizing the text in the image. Therefore, how to improve the accuracy of image information extraction has become a technical problem that needs to be solved urgently.

[0074] Based on this, the embodiments of the present application provide an image information extraction method and device, an electronic device and a storage medium, aiming to improve the accuracy of image information extraction.

[0075] The image information extraction method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image information extraction method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0077] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0078] The image information extraction method provided in the embodiment of the present application relates to the field of image recognition and artificial intelligence technology. The image information extraction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image information extraction method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] Figure 1 is an optional flowchart of the image information extraction method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0081] Step S101, obtaining a sample image containing text content;

[0082] Step S102, performing text recognition on the sample image to obtain text information;

[0083] Step S103, annotating the text information with key information to obtain text annotation information;

[0084] Step S104, performing key information recognition on the text information based on the preset extraction instruction information and the preset image information extraction model to obtain first predicted text information;

[0085] Step S105, adjusting the extraction indication information based on the text annotation information and the first predicted text information to obtain target indication information;

[0086] Step S106, based on the target indication information and the image information extraction model, key information is identified on the text information to obtain second predicted text information;

[0087] Step S107, adjusting the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model;

[0088] Step S108, acquiring a target image, and extracting information from the target image based on the target indication information and the target information extraction model.

[0089] Steps S101 to S108 shown in the embodiment of the present application are to obtain a sample image containing text content, and then perform text recognition on the sample image to obtain text information. Secondly, key information is annotated on the text information to obtain text annotation information. Then, key information is identified on the text information according to pre-set extraction indication information and a pre-set image information extraction model to obtain first predicted text information. If the first predicted text information is different from the text annotation information, the extraction indication information is adjusted to obtain target indication information. Then, based on the target indication information and the image information extraction model, key information is identified on the text information to obtain second predicted text information. If the second predicted text information is different from the text annotation information, the image information extraction model is adjusted to obtain a target information extraction model. Finally, a target image is obtained, and information is extracted from the target image according to the target indication information and the target information extraction model. Therefore, the present application obtains text information by performing text recognition on a sample image, and can accurately recognize the text information in the image. Then, according to the preset extraction indication information, the image information extraction model is instructed to perform key information recognition on the text information to obtain the first predicted text information, which can improve the recognition of the extracted information in the text information, thereby improving the accuracy of image information extraction. Secondly, according to the text annotation information and the first predicted text information, the extraction indication information is adjusted to obtain the target indication information, which can improve the accuracy of the indication information, thereby improving the accuracy of image information extraction. Then, according to the text annotation information and the second predicted text information, the image information extraction model is adjusted to obtain the target information extraction model, which can improve the accuracy of the image information extraction model, thereby improving the accuracy of image information extraction. Finally, according to the target indication information and the target information extraction model, information is extracted from the target image, which can improve the success rate of target image extraction.

[0090] In step S101 of some embodiments, the text content refers to text containing text information, wherein the text information may be in Chinese, English, etc. The sample image may be a text image containing text information, such as an electronic insurance policy image, an electronic expense report, etc.

[0091] The present application can obtain an initial image containing text content by searching the Internet, and then perform image cleaning on the initial image to obtain a cleaned and visible sample image.

[0092] In step S102 of some embodiments, text information in the sample image can be obtained by performing operations such as text detection and text extraction on the sample image.

[0093] For details, see Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0094] Step S201, performing text detection on the sample image to obtain a text area;

[0095] Step S202: extract text from the text area to obtain text information.

[0096] In step S201 of some embodiments, the sample image may be preprocessed to obtain a standard text image, and then image features in the standard text image may be extracted to obtain a text image feature vector. Finally, a text region may be generated based on the text image feature vector.

[0097] For details, see Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S305:

[0098] Step S301, performing image preprocessing on the sample image to obtain a text standard image;

[0099] Step S302, performing convolution pooling processing on the text standard image to obtain a text image feature vector;

[0100] Step S303, generating a candidate text box based on the text image feature vector;

[0101] Step S304, scoring the candidate text boxes to obtain text box score data;

[0102] Step S305: Filter out text areas from the candidate text boxes based on the text box score data.

[0103] In step S301 of some embodiments, a standard text image can be obtained by performing image preprocessing such as image scaling, contrast enhancement, and edge detection on the sample image.

[0104] In step S302 of some embodiments, by performing convolution processing on the text standard image, image features in the text standard image can be extracted, and then pooling processing is performed on the image features to achieve dimensionality reduction processing of the image features, thereby reducing the computational difficulty of the text standard image.

[0105] In step S303 of some embodiments, based on the text image feature vector, a region in the sample image having features similar to those of the text image can be identified, thereby obtaining a candidate text box.

[0106] In step S304 and step S305 of some embodiments, the candidate text boxes are scored for text similarity to obtain text box score data for each candidate text box, and then a target area whose text box score data exceeds a preset value is selected from the candidate text boxes and used as the text area of ​​the sample image.

[0107] In steps S301 to S305 shown in this embodiment, a standard text image is obtained by performing image preprocessing on the sample image to avoid unclear or overly large images in the sample image, thereby improving the accuracy of the standard text image. Furthermore, by performing convolution pooling, text box generation and other processing on the standard text image, the text area can be found in the sample image, and the accuracy of the text area is guaranteed, thereby improving the accuracy of the text information.

[0108] In step S202 of some embodiments, the above-mentioned text area is serialized to obtain a text sequence, and then the context information of the text sequence is captured to obtain a text feature sequence. Then, based on the text feature sequence, the confidence of the accuracy of each text word vector in the text sequence in the text content, that is, the phrase probability distribution, can be calculated. Finally, according to the phrase probability distribution, the text word vectors are sorted to obtain text information.

[0109] For details, see Figure 4 In some embodiments, step S202 may include but is not limited to steps S401 to S404:

[0110] Step S401, serializing the text region to obtain a text sequence, wherein the text sequence includes a text word vector;

[0111] Step S402, performing context information capture processing on the text sequence to obtain a text feature sequence;

[0112] Step S403, calculating the phrase probability distribution of the text word vector based on the text feature sequence;

[0113] Step S404, sorting the text word vectors based on the phrase probability distribution to obtain text information.

[0114] In step S401 of some embodiments, by serializing pixel features in the text area, a text sequence in the text area can be obtained. It should be noted that the text sequence includes a vector representation of each word in the text area, that is, a text word vector.

[0115] In step S402 of some embodiments, by capturing the dependency relationship between each text word vector in the text sequence and the context, the context information step of the text sequence can be implemented to obtain a text feature sequence.

[0116] In step S403 of some embodiments, based on the above text feature sequence, the accuracy of the position of each text word vector in the text sequence can be determined, thereby calculating the phrase probability distribution of each text word vector.

[0117] In step S404 of some embodiments, unreasonable phrases in the text sequence can be rearranged according to the above-mentioned phrase probability distribution to obtain text information in the sample image.

[0118] In steps S401 to S404 shown in the present embodiment, the text region is serialized to obtain a text sequence, and then the text sequence is processed for context information capture to obtain a text feature sequence. Secondly, based on the text feature sequence, the phrase probability distribution of each text word vector in the text sequence is calculated. Finally, based on the phrase probability distribution, the text word vectors are sorted to obtain text information, which can realize the serial correction of the text content, thereby improving the accuracy of text recognition of the sample image.

[0119] In steps S201 to S202 shown in this embodiment, by performing text detection on the sample image to obtain the text area, the area where the text content in the sample image is located can be accurately identified, and then text extraction is performed on the text area to obtain text information, which can achieve serial correction of the text content in the text area, thereby ensuring the rationality and readability of the text information.

[0120] In step S103 of some embodiments, the information to be extracted from the above text information is annotated to obtain text annotation information. For example, by annotating the key information name in the text information "Name: Xiao Ming, Gender: Male, Age: 18", the text annotation information "Xiao Ming" can be obtained.

[0121] In step S104 of some embodiments, the extraction instruction information may be instruction information for instructing the model to extract text content from the text information. For example, the extraction instruction information for instructing the model to extract the text content "Xiao Ming" from the text information "Name: Xiao Ming, Gender: Male, Age: 18" may be "extract the name in the text". The image information extraction model may be a large language model, such as a GPT large model, a Wenxin large model, etc.

[0122] The present application inputs the pre-set extraction indication information and the above-mentioned text information into the image information extraction model, and the image information extraction model can search and filter the information in the text information according to the extraction indication information, thereby extracting the first predicted text information.

[0123] In step S105 of some embodiments, the text annotation information is compared with the predicted text information, and then based on the comparison result, it is determined whether the prompt word adjustment of the extracted indication information is required. If the conditions for adjusting the prompt word of the extracted indication information are met, an indication information phrase is randomly selected from a pre-arranged indication information vocabulary to replace the indication information phrase in the extracted indication information, so that the target indication information can be obtained.

[0124] For details, see Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S504:

[0125] Step S501, comparing the text annotation information with the predicted text information to obtain an information verification result, wherein the information verification result indicates whether the text annotation information is the same as or different from the predicted text information;

[0126] Step S502, if the information proofreading result indicates that the predicted text information is different from the text annotation information, a target indication information phrase is selected from a preset indication information word library;

[0127] Step S503, performing indicator phrase positioning on the extracted indicator information to obtain the indicator phrase position;

[0128] Step S504: Based on the position of the indicator phrase and the indicator information phrase, the extracted indicator information is replaced with a phrase to obtain the target indicator information.

[0129] In step S501 of some embodiments, by comparing the text content of the text annotation information and the predicted text information, it is possible to determine whether the text annotation information and the predicted text information are the same, thereby generating an information proofreading result indicating whether the text annotation information and the predicted text information are the same or different.

[0130] In step S502 of some embodiments, the indication information vocabulary refers to a database storing a plurality of indication information phrases, wherein the indication information phrase refers to a phrase in the extraction indication information used to describe the prompt extraction content, for example, the "name" in the extraction indication information "extract the name in the text".

[0131] In an embodiment of the present application, when the information proofreading result indicates that the predicted text information is different from the text annotation information, the target indication information phrase can be obtained by searching the indication information vocabulary for an indication information phrase with a similar meaning to the indication information phrase in the extracted indication information.

[0132] In step S503 and step S504 of some embodiments, the phrase position of the indication information phrase in the extracted indication information is found to obtain the indication phrase position, and then the target indication information phrase is replaced with the indication information phrase in the indication phrase position to obtain the target indication information.

[0133] In steps S501 to S504 shown in the present embodiment, by comparing the text annotation information with the predicted text information, an information proofreading result indicating that the text annotation information is the same as or different from the predicted text information is obtained, and when the information proofreading result indicates that the predicted text information is different from the text annotation information, a target indication information phrase is selected from a preset indication information vocabulary, and then the indication phrase is located on the extracted indication information to obtain the indication phrase position, and finally, according to the indication phrase position and the indication information phrase, the extraction indication information is replaced with a phrase to obtain the target indication information, which can make the indication information phrase used to describe the text extraction content in the indication information more accurate, thereby improving the accuracy of text extraction and the accuracy of image information extraction.

[0134] In step S106 of some embodiments, when the extraction indication information is replaced by the target indication information, the target indication information and the text information are input into the image information extraction model again for key information identification, and the second predicted text information can be obtained by the image information extraction model extracting the text information based on the target indication information.

[0135] In step S107 of some embodiments, by comparing the text content of the text annotation information and the second predicted text information, it can be determined whether the target indication information is accurate. If the target indication information is still not accurate enough, the indication information phrases in the target indication information can be replaced multiple times. After the indication information phrases are replaced multiple times and the predicted text information extracted by the image information extraction model from the image information is still different from the text annotation information, the image information extraction model is fine-tuned to improve the accuracy of the model.

[0136] For details, see Figure 6In some embodiments, step S107 may include but is not limited to steps S601 to S603:

[0137] Step S601, comparing the text annotation information with the second predicted text information to obtain a second information proofreading result, wherein the second information proofreading result indicates whether the text annotation information is the same as or different from the second predicted text information;

[0138] Step S602: if the second information proofreading result indicates that the predicted text information is different from the text annotation information, a network layer is added to the image information extraction model to obtain an initial information extraction model;

[0139] Step S603: Based on the text information and the text annotation information, the initial information extraction model is trained to obtain a target information extraction model.

[0140] In step S601 of some embodiments, by comparing the text annotation information with the text content in the second predicted text information, a second information proofreading result indicating whether the text annotation information is the same as or different from the second predicted text information can be generated.

[0141] In step S602 of some embodiments, when the second information proofreading result indicates that the predicted text information is different from the text annotation information, the image information extraction model can be fine-tuned by increasing the number of network layers in the image information extraction model to obtain an initial information extraction model.

[0142] In step S603 of some embodiments, the above-mentioned text information and marked text annotation information are input into the above-mentioned initial information extraction model for model training, which can achieve parameter adjustment of the initial information extraction model, thereby ensuring that the trained target information extraction model can be applied to the above-mentioned text information.

[0143] In steps S601 to S603 shown in the present embodiment, the second proofreading information is obtained by comparing the text annotation information with the second predicted text information, and when the second proofreading information indicates that the predicted text information is different from the text annotation information, a network layer of the image information extraction model is added to obtain an initial information extraction model, thereby improving the accuracy of the initial information extraction model, and then using the text information and the text annotation information to train the initial information extraction model to obtain a target information extraction model, which can ensure that the target information extraction model is applicable to text information and texts similar to the text information, thereby improving the accuracy of information extraction.

[0144] In step S108 of some embodiments, the target image may be an image containing the same type of text content as the sample image. For example, when the text content in the sample image is insurance policy information, the text content of the target image is also insurance policy information.

[0145] The present application can obtain a target image by receiving an image containing text content uploaded by a user, and then perform text recognition on the text content in the target image to obtain a target text, and then input the target text and target indication information into a target information extraction model to extract specific content from the target text.

[0146] For details, see Figure 7 In some embodiments, step S108 may include but is not limited to steps S701 to S706:

[0147] Step S701, acquiring a target image, and performing text recognition on the target image to obtain a target text;

[0148] Step S702, encoding the target text based on the target information extraction model to obtain a text encoding vector;

[0149] Step S703, performing hidden state calculation on the text encoding vector to obtain a text hidden state vector;

[0150] Step S704, performing attention weighting processing on the text hidden state vector based on the target indication information to obtain the attention weight of the text hidden state vector;

[0151] Step S705, based on the attention weight, filter out the target word vector from the text hidden state vector;

[0152] Step S706, concatenate the target word vectors to obtain predicted text information.

[0153] In step S701 of some embodiments, after receiving the target image input by the user, the target text can be obtained by performing text recognition on the target image, wherein the text recognition method of the target image is similar to the text recognition method of the sample image, and thus will not be described in detail.

[0154] In steps S702 to S706 of some embodiments, the target information extraction model is used to encode the target text in the input model to obtain a text encoding vector, and then the context information of the text encoding vector is captured and learned to obtain a text latent state vector. Furthermore, after obtaining the text latent state vector containing the text encoding vector and the context information of the text encoding vector, the target information extraction model can perform attention weight allocation processing on each word vector in the text latent state vector according to the target indication information, thereby obtaining the attention weight of the text latent state vector. Secondly, according to the attention weight, an activation function operation is performed on each word vector in the text latent state vector, and the probability distribution of each word vector as a phrase in the predicted text information can be obtained. According to the probability distribution, the target word vector can be screened out from the word vector. Finally, the target word vectors are spliced, and the target information extraction model outputs the predicted text information.

[0155] In steps S701 to S706 shown in this embodiment, the target text is obtained by performing text recognition on the target image to ensure the accuracy of the target text, and then the target text is extracted using the adjusted target indication information and the target information extraction model to obtain predicted text information, thereby improving the accuracy of information extraction of the target image.

[0156] In one application scenario, taking the name extraction of an electronic insurance policy image as an example, an electronic insurance policy image containing electronic insurance policy information is obtained from a browser webpage through a network query, and then a text recognition model is used to perform text recognition on the electronic insurance policy image to obtain the insurance policy information. Then, the name in the insurance policy information is annotated by manual or intelligent annotation to obtain the insurance policy annotation information. Secondly, according to the pre-set extraction instruction information "extract the name in the text" and the insurance policy information extraction model GPT large model, the insurance policy information is extracted to obtain the first predicted name information containing the name. If the first predicted name information If the second predicted name information is different from the above-mentioned policy marking information, the extraction instruction information "Extract the name in the text" is replaced with the target instruction information "Extract the surname and first name in the text", and then the target instruction information and the policy information are input into the policy information extraction model GPT large model, and the second predicted name information of the policy information can be obtained. If the second predicted name information is different from the above-mentioned policy marking information, the policy information extraction model is fine-tuned to obtain the target policy extraction model. Finally, after obtaining the target electronic policy image, information is extracted from the target electronic policy image according to the above-mentioned target instruction information and the target policy extraction model.

[0157] The present application obtains text information by performing text recognition on a sample image, and can accurately recognize the text information in the image. Then, according to preset extraction indication information, the image information extraction model is instructed to perform key information recognition on the text information to obtain the first predicted text information, which can improve the recognition of the extracted information in the text information, thereby improving the accuracy of image information extraction. Secondly, according to the text annotation information and the first predicted text information, the extraction indication information is adjusted to obtain the target indication information, which can improve the accuracy of the indication information, thereby improving the accuracy of image information extraction. Then, according to the text annotation information and the second predicted text information, the image information extraction model is adjusted to obtain the target information extraction model, which can improve the accuracy of the image information extraction model, thereby improving the accuracy of image information extraction. Finally, according to the target indication information and the target information extraction model, information is extracted from the target image, which can improve the success rate of target image extraction.

[0158] See also Figure 8 The present application also provides an image information extraction device, which can implement the above-mentioned image information extraction method, and the device includes:

[0159] The text image acquisition module 801 is used to acquire a sample image containing text content;

[0160] The text image recognition module 802 is used to perform text recognition on the sample image to obtain text information;

[0161] The text information annotation module 803 is used to annotate the text information with key information to obtain text annotation information;

[0162] A first information extraction module 804, configured to identify key information of the text information based on preset extraction instruction information and a preset image information extraction model, and obtain first predicted text information;

[0163] The indication information adjustment module 805 is used to adjust the extraction indication information based on the text annotation information and the first predicted text information to obtain the target indication information;

[0164] The second information extraction module 806 is used to identify key information of the text information based on the target indication information and the image information extraction model to obtain the second predicted text information;

[0165] An extraction model adjustment module 807 is used to adjust the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model;

[0166] The image information extraction module 808 is used to obtain a target image and extract information from the target image based on the target indication information and the target information extraction model.

[0167] The specific implementation of the image information extraction device is basically the same as the specific implementation of the above-mentioned image information extraction method, and will not be repeated here.

[0168] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image information extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0169] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image information extraction method of the embodiment of this application;

[0172] Input / output interface 903, used to implement information input and output;

[0173] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0174] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0175] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0176] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image information extraction method is implemented.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The image information extraction method, image information extraction device, electronic device and storage medium provided by the embodiments of the present application obtain a sample image containing text content, then perform text recognition on the sample image to obtain text information, then perform key information annotation on the text information to obtain text annotation information, then perform key information identification on the text information according to preset extraction indication information and a preset image information extraction model to obtain first predicted text information, if the first predicted text information is different from the text annotation information, adjust the extraction indication information to obtain target indication information, then perform key information identification on the text information based on the target indication information and the image information extraction model to obtain second predicted text information, if the second predicted text information is different from the text annotation information, adjust the image information extraction model to obtain a target information extraction model, finally, obtain a target image, and extract information from the target image according to the target indication information and the target information extraction model.

[0179] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0180] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for extracting image information, characterized in that: The method comprises: Get a sample image containing text content; Performing text recognition on the sample image to obtain text information; Performing key information annotation on the text information to obtain text annotation information; Based on preset extraction instruction information and a preset image information extraction model, key information recognition is performed on the text information to obtain first predicted text information; Based on the text annotation information and the first predicted text information, adjusting the extraction indication information to obtain target indication information; Based on the target indication information and the image information extraction model, performing key information recognition on the text information to obtain second predicted text information; Based on the text annotation information and the second predicted text information, adjusting the image information extraction model to obtain a target information extraction model; A target image is acquired, and information is extracted from the target image based on the target indication information and the target information extraction model.

2. The method according to claim 1, characterized in that: The acquiring of the target image and extracting information from the target image based on the target indication information and the target information extraction model comprises: Acquire a target image, and perform text recognition on the target image to obtain a target text; Based on the target information extraction model, encoding the target text to obtain a text encoding vector; Performing hidden state calculation on the text encoding vector to obtain a text hidden state vector; Based on the target indication information, performing attention weighting processing on the text hidden state vector to obtain an attention weight of the text hidden state vector; Based on the attention weight, a target word vector is selected from the text hidden state vector; The target word vectors are concatenated to obtain the predicted text information.

3. The method according to claim 1, characterized in that The performing text recognition on the text image to obtain text information includes: Performing text detection on the sample image to obtain a text area; Perform text extraction on the text area to obtain the text information.

4. The method according to claim 3, characterized in that The performing text detection on the sample image to obtain a text area includes: Performing image preprocessing on the sample image to obtain a text standard image; Performing convolution pooling processing on the text standard image to obtain a text image feature vector; Based on the text image feature vector, generating a candidate text box; Scoring the candidate text boxes to obtain text box score data; The text area is screened out from the candidate text boxes based on the text box score data.

5. The method according to claim 3, characterized in that: The extracting text from the text area to obtain the text information includes: Serializing the text region to obtain a text sequence, wherein the text sequence includes a text word vector; Performing context information capture processing on the text sequence to obtain a text feature sequence; Based on the text feature sequence, calculating the phrase probability distribution of the text word vector; Based on the phrase probability distribution, the text word vectors are sorted to obtain the text information.

6. The method according to any one of claims 1 to 5, characterized in that: The step of adjusting the prompt word of the extraction indication information based on the text annotation information and the predicted text information to obtain the target indication information includes: Comparing the text annotation information with the predicted text information to obtain an information proofreading result, wherein the information proofreading result indicates whether the text annotation information is the same as or different from the predicted text information; If the information proofreading result indicates that the predicted text information is different from the text annotation information, selecting a target indication information phrase from a preset indication information word library; Performing indicator phrase positioning on the extracted indicator information to obtain an indicator phrase position; Based on the indication phrase position and the indication information phrase, the extracted indication information is phrase-replaced to obtain the target indication information.

7. The method according to any one of claims 1 to 5, characterized in that: The step of adjusting the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model includes: Comparing the text annotation information with the second predicted text information to obtain a second information proofreading result, wherein the second information proofreading result indicates that the text annotation information is the same as or different from the second predicted text information; If the second information proofreading result indicates that the predicted text information is different from the text annotation information, adding a network layer to the image information extraction model to obtain an initial information extraction model; Based on the text information and the text annotation information, the initial information extraction model is trained to obtain the target information extraction model.

8. An image information extraction device, characterized in that: The device comprises: A text image acquisition module is used to acquire a sample image containing text content; A text image recognition module, used to perform text recognition on the sample image to obtain text information; A text information annotation module, used to annotate the text information with key information to obtain text annotation information; A first information extraction module, configured to identify key information of the text information based on preset extraction instruction information and a preset image information extraction model, to obtain first predicted text information; an indication information adjustment module, configured to adjust the extraction indication information based on the text annotation information and the first predicted text information to obtain target indication information; A second information extraction module, configured to perform key information recognition on the text information based on the target indication information and the image information extraction model to obtain second predicted text information; An extraction model adjustment module, used to adjust the image information extraction model based on the text annotation information and the second predicted text information to obtain a target information extraction model; The image information extraction module is used to obtain a target image and extract information from the target image based on the target indication information and the target information extraction model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image information extraction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image information extraction method according to any one of claims 1 to 7 is implemented.

Images