Image information extraction method and device, electronic equipment, and storage medium

Abstract

The image information extraction method and device, the electronic equipment and the storage medium provided by the present application relate to the artificial intelligence technical field and the digital medical field. The method comprises the following steps: acquiring a first image, and acquiring a category of the first image to obtain a target image category; acquiring a second image according to the target image category; performing first segmentation on the first image according to a preset size to obtain at least two first regions; performing second segmentation on the second image according to the preset size to obtain at least two second regions; inputting the first regions and the second regions into a preset attention model for semantic learning to obtain third regions; splicing the third regions to obtain a target mask image; performing image segmentation on the first image according to the target mask image to obtain a target information region; and performing character recognition on the target information region to obtain target character information. The present application improves the extraction efficiency of image information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the field of digital medicine, and in particular relates to an image information extraction method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the process of intelligent medical treatment or insurance business, the name, gender, birth date, customer number and other information in the materials such as certificates and forms need to be verified. The verification method adopted by the related technology is to first identify all the text information in the entire image through text recognition technology (such as OCR technology), and then use a natural language processing algorithm (such as NLP algorithm) to obtain the verification information from all the text information.

[0003] However, the above verification method has the disadvantages of text recognition on the entire image, which not only greatly affects the efficiency of text recognition, but also leads to the identification of a large amount of unnecessary text information. In addition, using a natural language processing algorithm to extract verification information from all the identified text information will also lead to low extraction efficiency due to a large amount of unnecessary text information.

[0004] It can be seen that the verification method of the related technology has low efficiency in extracting image information. SUMMARY

[0005] The main purpose of the embodiments of the present application is to provide an image information extraction method and device, an electronic device and a storage medium, which can improve the extraction efficiency of image information.

[0006] To achieve the above purpose, a first aspect of the embodiments of the present application provides an image information extraction method, which comprises:

[0007] obtaining a first image and obtaining the category of the first image to obtain a target image category;

[0008] obtaining a second image according to the target image category;

[0009] performing first segmentation on the first image according to a preset size to obtain at least two first regions; wherein the size of each first region is the preset size;

[0010] performing second segmentation on the second image according to the preset size to obtain at least two second regions, wherein the size of each second region is the preset size;

[0011] inputting the first region and the second region into a preset attention model for semantic learning to obtain a third region;

[0012] performing splicing according to the third region to obtain a target mask image;

[0013] perform image segmentation on the first image according to the target mask image to obtain a target information region;

[0014] perform character recognition on the target information region to obtain target character information.

[0015] Optionally, a number of the at least two first regions is a first number, and a number of the at least two second regions is a second number; the inputting the first regions and the second regions into a preset attention model to perform semantic learning to obtain third regions comprises:

[0016] the inputting the first number of the first regions and the second number of the second regions into the preset attention model to perform semantic learning to obtain a third number of candidate regions, wherein the third number is a sum of the first number and the second number;

[0017] the taking out the first number of the candidate regions from the third number of the candidate regions according to a preset order to obtain the first number of the third regions.

[0018] Optionally, the preset attention model comprises a weight convolution layer and a first convolution layer; the inputting the first number of the first regions and the second number of the second regions into the preset attention model to perform semantic learning to obtain a third number of candidate regions comprises:

[0019] taking the first number of the first regions as first target regions alternately;

[0020] performing first convolution on the first target regions by the first convolution layer to obtain first reference vectors;

[0021] performing convolution on the first target regions and the second number of the second regions by the weight convolution layer to obtain a second number of attention weights;

[0022] performing weighting sum on the first reference vectors according to the second number of the attention weights to obtain the candidate regions.

[0023] Optionally, the weight convolution layer comprises a second convolution layer and a third convolution layer; the performing convolution on the first target regions and the second number of the second regions by the weight convolution layer to obtain a second number of attention weights comprises:

[0024] performing second convolution on the first target regions by the second convolution layer to obtain second reference vectors;

[0025] performing third convolution on the second number of the second regions by the third convolution layer to obtain a third number of third reference vectors.

[0026] normalizing the product of the second reference vector and each of the third reference vectors to obtain the second number of attention weights.

[0027] Optionally, the obtaining the second image according to the target image category comprises:

[0028] determining a target information extraction type according to the target image category;

[0029] constructing the second image according to the target information extraction type.

[0030] Optionally, the constructing the second image according to the target information extraction type comprises:

[0031] extracting a target field from the preset database according to the target information extraction type;

[0032] generating the second image according to the target field.

[0033] Optionally, before the inputting the first region and the second region into the preset attention model to perform semantic learning to obtain a third region, the method further comprises:

[0034] training the preset attention model, specifically comprising:

[0035] constructing a sample set, wherein samples in the sample set comprise a first sample image, a second sample image, and a label mask image;

[0036] performing first sample segmentation on the first sample image according to a preset sample size to obtain at least two first sample regions, wherein a size of each first sample region is the preset sample size;

[0037] performing second sample segmentation on the second sample image according to the preset sample size to obtain at least two second sample regions, wherein a size of each second sample region is the preset sample size;

[0038] inputting the first sample region and the second sample region into an initial attention model to perform semantic learning to obtain a third sample region;

[0039] performing splicing according to the third sample region to obtain a sample mask image;

[0040] calculating an image difference between the label mask image and the sample mask image to determine loss data;

[0041] adjusting parameters of the initial attention model according to the loss data to obtain the preset attention model.

[0042] To achieve the above object, a second aspect of the embodiment of the present application provides an image information extraction device, which comprises:

[0043] a first image acquisition module, configured to acquire a first image and a category of the first image, and obtain a target image category;

[0044] a second image acquisition module, configured to acquire a second image according to the target image category;

[0045] a first segmentation module, configured to perform first segmentation on the first image according to a preset size, and obtain at least two first regions; wherein the size of each first region is the preset size;

[0046] a second segmentation module, configured to perform second segmentation on the second image according to the preset size, and obtain at least two second regions; wherein the size of each second region is the preset size;

[0047] a semantic learning module, configured to input the first regions and the second regions into a preset attention model for semantic learning, and obtain third regions;

[0048] an image splicing module, configured to splice the third regions, and obtain a target mask image;

[0049] an image segmentation module, configured to perform image segmentation on the first image according to the target mask image, and obtain a target information region;

[0050] a character recognition module, configured to perform character recognition on the target information region, and obtain target character information.

[0051] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the image information extraction method of the first aspect when executing the computer program.

[0052] To achieve the above object, a fourth aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the image information extraction method of the first aspect.

[0053] The image information extraction method, apparatus, electronic device, and storage medium proposed in this application embodiment address the issue that, in addition to the target information region, the first image generally contains other information regions or blank regions. These regions are meaningless for information extraction. Therefore, instead of recognizing the entire first image, this application embodiment first segments the first image into a first region and the second image into a second region. Then, an attention model is used to perform semantic learning on the first and second regions to generate a target mask image. Based on the target mask image, the target information region is segmented from the first image, and text recognition is then performed on the target information region to directly obtain the target text information. This reduces the extraction of other redundant information and significantly improves the efficiency of image information extraction. Attached Figure Description

[0054] Figure 1 This is a flowchart of the image information extraction method provided in the embodiments of this application;

[0055] Figure 2A This is a schematic diagram of the first image provided in the embodiments of this application;

[0056] Figures 2B-2D This is a schematic diagram of the second image provided in the embodiments of this application;

[0057] Figure 3 yes Figure 1 The flowchart for step 102 in the document;

[0058] Figure 4 yes Figure 3 The flowchart for step 202 in the document;

[0059] Figure 5 This is a schematic diagram illustrating the specific implementation process of generating the second image provided in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram illustrating the specific implementation process of performing a first segmentation on a first image according to an embodiment of this application;

[0061] Figure 7 This is a schematic diagram illustrating the specific implementation process of performing a second segmentation on a second image according to an embodiment of this application;

[0062] Figure 8 yes Figure 1 The flowchart for step 105 in the document;

[0063] Figure 9 yes Figure 8 The flowchart for step 401 in the document;

[0064] Figure 10 yes Figure 9 The flowchart for step 503 in the document;

[0065] Figure 11 is a specific implementation process schematic diagram for obtaining a third region by using a first region and a second region provided by an embodiment of the present application;

[0066] Figure 12 is a specific implementation process schematic diagram for obtaining a target mask image by using a third region provided by an embodiment of the present application;

[0067] Figure 13 is a specific implementation process schematic diagram for image segmentation and character recognition provided by an embodiment of the present application;

[0068] Figure 14 is a flowchart of an image information extraction method provided by another embodiment of the present application;

[0069] Figure 15 is a module structure block diagram of an image information extraction device provided by an embodiment of the present application;

[0070] Figure 16 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0072] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0074] In order to more clearly understand the technical solutions provided by the embodiments of the present application, first, some terms involved in the present application are analyzed:

[0075] Artificial intelligence (AI): It is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0076] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, also known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0077] Optical character recognition technology (OCR) refers to the process of checking characters printed on paper by electronic devices (such as scanners or digital cameras), detecting light and dark patterns to determine their shape, and then translating the shape into computer text using character recognition methods; that is, for printed characters, use optical methods to convert the characters in paper documents into black and white bitmap image files, and convert the characters in the image into text format through recognition software for further editing and processing by word processing software.

[0078] In the related art, when extracting verification information from an image, the entire image needs to be subjected to character recognition, which not only greatly affects the efficiency of character recognition, but also leads to the recognition of a large amount of unnecessary text information. In addition, using a natural language processing algorithm to extract verification information from all recognized text information will also result in low extraction efficiency due to the presence of a large amount of unnecessary text information.

[0079] Therefore, there is an urgent need for a technology that can accelerate the extraction efficiency of image information to improve the efficiency of information extraction.

[0080] The image information extraction method provided by the embodiments of the present application can be applied to a server end, and can also be software running in a terminal or a server end. The server end can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The software can be an application that implements the image information extraction method, but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0082] The embodiments of the present application provide an image information extraction method and device, an electronic device, and a storage medium. The embodiments are specifically described as follows. First, the image information extraction method in the embodiments of the present application is described.

[0083] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of the data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0084] Figure 1 is an optional flowchart of the image information extraction method provided by the embodiments of the present application, which can include but is not limited to steps 101 to 108.

[0085] Step 101, obtaining a first image, and obtaining a category of the first image, to obtain a target image category;

[0086] Step 102, obtaining a second image according to the target image category;

[0087] Step 103, performing first segmentation on the first image according to a preset size, to obtain at least two first regions; wherein a size of each first region is the preset size;

[0088] Step 104, performing second segmentation on the second image according to the preset size, to obtain at least two second regions; wherein a size of each second region is the preset size;

[0089] Step 105, inputting the first regions and the second regions into a preset attention model for semantic learning, to obtain third regions;

[0090] Step 106, performing splicing according to the third regions, to obtain a target mask image;

[0091] Step 107, performing image segmentation on the first image according to the target mask image, to obtain a target information region;

[0092] Step 108, performing character recognition on the target information region, to obtain target character information.

[0093] The steps 101 to 108 shown in the embodiments of the present application, since the first image generally contains other information regions or blank regions in addition to the target information region, these regions are regions without extraction significance for information extraction. The embodiments of the present application do not perform recognition based on the whole first image, but first segment the first image into first regions, and segment the second image into second regions, then perform semantic learning on the first regions and the second regions through an attention model, so as to generate a target mask image. Then, based on the target mask image, the target information region is segmented from the first image, and then character recognition is performed on the target information region, so that the target character information can be directly obtained, the extraction of other redundant information is reduced, and the efficiency of image information extraction is greatly improved.

[0094] In step 101 of some embodiments, the first image refers to an image capable of triggering an image information extraction service. Generally, the first image is an image containing character information. For example, in a medical application scenario, the first image is a certificate image (such as an ID card image), or a case text image, or an insurance policy image. These first images generally contain the basic information of the object, which includes, for example, name, gender, identity number, etc.

[0095] In an embodiment, the acquisition method of the first image includes but is not limited to the following methods:

[0096] (1) Obtain a first image from an image database.

[0097] (2) Start the camera to collect images and obtain a first image.

[0098] (3) Obtain an image uploaded by the object from a preset image interface to obtain a first image.

[0099] After obtaining the first image, the category of the first image needs to be obtained to obtain a target image category.

[0100] The category of the first image can be obtained by a preset classification model or manual annotation to obtain the target image category. The preset classification model refers to a model for classifying images, such as a Bert model. The target image category refers to a label that can distinguish different first images. For example, if the first image is a certificate image, the target image category is a certificate category. For another example, if the first image is a case text image, the target image category is a case category. For another example, if the first image is an insurance policy image, the target image category is a policy category.

[0101] In step 102 of some embodiments, a second image is obtained according to the target image category. The second image refers to an image that can generate a mask image together with the first image. As described above, the first image is generally an image containing text information, and at least part of the text information in the text information will be reflected on the second image, so that the text information on the first image can be located according to the second image. For example, referring to Figure 2A , the first image is a certificate image, and generally, the certificate image will have the text of name, birth, and identity number. Referring to Figure 2B , the second image is a text image composed of the "name" field. Referring to Figure 2C , the second image is a text image composed of the "birth" field. Referring to Figure 2D , the second image is a text image composed of the "identity number" field.

[0102] In another embodiment, referring to Figure 3 , step 102 can include but is not limited to steps 201 to 202:

[0103] Step 201, determining a target information extraction type according to the target image category;

[0104] Step 202, constructing a second image according to the target information extraction type.

[0105] The advantage of this embodiment is that considering that the information contained in different first images is not the same, this embodiment first determines the target information extraction type according to the target image category, and then constructs the second image, which can adapt to a large number of and different first images, and improve the efficiency and accuracy of obtaining the second image.

[0106] In step 201 of some embodiments, the target information extraction type refers to the type of target information to be extracted. As shown in the following table, in the case of the first image being a certificate image, the information that can be extracted from the first image includes but is not limited to name, birth date and identity number, and therefore the target information extraction type includes but is not limited to name type, birth date type and identity number type. Figure 2A

[0107] In step 202 of some embodiments, the second image can be obtained from a preset image database according to the target information extraction type. At this time, the preset image database needs to store the second image in advance, for example, the preset database stores the second image in advance as shown in the following table: Figure 2B 、 2C , 2D second image, which occupies a certain storage space.

[0108] In another embodiment, referring to Figure 4 , step 202 can include but is not limited to steps 301 to 302:

[0109] Step 301, obtaining the target field from the preset database according to the target information extraction type;

[0110] Step 302, generating the second image according to the target field.

[0111] The advantage of this embodiment is that the preset database does not need to store the second image in advance, but generates the second image according to the target field after determining the target field, saving the storage space of the preset database.

[0112] It should be noted that in step 202, the target field can be input into an existing text-to-image model for image generation to obtain the second image. The text-to-image model here refers to a model that converts input text into an image.

[0113] For example, referring to Figure 5 , the target field obtained from the preset database according to the target information extraction type being the name type is the name, and the name is input into the text-to-image model, and the output of the model is a second image with the word "name" on it.

[0114] In step 103 of some embodiments, the first image is first divided according to a preset size to obtain at least two first regions; wherein the size of each first region is the preset size, and any two first regions do not overlap. For example, referring to Figure 6 ​, assuming that the size of the first image is 60*40 pixels and the preset size is 20*20 pixels, the first image is divided into 6 first regions, denoted as a first region q1, a first region q2, a first region q3, a first region q4, a first region q5, and a first region q6, after the first division.

[0115] In an embodiment, the step 103 can include:

[0116] determining a first target size according to the preset size and the first size coefficient;

[0117] transforming the size of the first image into the first target size to obtain a transformed first image;

[0118] performing a first division on the transformed first image according to the preset size to obtain at least two first regions.

[0119] For example, the preset size is 20*20 pixels and the first size coefficient is 3*1, and then the first target size is 60*20 pixels. For another example, the size of the first image is 40*10 pixels, and then the size of the transformed first image is 60*20 pixels. After the first division, the transformed first image is divided into 3 first regions, and the size of each of the 3 first regions is 20*20 pixels.

[0120] It should be noted that if the size of the first image is not transformed, the size of the first region needs to be transformed separately. For example, the size of the first image is 40*10 pixels, and then 2 first regions are obtained after the first division, and the size of each of the 2 first regions is 20*10 pixels. Then the size of the first region with the size of 20*10 pixels is transformed to obtain a first region with the size of 20*20 pixels.

[0121] The advantage of this embodiment is that no matter what the size of the first image is, the size of the first image is transformed to obtain a transformed first image with a first target size. In this way, when the transformed first image is divided according to the preset size, first regions with the preset size can be quickly obtained, and the size of the first region does not need to be adjusted separately, thereby improving the division efficiency.

[0122] In the step 104 of some embodiments, a second division is performed on the second image according to the preset size to obtain at least two second regions, wherein the size of each second region is the preset size, and any two second regions do not overlap with each other. For example, referring to Figure 7 , assuming that the size of the second image is 40*20 pixels and the preset size is 20*20 pixels, the second image is divided into 2 second regions, denoted as a first region w1 and a first region w2, after the second division.

[0123] In an embodiment, step 104 can include:

[0124] determining a second target size according to the preset size and the second size coefficient;

[0125] transforming the size of the second image into the second target size to obtain a transformed second image;

[0126] performing a second segmentation on the transformed second image according to the preset size to obtain at least two second regions.

[0127] The embodiment has the advantage that no matter what the size of the second image is, the size of the second image is transformed to obtain a transformed first image with a second target size, so that when the transformed second image is segmented according to the preset size, the second regions with the preset size can be quickly obtained, and the size of the second regions does not need to be adjusted separately, thereby improving the segmentation efficiency.

[0128] In step 105 of some embodiments, the first regions and the second regions are input into a preset attention model for semantic learning to obtain third regions.

[0129] Referring to Figure 8 In an embodiment, the number of the at least two first regions is a first number, the number of the at least two second regions is a second number, and step 105 includes but is not limited to steps 401 to 402:

[0130] Step 401: inputting the first number of first regions and the second number of second regions into a preset attention model for semantic learning to obtain a third number of candidate regions, wherein the third number is the sum of the first number and the second number;

[0131] Step 402: extracting the first number of candidate regions from the third number of candidate regions according to a preset order to obtain the first number of third regions.

[0132] The embodiment has the advantage that the attention model is used for semantic learning, and the first number of third regions is extracted from the third number of candidate regions, which greatly improves the accuracy of the third regions and helps to splice the first number of third regions to obtain a target mask image with higher accuracy, thereby improving the accuracy of the target information region segmented from the image.

[0133] Referring to Figure 9 In an embodiment, the preset attention model includes a weight convolution layer and a first convolution layer, and step 401 includes but is not limited to steps 501 to 504:

[0134] Step 501: taking the first number of first regions as first target regions alternately;

[0135] Step 502, performing first convolution on the first target region through a first convolution layer to obtain a first reference vector;

[0136] Step 503, performing convolution on the first target region and the second quantity of second regions through a weight convolution layer to obtain a second quantity of attention weights;

[0137] Step 504, weighting and summing the first reference vector according to the second quantity of attention weights to obtain a candidate region.

[0138] The advantage of this embodiment is that, based on the first convolution layer and the weight convolution layer, attention weights from the second quantity of second regions can be applied to each first region, reflecting the connection and influence between the second regions and the first region, greatly improving the extraction accuracy of the third region, thereby improving the accuracy of the mask image obtained by splicing the third region.

[0139] In step 501 of some embodiments, there are a total of 6 first regions, that is, first regions q1-q6, and there are 6 first target regions corresponding to them, denoted as first target regions d1-d6.

[0140] In step 502 of some embodiments, the first convolution layer refers to a module for performing convolution processing on the first target region. For the first target region d1, the first convolution layer performs first convolution on the first target region d1 to obtain the first reference vector g1. Similarly, the first convolution layer performs first convolution on the first target region d2 to obtain the first reference vector g2; the first convolution layer performs first convolution on the first target region d3 to obtain the first reference vector g3; the first convolution layer performs first convolution on the first target region d4 to obtain the first reference vector g4; the first convolution layer performs first convolution on the first target region d5 to obtain the first reference vector g5; and the first convolution layer performs first convolution on the first target region d6 to obtain the first reference vector g6.

[0141] In step 503 of some embodiments, the weight convolution layer refers to a module for performing convolution processing on the first target region and the second quantity of second regions.

[0142] In an embodiment, referring to Figure 10 , the weight convolution layer includes a second convolution layer and a third convolution layer, and step 503 includes but is not limited to steps 601-603:

[0143] Step 601, performing second convolution on the first target region through the second convolution layer to obtain a second reference vector;

[0144] Step 602, performing third convolution on the second quantity of second regions through the third convolution layer to obtain a second quantity of third reference vectors;

[0145] In step 603, the product of the second reference vector and each of the second number of third reference vectors is normalized to obtain a second number of attention weights.

[0146] In step 601 of some embodiments, the second convolutional layer refers to a module for performing convolutional processing on the first target region. For example, for the first target region d1, the second convolutional layer performs a second convolution on the first target region d1 to obtain a second reference vector h1.

[0147] In step 602 of some embodiments, the third convolutional layer refers to a module for performing convolutional processing on the second region. For the second region w1, the third convolutional layer performs a third convolution on the second region w1 to obtain a third reference vector k1. Similarly, the third convolutional layer performs a third convolution on the second region w2 to obtain a third reference vector k2.

[0148] In step 603 of some embodiments, the product of the second reference vector and each of the second number of third reference vectors is normalized to obtain a second number of attention weights. For example, for the first target region d1, the product of the second reference vector h1 and the second region w1 is normalized to obtain an attention weight z 11 ; the product of the second reference vector h1 and the second region w2 is normalized to obtain an attention weight z 12 .

[0149] The advantage of this embodiment is that, based on the second convolutional layer and the third convolutional layer, an attention weight from the second number of second regions can be applied to each first region, further fully reflecting the connection and influence between the second region and the first region, and greatly improving the calculation accuracy of the attention weight.

[0150] In step 504 of some embodiments, the first reference vector is weighted and summed according to the second number of attention weights to obtain a third region. For example, for the first target region d1, the first reference vector g1 is weighted and summed according to the attention weights z 11 and z 12 , that is, z 11 *g1+z 12 *g2, to obtain a candidate region e1.

[0151] Referring to Figure 11In one example, six first regions and two second regions are input into an attention model. After semantic learning, the attention model outputs a sequence of eight candidate regions: e1, e2, e3, e4, e5, e6, e7, and e8. These eight candidate regions are then selected sequentially according to a preset order, for example, in... Figure 11 In this context, the preset order specifically refers to the order in which the candidate regions are arranged from front to back in the output sequence, resulting in 6 third regions, including third region r1, third region r2, third region r3, third region r4, third region r5, and third region r6.

[0152] The above is a detailed description of step 105.

[0153] In step 106 of some embodiments, the target mask image is obtained by stitching together the images based on the third region. For example, refer to... Figure 12 The third regions r1, r2, r3, r4, r5, and r6 are stitched together in the order of the first regions to obtain the target mask image.

[0154] In step 107 of some embodiments, the first image is segmented based on the target mask image to obtain the target information region. Then, in step 108, text recognition is performed on the target information region to obtain the target text information.

[0155] For example, refer to Figure 13 The target information region is segmented from the first image based on the target mask image. Then, a character recognition model is used to recognize the characters in the target information region, yielding the target text information as a "pronoun". For example, the second image... Figure 2C As shown, by using a character recognition model to recognize the text in the target information region, the target text information can be obtained as "June 6, 2000". For example, the second image... Figure 2D As shown, by using a character recognition model to recognize the target information region, the target text information can be obtained as "123456789123".

[0156] It should be noted that the above text recognition model can be an OCR model or a text recognition network model (Convolutional Recurrent Neural Network, CRNN).

[0157] It can be seen that the embodiment of the present application does not need to perform character recognition on the entire first image, but performs character recognition on the target information region segmented from the first image, thereby reducing the area required for character recognition, improving the efficiency of character recognition, and thus improving the efficiency of extracting information from the image. In addition, the embodiment of the present application also generates a third region based on the second image to obtain a target mask image. Different second images result in different target mask images, which not only improves the flexibility of the target mask image, but also improves the flexibility of image information extraction.

[0158] The above is a detailed description of steps 101 to 108. The training process of the preset attention model is described in detail below.

[0159] Referring to Figure 14 In an embodiment, before step 107, the image information extraction method of the embodiment further includes training a preset attention model, specifically including:

[0160] Step 701, a sample set is constructed, and the samples in the sample set include a first sample image, a second sample image, and a label mask image;

[0161] Step 702, the first sample image is first sample segmented according to a preset sample size, to obtain at least two first sample regions; wherein the size of each first sample region is the preset sample size;

[0162] Step 703, the second sample image is second sample segmented according to the preset sample size, to obtain at least two second sample regions, wherein the size of each second sample region is the preset sample size;

[0163] Step 704, the first sample region and the second sample region are input to an initial attention model for semantic learning, to obtain a third sample region;

[0164] Step 705, the third sample region is spliced to obtain a sample mask image;

[0165] Step 706, an image difference between the label mask image and the sample mask image is calculated to determine loss data;

[0166] Step 707, the parameters of the initial attention model are adjusted according to the loss data to obtain a preset attention model.

[0167] Specifically, the sample set in step 701 includes a plurality of samples. Each sample includes a first sample image, a second sample image, and a label mask image. The more the number of samples, the better the training effect. The first sample image here is similar to the first image of step 101 described above, and the second sample image is similar to the second image of step 102 described above, except that the first sample image and the second sample image are used for model training, and the first image and the second image are used for actual use of the model. Note that the label corresponding to the sample is generally required during model training, and the label mask image in the sample is used as the label in this embodiment.

[0168] Steps 702 to 705 are similar to the processes of steps 103 to 106 described above, and will not be described again.

[0169] In step 706, the formula for calculating the image difference between the label mask image and the sample mask image can be selected as the minimum absolute value deviation formula, also known as L1 loss. L1 loss is generally the error obtained by taking the absolute value of the difference between the target value and the model output (estimated value). In this embodiment, the absolute value of the vector difference between the vector representation of the label mask image and the vector representation of the sample mask image is taken as the image difference, and then the loss data is determined according to the image difference. The loss data is a function for measuring the judgment loss of the preset attention model, and the smaller the loss function, the better the preset attention model is trained.

[0170] In an embodiment, step 706 includes:

[0171] determining the image difference for each sample;

[0172] averaging the image differences of each sample in the sample set to obtain an average image difference;

[0173] taking the average image difference as the loss function.

[0174] In step 707, the initial attention model has the same structure as the preset attention model, but the parameters are different. The parameters of the initial attention model are adjusted to obtain the preset attention model.

[0175] The advantage of this embodiment is that the parameters of the initial attention model are adjusted by the loss data calculated from the label mask image and the sample mask image, which improves the processing performance of the preset attention model. In addition, if the label mask image is different, the obtained preset attention model is different, which improves the flexibility of the preset attention model.

[0176] Please refer to Figure 15 The embodiment of the present application also provides an image information extraction device, which can implement the image information extraction method described above, Figure 15A module structure block diagram of an image information extraction device provided by the embodiment of the application is provided, and the device comprises: a first image acquisition module 801, a second image acquisition module 802, a first segmentation module 803, a second segmentation module 804, a semantic learning module 805, an image splicing module 806, an image segmentation module 807 and a character recognition module 808. The first image acquisition module 801 is configured to acquire a first image and acquire a category of the first image to obtain a target image category. The second image acquisition module 802 is configured to acquire a second image according to the target image category. The first segmentation module 803 is configured to perform first segmentation on the first image according to a preset size to obtain at least two first regions. The size of each first region is the preset size. The second segmentation module 804 is configured to perform second segmentation on the second image according to the preset size to obtain at least two second regions. The size of each second region is the preset size. The semantic learning module 805 is configured to input the first regions and the second regions into a preset attention model to perform semantic learning to obtain third regions. The image splicing module 806 is configured to splice the third regions to obtain a target mask image. The image segmentation module 807 is configured to perform image segmentation on the first image according to the target mask image to obtain a target information region. The character recognition module 808 is configured to perform character recognition on the target information region to obtain target character information.

[0177] It should be noted that the specific implementation of the image information extraction device is basically the same as the specific embodiment of the image information extraction method described above, and will not be repeated here.

[0178] The embodiment of the application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and capable of running on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the image information extraction method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0179] Please refer to Figure 16 , Figure 16 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0180] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application specific integrated circuit (ASIC) or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiment of the application.

[0181] The memory 902 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the image information extraction method of the embodiments of the present application.

[0182] The input / output interface 903 is configured to realize information input and output.

[0183] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0184] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0185] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0186] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and is configured to store computer readable information. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image information extraction method.

[0187] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0188] The image information extraction method, the image information extraction device, the electronic equipment and the storage medium provided by the embodiments of the present application can greatly improve the efficiency of image information extraction because the first image generally contains other information regions or blank regions in addition to the target information region, and these regions are meaningless for information extraction. The embodiments of the present application do not identify based on the whole first image, but first divide the first image into a first region and the second region into a second region, and then perform semantic learning on the first region and the second region through an attention model to generate a target mask image. Then, the target information region is segmented from the first image based on the target mask image, and then the target information region is subjected to character recognition, so that the target character information can be directly obtained, the extraction of other redundant information is reduced, and the efficiency of image information extraction is greatly improved.

[0189] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0190] Those skilled in the art can understand that, Figure 1 , Figures 3-4 , Figures 8-10 , Figure 14 The technical solutions shown in the above description do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown, or combine certain steps, or different steps.

[0191] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0192] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0193] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect between similar objects.

[0194] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0195] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0197] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist alone physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0198] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0199] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An image information extraction method characterized by comprising: The method comprises: acquiring a first image and acquiring a category of the first image to obtain a target image category; acquiring a second image according to the target image category; performing first segmentation on the first image according to a preset size to obtain a first number of first regions, wherein the size of each first region is the preset size; performing second segmentation on the second image according to the preset size to obtain a second number of second regions, wherein the size of each second region is the preset size; inputting the first number of first regions and the second number of second regions into a preset attention model for semantic learning to obtain a third number of candidate regions, wherein the third number is the sum of the first number and the second number, and the preset attention model comprises a weight convolution layer and a first convolution layer; extracting the first number of candidate regions from the third number of candidate regions according to a preset order to obtain a third number of third regions; performing splicing according to the third regions to obtain a target mask image; performing image segmentation on the first image according to the target mask image to obtain a target information region; performing character recognition on the target information region to obtain target character information; wherein inputting the first number of first regions and the second number of second regions into a preset attention model for semantic learning to obtain a third number of candidate regions comprises: alternately taking the first number of first regions as first target regions; performing first convolution on the first target regions through the first convolution layer to obtain first reference vectors; performing convolution on the first target regions and the second number of second regions through the weight convolution layer to obtain the second number of attention weights; performing weighting and summing on the first reference vectors according to the second number of attention weights to obtain the candidate regions.

2. The image information extraction method according to claim 1, characterized by, The weight convolution layer comprises a second convolution layer and a third convolution layer; performing convolution on the first target regions and the second number of second regions through the weight convolution layer to obtain the second number of attention weights comprises: performing second convolution on the first target regions through the second convolution layer to obtain second reference vectors; performing third convolution on the second number of second regions through the third convolution layer to obtain the second number of third reference vectors; performing normalization processing on the product of the second reference vectors and the second number of third reference vectors respectively to obtain the second number of attention weights.

3. The image information extraction method according to any one of claims 1 to 2, characterized by, The acquiring a second image according to the target image category comprises: determining a target information extraction type according to the target image category; constructing the second image according to the target information extraction type.

4. The image information extraction method according to claim 3, characterized by, The constructing the second image according to the target information extraction type comprises: extracting a target field from a preset database according to the target information extraction type; generating the second image according to the target field.

5. The image information extraction method according to any one of claims 1 to 2, characterized by, Before the first quantity of the first regions and the second quantity of the second regions are input into a preset attention model for semantic learning to obtain a third quantity of candidate regions, the method further comprises: training the preset attention model, specifically comprising: constructing a sample set, wherein samples in the sample set comprise a first sample image, a second sample image, and a label mask image; performing first sample segmentation on the first sample image according to a preset sample size to obtain at least two first sample regions, wherein the size of each first sample region is the preset sample size; performing second sample segmentation on the second sample image according to the preset sample size to obtain at least two second sample regions, wherein the size of each second sample region is the preset sample size; inputting the first sample regions and the second sample regions into an initial attention model for semantic learning to obtain third sample regions; performing splicing according to the third sample regions to obtain a sample mask image; calculating the image difference between the label mask image and the sample mask image to determine loss data; adjusting the parameters of the initial attention model according to the loss data to obtain the preset attention model.

6. An image information extraction apparatus characterized by comprising: The device comprises: a first image acquisition module configured to acquire a first image and a category of the first image to obtain a target image category; a second image acquisition module configured to acquire a second image according to the target image category; a first segmentation module configured to perform first segmentation on the first image according to a preset size to obtain a first quantity of first regions, wherein the size of each first region is the preset size; a second segmentation module configured to perform second segmentation on the second image according to the preset size to obtain a second quantity of second regions, wherein the size of each second region is the preset size; a semantic learning module configured to input the first quantity of the first regions and the second quantity of the second regions into a preset attention model for semantic learning to obtain a third quantity of candidate regions, and to obtain a first quantity of third regions from the third quantity of the candidate regions according to a preset order, wherein the third quantity is the sum of the first quantity and the second quantity, and the preset attention model comprises a weight convolution layer and a first convolution layer; an image splicing module configured to perform splicing according to the third regions to obtain a target mask image; an image segmentation module configured to perform image segmentation on the first image according to the target mask image to obtain a target information region; a character recognition module configured to perform character recognition on the target information region to obtain target character information; wherein inputting the first quantity of the first regions and the second quantity of the second regions into a preset attention model for semantic learning to obtain a third quantity of candidate regions comprises: alternately taking the first quantity of the first regions as first target regions; performing first convolution on the first target regions through the first convolution layer to obtain first reference vectors; convolve the first target region and the second quantity of the second regions through the weight convolution layer to obtain the second quantity of attention weights; weight and sum the first reference vector according to the second quantity of the attention weights to obtain the candidate region.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image information extraction method in any one of claims 1 to 5 when executing the computer program.

8. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the image information extraction method in any one of claims 1 to 5.

Images