Image processing

Abstract

Provided in the embodiments of the description are an image processing method and apparatus. The image processing method comprises: acquiring a plurality of page images of a document and a question, and then performing character recognition on the page images, so as to obtain character information; performing text feature extraction on combined text which is obtained by means of combining the character information, so as to obtain a text feature, and also performing visual feature extraction on the page images, so as to obtain visual features of the page images; performing feature fusion processing on a visual feature sequence formed by combining the text feature and the visual features of the page images, so as to obtain a fused feature; and performing character prediction on the fused feature on the basis of the question, and determining answer content for the question on the basis of a character prediction result.

Description

Image processing Technical Field

[0001] This document relates to the field of image processing technology, and in particular to methods and devices for image processing. Background Art

[0002] With the continuous development of digitalization, there are a large number of document recognition and key information extraction situations in all walks of life. Document understanding has gradually become a major demand in many industries and fields. However, in the process of document understanding, different document styles have different levels of difficulty in understanding and extracting key information. That is, simple document styles are easier to understand and easy to extract the required information from the document, while complex document styles are more difficult to understand. For example, the content of the previous page in the document is not finished and needs to be continued on the next page, which means that there is cross-page content in the document. This is a complex document style. In this regard, document processors still face great challenges in document understanding. Summary of the Invention

[0003] One or more embodiments of the present specification provide an image processing method, comprising: acquiring multiple page images and questions of a document, and performing character recognition on each page image to obtain character information; performing text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each page image to obtain visual features of each page image; performing feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fused features; performing character prediction on the fused features based on the question, and determining the answer content of the question based on the character prediction result.

[0004] One or more embodiments of the present specification provide an image processing device, comprising: a character recognition module, configured to obtain multiple page images and questions of a document, and perform character recognition on each page image to obtain character information; a feature extraction module, configured to perform text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and perform visual feature extraction on each page image to obtain visual features of each page image; a feature fusion module, configured to perform feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fused features; and a character prediction module, configured to perform character prediction on the fused features based on the question, and determine the answer content of the question according to the character prediction result.

[0005] One or more embodiments of the present specification provide an image processing device, comprising a processor and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to: obtain multiple page images and questions of a document, and perform character recognition on each page image to obtain character information; perform text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and perform visual feature extraction on each page image to obtain visual features of each page image; perform feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fused features; perform character prediction on the fused features based on the question, and determine the answer content of the question based on the character prediction result.

[0006] One or more embodiments of the present specification provide a computer-readable storage medium for storing computer-executable instructions, which implement the following steps when executed: obtaining multiple page images and questions of a document, and performing character recognition on each page image to obtain character information; performing text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each page image to obtain visual features of each page image; performing feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fused features; performing character prediction on the fused features based on the question, and determining the answer content of the question based on the character prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0008] FIG1 is a schematic diagram of an implementation environment of an image processing method provided by one or more embodiments of this specification.

[0009] FIG2 is a flowchart of an image processing method provided by one or more embodiments of this specification.

[0010] FIG3 is a flowchart of an image processing method applied to a security application scenario provided by one or more embodiments of this specification.

[0011] FIG4 is a schematic diagram of an image processing device embodiment provided by one or more embodiments of this specification.

[0012] FIG5 is a schematic diagram of the structure of an image processing device provided by one or more embodiments of this specification. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0014] The image processing method provided in one or more embodiments of this specification may be applicable to the implementation environment of image processing. Referring to FIG1 , the implementation environment includes at least an image parsing model, which may include a recognition network 101 for character recognition, a text feature extraction network 102, a visual feature extraction network 103, a feature fusion network 104, and an answer determination network 105.

[0015] The image parsing model can be deployed on a server, which can be a single server, a server cluster consisting of several servers, or one or more cloud servers in a cloud computing platform.

[0016] In this implementation environment, after the recognition network 101 obtains multiple page images and questions of the document, it performs character recognition on each page image in the multiple page images to obtain character information, the text feature extraction network 102 performs text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, the visual feature extraction network 103 performs visual feature extraction on each page image to obtain visual features of each page image, the feature fusion network 104 performs feature fusion processing on the visual feature sequence composed of text features and visual features of each page image to obtain fused features, and the answer determination network 105 uses the input question to predict characters on the fused features and determines the answer content of the question based on the character prediction results.

[0017] One or more embodiments of an image processing method provided in this specification are as follows.

[0018] 2 , the image processing method provided in this embodiment specifically includes steps S202 to S208 .

[0019] Step S202: Acquire multiple page images and questions of the document, and perform character recognition on each page image to obtain character information.

[0020] The document in this embodiment refers to a document to be processed. The document can be the same document file (the same document file), such as a case document of a user in a security application project, or a resource data document of a user in a resource service, or specifically a fund data document of a user in a loan service; the document can also be the same web page, that is, the entire web page. The multiple page images refer to page images of multiple pages in a document, that is, the document is a document with multiple pages, and the multiple page images can be page images of multiple consecutive pages in a document, such as multiple page images with page numbers 1, 2, and 3 in a document; the multiple page images of a document in this embodiment can be page images with cross-page content, such as the content of the page image of the first page of the multiple page images is not completed, and continues to be displayed in the page image of the second page, that is, the content is cross-page, that is, multiple cross-page page images of the document.

[0021] Optionally, the multiple page images of the document include multiple page images corresponding to multiple document page numbers in the same document or multiple screenshot images of the web page of the same web page. Specifically, the multiple document page numbers can be multiple consecutive document page numbers, such as document page numbers 1, 2, and 3; the multiple screenshot images can be obtained after continuous screenshots of the web page, that is, the total image content of the multiple screenshot images can constitute the entire web page or a part of the web page, but the total image content of the multiple screenshot images is continuous and uninterrupted; for example, there are multiple document page numbers 1, 2, and 3 in the same document, and the multiple page images corresponding to the multiple document page numbers are the document. The page image of the document page number 1, the page image of the document page number 2, and the page image of the document page number 3 in the document; for example, a screenshot of the web page of the same web page is obtained by taking a screenshot 1 and a screenshot image 2, and the screenshot image 1 and the screenshot image 2 are images obtained by taking screenshots in the order in which the web page content is arranged, that is, the image content composed of the screenshot image 1 and the screenshot image 2 belongs to the web page content, and the image content is continuous and uninterrupted in the web page content; that is, the multiple page images of the document include multiple page images corresponding to multiple document page numbers of the document or multiple screenshot images of the web page; the image processing method provided in this embodiment can be executed based on the image parsing model.

[0022] The question is a question that needs to be answered using multiple page images of the document. For example, if the document is a user's medical record or claim document, the question might be "What is the patient's discharge diagnosis?" There can be one or more questions. Another example is if the document is a user's financial data document, the question might be "How many times has the user been overdue?"

[0023] In actual applications, there is a need to obtain required information through multiple page images of a document. To this end, in order to meet the diverse needs of obtaining required information from multiple page images of a document, a question about multiple page images of a document can be introduced, that is, the answer content of the question is obtained from multiple page images of a document; in an optional implementation provided by this embodiment, the question is obtained in the following manner: obtaining the question input for the multiple page images; or determining the object type to which the object input into the multiple page images belongs, and determining the question based on the object type.

[0024] The objects include institutions or users; the object types include institution types or user types, and the institution type may be the field in which the institution is located, such as the institution type being a resource institution type (insurance institution type), a real estate transaction institution type, etc.; the user type may be a user occupation type, a user growth time type (a type corresponding to the time period of the user's growth time), etc.

[0025] Specifically, in one case, questions can be input for multiple page images. In another case, in order to avoid manual input of objects, improve the flexibility of problem determination, and realize automation and intelligence of problem determination, the object type to which the object for which multiple page images are input belongs can be determined, and the question can be determined in the question pool according to the object type; for example, an insurance institution can input questions for multiple page images, and can also determine the type of insurance institution to which the insurance institution for which multiple page images are input belongs, and determine in the question pool according to the type of insurance institution that questions include "What is the name" and "What is the disease"; for another example, a lending institution can input questions for multiple page images, and can also determine the type of lending institution to which the lending institution for which multiple page images are input belongs, and determine in the question pool according to the type of lending institution that questions include "How many times is the user overdue?" and "What is the user's overdue amount."

[0026] In addition, the problem can also be determined based on the object identifier of the object. For example, different insurance institutions need to know different user information during the claims process, so the problem can be determined in the problem pool corresponding to the object type based on the object identifier.

[0027] In a specific implementation, after obtaining multiple page images and questions of a document, character recognition can be performed on each of the multiple page images to obtain character information; the character information refers to the character information obtained after character recognition; the character recognition can be OCR (Optical Character Recognition); in addition, in the process of performing character recognition on each of the multiple page images to obtain character information, each of the multiple page images can also be input into a character recognition network in an image parsing model to perform character recognition to obtain character information; the character recognition network can be a character recognition algorithm, which can be an OCR recognition algorithm, which can be obtained after the network to be trained is trained using multiple page image samples.

[0028] Step S204 , performing text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on the page images to obtain visual features of the page images.

[0029] The above-mentioned process obtains multiple page images and questions of the document, and performs character recognition on each page image in the multiple page images to obtain character information. In this step, on the one hand, text feature extraction is performed on the spliced ​​text obtained by splicing the character information of each page image to obtain text features. On the other hand, visual feature extraction is performed on each page image to obtain visual features of each page image.

[0030] The text features described in this embodiment refer to the feature vectors of the text modal dimension; the visual features of each page image refer to the feature vectors of each page image in the visual modal dimension, for example, the visual features include page layout features and text visual features, and the page layout features refer to the layout features of the image content of each page image, such as text position features, etc.; the text visual features include text outline features, text color features, text size features, etc.

[0031] In actual applications, there are situations where each page image among multiple page images contains cross-page content. For example, the content of the page image of the first page among multiple page images is unfinished, and continues to be displayed in the page image of the second page, that is, the content spans pages. In response to this, in order to better perform text feature extraction in combination with the context, so that the extracted text features can better characterize the characters in the multiple page images, this embodiment can splice the character information of the above-mentioned page images to obtain spliced ​​text, and perform text feature extraction on the spliced ​​text to obtain text features. In this way, the complete spliced ​​text of multiple page images can be used to avoid information loss caused by extracting text features from the character information of a single page image. Combining the spliced ​​text of multiple page images can better understand the document, making the extracted text features more effective and accurate.

[0032] During the process of extracting text features from the spliced ​​text obtained by splicing the character information, the spliced ​​text can be input into a text feature extraction network in the image parsing model to extract text features and obtain text features. The text feature extraction network here can be a text encoder, specifically a Longformer (long document converter), which is a model that can efficiently process long text. Furthermore, text feature extraction can also be performed directly on the character information of each page image obtained by character recognition to obtain text features for each page image.

[0033] During the specific execution process, since the image capacity of each page image may be large, in order to improve the efficiency of visual feature extraction, visual feature extraction can be performed on each page image to obtain the visual features of each page image. Specifically, each page image can be input into a visual feature extraction network for visual feature extraction to obtain the visual features of each page image; the visual feature extraction network can be composed of one or more visual encoders. When there are multiple visual encoders, each page image can be input into each visual encoder of the multiple visual encoders for visual feature extraction to obtain the visual features of each page image output by each visual encoder; the parameters of each visual encoder can be shared parameters; the visual feature extraction network can be a visual encoder, which can adopt a visual backbone based on CNN (Convolutional Neural Network) or a visual backbone based on Transformer architecture, such as Swin Transformer (deep learning model).

[0034] In actual applications, since the text in multiple page images of a document may be tilted, etc., the character information obtained by character recognition may be erroneous, and further, the text features obtained by text feature extraction of the spliced ​​text may also be erroneous. To this end, in order to avoid errors in the answer content of the question determined based on a single text feature, visual features can be introduced; in an optional implementation provided by this embodiment, in the process of performing visual feature extraction on the page images and obtaining the visual features of the page images, the following operations are performed: performing layout feature extraction on the page images and obtaining the page layout features of the page images; performing text feature extraction on the page images and obtaining the text visual features of the page images, and using the page layout features and the text visual features as the visual features.

[0035] Step S206 , performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain fused features.

[0036] On the one hand, starting from the text dimension, the text features of the spliced ​​text obtained by splicing character information are extracted to obtain text features. On the other hand, starting from the visual dimension, the visual features of each page image are extracted to obtain the visual features of each page image. In this step, in order to improve the accuracy of the subsequent determination of the answer content of the question, the visual feature sequence composed of the text features and the visual features of each page image can be subjected to feature fusion processing to obtain fusion features. In this way, the answer to the question can be determined through the fusion features of the two modalities of text modality and visual modality, thereby improving the comprehensiveness of the answer.

[0037] The visual feature sequence in this embodiment refers to a feature sequence in which the visual features of each page image are arranged in the order of each page. The fusion feature includes a multimodal fusion feature.

[0038] In a specific implementation, the visual feature sequence composed of the text features and the visual features of each page image can be summed at the same feature position to obtain the fused feature; or the visual feature sequence composed of the text features and the visual features of each page image can be input into a feature fusion network for feature fusion processing to obtain the fused feature; in the first optional implementation provided by this embodiment, in the process of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of each page image to obtain the fused feature, the following operation is performed: the text features and the visual feature sequence are input into a feature fusion network for feature fusion to obtain the fused feature; optionally, the feature fusion network includes a fully connected network. The feature fusion network can be a feature fusion network in an image parsing model.

[0039] In actual applications, since text features are features obtained by extracting text features from spliced ​​text, text features do not have layout information of multiple page images. In order to better fuse text features with visual feature sequences and improve the convenience and accuracy of feature fusion, the text features can be character located based on the page layout features in the visual feature sequence, and the visual feature sequence and text features can be feature fused according to the character positioning results to obtain fused features. Specifically, in the second optional implementation provided by this embodiment, in the process of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain fused features, the following operations are performed: based on the page layout features in the visual feature sequence, the text features are character located to obtain character features in the page unit corresponding to the page layout features in the text features; the visual features in the visual feature sequence are feature fused with the character features in the page unit to obtain the fused features.

[0040] The page layout features may be page layout features in the visual features of each page image in the visual feature sequence. The page units corresponding to the page layout features include each page corresponding to the page layout features, i.e., one page unit is each page corresponding to the page layout features. The character features within the page units include the character features within each page unit in the text features, i.e., the character features within each page in the text features.

[0041] Specifically, based on the page layout features in the visual features of each page image in the visual feature sequence, character positioning can be performed on the text features to obtain the character features in the page units corresponding to each page image in the text features, and the visual features of each page image in the visual feature sequence and the character features in each page unit can be fused to obtain fused features.

[0042] In actual application scenarios, invalid features may exist in the visual feature sequence and text features, such as page numbers and headers, which may lead to deviations in the fused features, and may have a negative impact on the subsequent determination of the answer content of the question. In view of this, in order to improve the comprehensiveness and effectiveness of the fused features, the invalid character features in the text features can be detected based on the page layout features in the visual feature sequence, and the visual feature sequence and text features can be updated according to the invalid character features to obtain the target visual feature sequence and target text features. The target text features and the target visual feature sequence are then fused to obtain the fused features.

[0043] In a third optional implementation manner provided by this embodiment, in the process of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain the fused features, the following operations are performed: based on the page layout features in the visual feature sequence, invalid character features in the text features are detected, and the invalid character features are deleted from the text features to obtain the target text features; the visual features of the page images in the visual feature sequence are filled in according to the configuration information to obtain the target visual feature sequence; the target text features and the target visual feature sequence are feature fused to obtain the fused features.

[0044] The invalid character feature refers to a character feature that has no effect in the text feature, for example, an invalid character feature may be a page number feature, a header feature, and the like.

[0045] Specifically, based on the page layout features in the visual features of each page image in the visual feature sequence, invalid character features in the text features can be detected, and the invalid character features can be deleted from the text features to obtain the target text features, and the target visual features corresponding to the invalid character features in the visual features of each page image can be replaced with configuration information to obtain the target visual feature sequence, and the target text features and the target visual feature sequence can be feature fused to obtain the fused features; in the process of feature fusion of the target text features and the target visual feature sequence to obtain the fused features, the target text features can be character located based on the page layout features in the target visual feature sequence to obtain the character features in each page unit in the target text features, and the updated visual features of each page image in the target visual feature sequence and the character features in each page unit can be feature fused to obtain the fused features; the updated visual features of each page image here refer to the updated visual features of each page image obtained after replacing the target visual features in the visual features of each page image with the configuration information.

[0046] On this basis, in an optional implementation provided by this embodiment, the configuration information is obtained in the following manner: determining the target visual feature among the visual features of each page image based on the invalid character feature; using the adjacent background visual feature of the target visual feature as the configuration information; or using the preset visual feature as the configuration information.

[0047] The preset visual feature refers to a pre-set visual feature, such as a white visual feature, a black visual feature, etc. The adjacent background visual feature of the target visual feature includes the background visual feature of the surrounding background of the target visual feature. For example, if the surrounding background of the target visual feature is white, the adjacent background visual feature is the white visual feature.

[0048] Specifically, in the process of determining the target visual feature among the visual features of each page image based on the invalid character feature, the target visual feature corresponding to the invalid character feature may be determined among the visual features of each page image.

[0049] It should be noted that the third optional implementation method for feature fusion processing provided above can also be performed on the basis of the optional implementation method of extracting visual features from the page images to obtain the visual features of the page images.

[0050] Step S208: performing character prediction on the fused features based on the question, and determining the answer content of the question according to the character prediction result.

[0051] The above-mentioned visual feature sequence composed of text features and visual features of each page image is subjected to feature fusion processing to obtain fused features. In this step, character prediction is performed by combining the fused features obtained by combining the text features and the visual feature sequence, and the answer content of the question is determined based on the character prediction results.

[0052] As mentioned above, the image processing method provided in this embodiment can be executed based on an image parsing model. Based on this, the input of the image parsing model may include the multiple page images and the question; and the output of the image parsing model may include the answer content. The answer content includes the answer to the question and answer details. For example, if the question is "how many diseases does a patient have?" and the answer is "two," the answer details may be "Disease 1" and "Disease 2."

[0053] In specific implementation, the question and fusion features can be input into the character prediction network in the image parsing model for character prediction, and the answer content of the question can be determined based on the character prediction results; the character prediction network can be a decoding network or a decoder, and the network structure of the character prediction network can be Bart (Bidirectional and Auto-Regressive Transformers, a Transformer with both bidirectional language modeling and autoregressive mechanism) or a variant structure of Bart.

[0054] In an optional implementation provided by this embodiment, in the process of performing character prediction on the fused features based on the question and determining the answer content of the question according to the character prediction result, the following operation is performed: the fused features and the question of the security project document are input into the decoding network for decoding processing, and the answer to the question and key security content are obtained, so as to perform security application processing based on the answer to the question and the key security content.

[0055] Among them, the decoding network can be a decoding network in an image parsing model. The problem in this embodiment can be a problem with the security project document, specifically a security application problem, that is, a problem with the security application project document. The key security content refers to the key content that specifically describes the answer to the question, and the key security content can be extracted from the character prediction results. The key security content here can be the answer details mentioned in this implementation. The security application project in this embodiment can be a claim project, and the security application processing can be a claim processing.

[0056] In actual applications, the answer content may include the actual answer to the question and answer details that specifically describe the answer to the question. Different objects, different object types or different question classifications may result in different answer contents to the question. In this regard, in order to improve the flexibility of determining the answer content of the question, this embodiment provides an optional implementation method. In the process of determining the answer content of the question based on the character prediction result, the following operations are performed: based on the object type and / or the question classification of the question, a reply strategy for answering the question is determined; based on the reply strategy and the character prediction result, the question answer to the question is generated and / or the answer details of the question answer are extracted from the character prediction result.

[0057] Furthermore, in order to improve the degree of refinement of the reply strategy, this embodiment provides an optional implementation method, in the process of determining the reply strategy for replying to the question based on the object type and / or the question classification of the question, the following operations are performed: if the object type is the first object type and the question classification is the first question classification, the reply strategy is determined to be the answer reply strategy and the detail reply strategy; if the object type is the second object type or the question classification is the second question classification, the reply strategy is determined to be the answer reply strategy.

[0058] Among them, the answer reply strategy refers to a reply strategy for generating an answer to a question; the detail reply strategy refers to a reply strategy for extracting the answer details of the answer to the question from the character prediction result. The first question classification can be a question type corresponding to other questions except user attribute questions, and the second question classification can be a question type corresponding to user attribute questions, such as user attribute questions including what is the name and how long is the growth time. The object in this embodiment can be an organization or a user; the first object type can be a resource organization type, such as an insurance organization, and the second object type can be a remaining organization type except the resource organization type or a user type of the user.

[0059] Specifically, if the object type is the first object type and the question classification is the first question classification, the reply strategy is determined to be the answer reply strategy and the detail reply strategy, and the question answer to the question is generated according to the reply strategy and the character prediction result, and the answer details of the question answer are extracted from the character prediction result. If the object type is the second object type or the question classification is the second question classification, the reply strategy is determined to be the answer reply strategy, and the question answer to the question is generated according to the reply strategy and the character prediction result.

[0060] In addition, in the process of determining the answer content of the question based on the character prediction results, the answer to the question can also be generated based on the reply strategy and the character prediction results and / or the answer details of the answer to the question can be extracted from the character prediction results. The reply strategy here can be a reply strategy input at the same time as inputting multiple page images and questions, that is, obtaining multiple page images and questions of the document can be replaced by obtaining multiple page images of the document, questions and reply strategies.

[0061] In summary, the present embodiment provides one or more image processing methods, first, obtaining multiple page images and questions of a document, and performing character recognition on each page image to obtain character information, secondly, performing text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each page image to obtain visual features of each page image, thirdly, based on each page layout feature in the visual feature sequence, performing character positioning on the text features to obtain character features within the page unit corresponding to each page layout feature in the text features, performing feature fusion on the visual features in the visual feature sequence and the character features within the page unit to obtain fused features, thereby recognizing the character features within the page unit in the text features with the help of the page layout features in the visual feature sequence, realizing feature alignment of the visual features and the character features within the page unit, and improving the effectiveness and accuracy of the fused features; or, based on each page layout feature in the visual feature sequence, detecting invalid character features in the text features. Features are detected, and invalid character features are deleted from the text features to obtain target text features, the visual features of each page image in the visual feature sequence are filled in according to the configuration information to obtain the target visual feature sequence, the target text features and the target visual feature sequence are fused to obtain fused features, and the feature deviation of the target text features and the target visual feature sequence is reduced by the invalid character features, thereby improving the accuracy of the fused features; finally, character prediction is performed on the fused features based on the question, and a response strategy for answering the question is determined based on the object type (which can be determined based on the input of multiple page images and the object of the question) and the question classification of the question, and the answer to the question is generated according to the response strategy and the character prediction result and / or the answer details of the answer to the question are extracted from the character prediction result, thereby improving the flexibility and diversity of determining the answer content through the response strategy, realizing intelligent understanding of cross-page images such as multiple page images, and obtaining the answer content of the question from the intelligent understanding result, thereby improving the convenience of recognizing cross-page images.

[0062] The following further illustrates the image processing method provided by this embodiment by taking the application of an image processing method provided by this embodiment in a security application scenario as an example. Referring to FIG3 , the image processing method applied to the security application scenario specifically includes the following steps.

[0063] Step S302: Acquire multiple page images and questions of the security application document, and perform character recognition on each page image to obtain character information.

[0064] Step S304 , performing text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each page image to obtain visual features of each page image.

[0065] Step S306 , based on each page layout feature in the visual feature sequence composed of the visual features of each page image, invalid character features in the text features are detected, and the invalid character features are deleted from the text features to obtain target text features.

[0066] Step S308 : determining a target visual feature among the visual features of each page image based on the invalid character feature, and using the adjacent background visual feature of the target visual feature as configuration information.

[0067] Step S310 , filling the visual features of each page image in the visual feature sequence according to the configuration information to obtain a target visual feature sequence.

[0068] Step S312: performing feature fusion on the target text feature and the target visual feature sequence to obtain a fused feature.

[0069] In step S314, character prediction is performed on the fused features based on the question, and the answer content of the question is determined according to the character prediction result, so as to perform guarantee processing based on the answer content.

[0070] The protection processing here can be claim processing.

[0071] Among them, determining the answer content of the question based on the character prediction results may include: determining a response strategy for answering the question based on the object type and / or question classification of the question; generating the answer to the question based on the response strategy and the character prediction results and / or extracting the answer details of the answer to the question from the character prediction results.

[0072] The object type here can be determined based on the object of inputting multiple page images and questions.

[0073] It should be noted that any one of steps S302 to S314 or any combination of multiple steps can be replaced by the corresponding technical means provided in the above steps S202 to S208 according to the needs of implementation and deployment, and will not be described one by one here.

[0074] An embodiment of an image processing device provided in this specification is as follows.

[0075] In the above embodiment, an image processing method is provided, and correspondingly, an image processing device is also provided, which will be described below with reference to the accompanying drawings.

[0076] 4 , which shows a schematic diagram of an image processing device embodiment provided by this embodiment.

[0077] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0078] This embodiment provides an image processing device, including: a character recognition module 402, configured to obtain multiple page images and questions of a document, and perform character recognition on each page image to obtain character information; a feature extraction module 404, configured to perform text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and perform visual feature extraction on each page image to obtain visual features of each page image; a feature fusion module 406, configured to perform feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fusion features; a character prediction module 408, configured to perform character prediction on the fusion feature based on the question, and determine the answer content of the question according to the character prediction result.

[0079] An embodiment of an image processing device provided in this specification is as follows.

[0080] Corresponding to the image processing method described above, based on the same technical concept, one or more embodiments of this specification also provide an image processing device, which is used to execute the image processing method provided above. Figure 5 is a structural schematic diagram of an image processing device provided by one or more embodiments of this specification.

[0081] This embodiment provides an image processing device, as shown in FIG5 . The image processing device may vary greatly due to different configurations or performances, and may include one or more processors 501 and a memory 502. The memory 502 may store one or more applications or data. The memory 502 may be a temporary storage or a persistent storage. The application stored in the memory 502 may include one or more modules (not shown), each of which may include a series of computer-executable instructions in the image processing device. Furthermore, the processor 501 may be configured to communicate with the memory 502 to execute the series of computer-executable instructions in the memory 502 on the image processing device. The image processing device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, and the like.

[0082] In a specific embodiment, the image processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the image processing device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: obtaining multiple page images and questions of a document, and performing character recognition on each page image to obtain character information; performing text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on the each page image to obtain visual features of the each page image; performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the each page image to obtain fused features; performing character prediction on the fused features based on the question, and determining the answer content of the question based on the character prediction results.

[0083] An embodiment of a computer-readable storage medium provided in this specification is as follows.

[0084] Corresponding to the image processing method described above, based on the same technical concept, one or more embodiments of this specification further provide a computer-readable storage medium.

[0085] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which implement the following steps when executed: obtaining multiple page images and questions of a document, and performing character recognition on each page image to obtain character information; performing text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each page image to obtain visual features of each page image; performing feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain fused features; performing character prediction on the fused features based on the question, and determining the answer content of the question based on the character prediction result.

[0086] It should be noted that the embodiment of a computer-readable storage medium in this specification and the embodiment of an image processing method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0087] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. For example, the device embodiment, equipment embodiment and computer-readable storage medium embodiment are similar to the method embodiment, so the description is relatively simple. For relevant content in the device embodiment, equipment embodiment and computer-readable storage medium embodiment, please refer to the partial description of the method embodiment.

[0088] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] In the 1930s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0090] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0091] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0092] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0093] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0095] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable image processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0096] These computer program instructions can also be loaded onto a computer or other programmable image processing device so that a series of operating steps are executed on the computer or other programmable device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0097] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0099] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0100] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0101] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0102] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0103] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. An image processing method, comprising: Acquire multiple page images and questions of a document, and perform character recognition on each page image to obtain character information; Performing text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each of the page images to obtain visual features of each of the page images; Performing feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain a fusion feature; Character prediction is performed on the fused features based on the question, and the answer content of the question is determined according to the character prediction result.

2. The image processing method according to claim 1, wherein the step of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain fused features comprises: Based on each page layout feature in the visual feature sequence, character positioning is performed on the text feature to obtain character features in the page unit corresponding to each page layout feature in the text feature; The visual features in the visual feature sequence are fused with the character features in the page unit to obtain the fused features.

3. The image processing method according to claim 1, wherein extracting visual features from each page image to obtain the visual features of each page image comprises: Extracting layout features of each page image to obtain page layout features of each page image; Text features are extracted from the page images to obtain text visual features of the page images, and the page layout features and the text visual features are used as the visual features.

4. The image processing method according to claim 3, wherein the step of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain fused features comprises: Detecting invalid character features in the text features based on each page layout feature in the visual feature sequence, and deleting the invalid character features from the text features to obtain target text features; Filling the visual features of each page image in the visual feature sequence according to the configuration information to obtain a target visual feature sequence; The target text feature and the target visual feature sequence are subjected to feature fusion to obtain the fused feature.

5. The image processing method according to claim 4, wherein the configuration information is obtained by: determining a target visual feature among the visual features of each page image based on the invalid character feature; using the adjacent background visual features of the target visual features as the configuration information; or, The preset visual features are used as the configuration information.

6. The image processing method according to claim 1, wherein the problem is obtained by: Acquire the question input for the plurality of page images; or determine the object type to which the objects input for the plurality of page images belong, and determine the question according to the object type.

7. The image processing method according to claim 6, wherein determining the answer to the question based on the character prediction result comprises: determining a response strategy for responding to the question based on the object type and / or the question classification of the question; An answer to the question is generated according to the answer strategy and the character prediction result and / or answer details of the answer to the question are extracted from the character prediction result.

8. The image processing method according to claim 7, wherein determining a response strategy for the question based on the object type and / or the question classification comprises: If the object type is the first object type and the question classification is the first question classification, determining that the response strategy is an answer response strategy and a detail response strategy; If the object type is the second object type or the question classification is the second question classification, the reply strategy is determined to be an answer reply strategy.

9. The image processing method according to claim 1, wherein the step of performing character prediction on the fused features based on the question and determining the answer to the question according to the character prediction result comprises: The fusion features and the questions of the guarantee project document are input into a decoding network for decoding processing to obtain answers to the questions and key guarantee content, so as to process the guarantee application based on the answers to the questions and the key guarantee content.

10. The image processing method according to claim 1, wherein the method is performed based on an image analysis model; The input of the image parsing model includes the multiple page images and the question; the output of the image parsing model includes the answer content.

11. The image processing method according to claim 1, wherein the step of performing feature fusion processing on the visual feature sequence composed of the text features and the visual features of the page images to obtain fused features comprises: Inputting the text feature and the visual feature sequence into a feature fusion network for feature fusion to obtain the fused feature; The feature fusion network includes a fully connected network. 12 . The image processing method according to claim 1 , wherein the plurality of page images of the document comprise a plurality of page images corresponding to a plurality of document page numbers in the same document or a plurality of screenshot images of a web page of the same web page.

13. An image processing device, comprising: A character recognition module is configured to obtain multiple page images and questions of a document and perform character recognition on each page image to obtain character information; a feature extraction module configured to perform text feature extraction on the spliced ​​text obtained by splicing the character information to obtain text features, and to perform visual feature extraction on the respective page images to obtain visual features of the respective page images; a feature fusion module configured to perform feature fusion processing on a visual feature sequence composed of the text features and the visual features of the page images to obtain a fused feature; The character prediction module is configured to perform character prediction on the fused features based on the question, and determine the answer content of the question according to the character prediction result.

14. An image processing device comprising: processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to: Acquire multiple page images and questions of a document, and perform character recognition on each page image to obtain character information; Performing text feature extraction on a spliced ​​text obtained by splicing the character information to obtain text features, and performing visual feature extraction on each of the page images to obtain visual features of each of the page images; Performing feature fusion processing on a visual feature sequence composed of the text features and the visual features of each page image to obtain a fusion feature; Character prediction is performed on the fused features based on the question, and the answer content of the question is determined according to the character prediction result.

15. A computer-readable storage medium for storing computer-executable instructions, wherein the computer-executable instructions implement the steps of the method of claim 1 when executed.

Images