Method and system for determining custom information in a document image

Abstract

The application discloses a method and system for determining self-defined information in a document image, and relates to the field of computer vision. The method comprises the following steps: using a machine reading comprehension (MRC) dataset, combining the text features, image features, absolute position features and relative position features of each character, performing first question and answer task training on a pre-training model to obtain a re-training model; using a specified sample dataset to perform second question and answer task training on the re-training model to obtain a final model; inputting a document image to be determined for self-defined information, and outputting the text content of the self-defined information. The technical scheme of the application uses a pre-training model and a question and answer task to achieve zero sample learning, combines a large corpus pre-training model and a question and answer system, and further adds image and relative position features to achieve end-to-end training and prediction, and finally outputs the self-defined information in the document.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a method and system for determining self-defined information in a document image. BACKGROUND

[0002] In reality, there are many cases of extracting key information from documents. With the continuous development and popularization of deep learning, more and more people abandon the traditional low-efficiency method of one template for one type of document, and instead use deep learning methods to train various neural networks to extract information. The first thing needed to train a neural network is data, and the second is to determine the entity type to be extracted, i.e., predefined entities, so as to label the data accordingly. Through a large amount of training of the neural network on these labeled data, the network can learn useful information to extract predefined entities.

[0003] In reality, there are many cases of extracting key information from documents. With the continuous development and popularization of deep learning, more and more people abandon the traditional low-efficiency method of one template for one type of document, and instead use deep learning methods to train various neural networks to extract information. The first thing needed to train a neural network is data, and the second is to determine the entity type to be extracted, i.e., predefined entities, so as to label the data accordingly. Through a large amount of training of the neural network on these labeled data, the network can learn useful information to extract predefined entities.

[0004] In the era of big data, the amount of data available grows exponentially, but most of these data are unlabeled and may not be related to the specific task to be solved. How to learn useful knowledge from these massive data and apply it to specific tasks? This requires the use of pre-trained models. The training of pre-trained models usually designs some unsupervised training tasks to learn general information in the data, such as image classification, grammar and syntax in language, etc. Pre-trained models have made breakthrough progress in ImageNet in the field of computer vision. With the emergence of BERT and its excellent performance, pre-trained models have rapidly developed in the field of NLP and achieved good results. After obtaining the pre-trained model, it can be applied to different downstream tasks such as question and answer system, text classification, object detection, named entity recognition, etc. by changing its output layer. Compared with the model trained from zero, the pre-trained model can provide good preliminary knowledge, and these knowledge can greatly help the downstream task, allowing the model to converge faster and have higher accuracy.

[0005] As a classic task in natural language processing, the question and answer system is a more advanced form of information retrieval system, aiming to answer the user's question in natural language with accurate and concise natural language. The research on question and answer system can be traced back to the 1960s. At that time, the method was based on templates and rules, and the robustness and accuracy of the model were poor. There are many methods and technologies for the current question and answer system. Here, according to the different processing methods, it is divided into two kinds: knowledge graph-based question and answer system and reading comprehension-based question and answer system. The first kind is to build a fact-based question and answer system based on a knowledge graph, and find answers from the knowledge graph. The accuracy is relatively high. The disadvantage is that it is too dependent on the knowledge graph and cannot give answers outside the knowledge graph. This requires sufficient resources to establish a large-scale knowledge graph. The second kind is to perform reading comprehension on unstructured articles to obtain answers. The data form is to give an article and ask some questions around the article. The task is to directly extract the answer from the article. Common models include FastQAExt, BERT, RoBERTa, etc. SUMMARY

[0006] The present application relates to a method and system for determining self-defined information in a document image. The technical solution of the present application uses pre-trained models and question and answer tasks for zero-shot learning, and combines large corpus pre-trained models and question and answer systems. In addition, image and relative position features are added to achieve end-to-end training and prediction, and finally output the self-defined information in the document.

[0007] According to the first aspect of the technical solution of the present application, a method for determining self-defined information in a document image is provided, which comprises the following steps:

[0008] Step 1: using a machine reading comprehension (MRC) dataset, combining text features, image features, absolute position features and relative position features of each character, performing first question and answer task training on a pre-trained model to obtain a retraining model;

[0009] Step 2: using a specified sample dataset to perform second question and answer task training on the retraining model to obtain a final model;

[0010] Step 3: inputting a document image to be determined for custom information, and outputting text content of the custom information.

[0011] Here, "custom information" refers to information that is expected to be determined and is not in a predefined entity category previously trained by the model.

[0012] Further, in step 1, the MRC dataset includes medical, education, entertainment, encyclopedia, legal and other fields.

[0013] Further, step 1 specifically includes:

[0014] Step 11: selecting an initial sample from the machine reading comprehension dataset, inputting an initial sample image, text within a text box identified from the initial sample image, text box position coordinates, and entity information to form a question;

[0015] Step 12: based on a pre-trained model, extracting features of text information and image information, and dividing position information into absolute position and relative position and extracting features respectively, to obtain text features, image features, absolute position features and relative position features;

[0016] Step 13: based on a multi-head self-attention mechanism, encoding features of the text features, image features, absolute position features and relative position features through a Transformer to obtain an encoding feature vector of each character;

[0017] Step 14: performing two-stage task reasoning on the encoding feature vector of each character to obtain a retraining model.

[0018] Further, in step 12, the pre-trained model includes a pre-trained Chinese BERT model and a pre-trained ResNet-50 network.

[0019] Further, step 12 specifically includes:

[0020] Character encoding layer: input text information including questions and articles, and perform text feature extraction through a pre-trained Chinese BERT model to obtain text features;

[0021] image encoding layer: input image information, image feature extraction is performed through a pre-trained ResNet-50 network combined with ROIAlign;

[0022] absolute position encoding layer: each character in the input string is sorted from 0 to obtain an absolute position feature;

[0023] relative position encoding layer: the coordinates of the top-left vertex and the bottom-right vertex of the character box are converted into a feature vector to obtain a relative position feature.

[0024] Further, in the character encoding layer, the question is composed of the entity information: q1…qi, and the article is the text in the text box recognized from the initial sample image: t1…tj, i and j are positive integers.

[0025] Further, in the character encoding layer, the text feature is a text feature matrix of M*D, where M is the number of input characters and M=i+j+3; D is the input dimension size of the pre-trained Chinese BERT model.

[0026] Further, the encoding layer sequence in the character encoding layer is:

[0027] length 1 i 1 j 1 order content [CLS] q1 …… qi [SEP] t1 …… tj [SEP]

[0028] where [CLS] is a marker, and its word embedding is used for classification tasks; [SEP] is a separator used to separate the question and the article.

[0029] Further, the dimension of the character encoding layer weight is N*D, where D is the input dimension of the pre-trained Chinese BERT model, and N is the number of characters in the dictionary of the pre-trained Chinese BERT model, N and D are positive integers.

[0030] Further, in the image encoding layer, all characters in the same text box share the image feature of the text box.

[0031] Further, the encoding layer sequence in the image encoding layer is:

[0032] length 1 i+1 j 1 order content vf 0……0 v0 …… vp 0

[0033] where v0…vp is the image feature vector of each character, and p≥0.

[0034] Further, the encoding layer sequence in the absolute position encoding layer is:

[0035] length 1 + i + 1 + j + 1 order content 0……h

[0036] where h=1+i+1+j+1.

[0037] Further, the relative position encoding layer comprises:

[0038] x1 encoding layer, x1 being a coordinate value on the x-axis of the upper left point of the character frame of a certain character;

[0039] x2 encoding layer, x2 being a coordinate value on the x-axis of the lower right point of the character frame of a certain character;

[0040] y1 encoding layer, y1 being a coordinate value on the y-axis of the upper left point of the character frame of a certain character;

[0041] y2 encoding layer, y2 being a coordinate value on the y-axis of the lower right point of the character frame of a certain character.

[0042] Further, the sequence of encoding layers in the relative position encoding layer is:

[0043]

[0044]

[0045] wherein the x1 encoding layer and the x2 encoding layer have the same parameters, and the y1 encoding layer and the y2 encoding layer have the same parameters.

[0046] Further, in step 13, a self-attention mechanism with 12 heads is used to encode the features.

[0047] Further, in step 13, the calculation formula is as follows:

[0048]

[0049] wherein the matrices Q, K and V represent Query, Key and Value, respectively, representing the mapping of the input encoder's text feature vector / image feature vector / absolute position feature vector / relative position feature vector in three different low-dimensional spaces, d k representing the dimension size of the corresponding feature vector.

[0050] Further, the step 14 specifically comprises:

[0051] First stage: constructing a first binary classifier, according to the encoding feature vector of each character, dividing all characters into meaningful characters involving start / terminal positions and meaningless characters involving other information;

[0052] Second stage: constructing a second binary classifier, according to the encoding feature vector of each meaningful character, dividing the meaningful characters involving start / terminal positions into start position characters and terminal position characters,

[0053] Thus, the retraining model is obtained.

[0054] Further, in step 2, the specified sample data set includes an article composed of a custom question and specified data.

[0055] Further, step 3 specifically includes: inputting a document image to be determined into a custom information into a final model, outputting a start position character and an end position character of the custom information, and further determining the custom information.

[0056] According to a second aspect of the present application, a system for determining custom information in a document image is provided, the system comprising: a processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the method for determining custom information in a document image as described in any of the above aspects.

[0057] According to a third aspect of the present application, a computer readable storage medium is provided, characterized in that a computer program is stored thereon, the computer program being executed by a processor to implement the method for determining custom information in a document image as described in any of the above aspects.

[0058] Advantages of the present application:

[0059] 1. Through the combination of the pre-trained model and the question and answer task, not only the precision of the trained entity extraction is greatly improved, but also the custom entity can be better extracted, and the generalization ability of the model is strong;

[0060] 2. Before performing the specified sample training, the pre-trained model is retrained using a large number of question and answer task training sets composed of general texts, which can greatly improve the understanding and solving ability of the model for the question and answer task;

[0061] 3. The model fully and efficiently utilizes document features, including syntax and semantics within the text, relationships between texts within the sentence, image features of the document, and position information of the text on the image, etc. The multi-modal feature allows the model to better understand the content and layout information of the document;

[0062] 4. The popular and effective pre-trained models such as Resnet-50 and BERT are applied, which makes the information learned by the model more rich, the learning speed faster, and the precision higher. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0064] Figure 1 An exemplary shopping receipt is shown.

[0065] Figure 2 An algorithm structure diagram of a pre-training model according to an embodiment of the present application is shown.

[0066] Figure 3 An algorithm flowchart according to an embodiment of the present application is shown.

[0067] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0068] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The following description is presented with reference to the drawings, wherein the same reference numerals are used to refer to like or similar elements throughout the several exemplary embodiments and / or aspects of the present disclosure. The following detailed description is not intended to restrict the present disclosure to one or more particular embodiments. Rather, the following detailed description is intended to describe illustrative embodiments and / or aspects of the present disclosure in sufficient details to enable one of ordinary skill in the art to practice the present disclosure.

[0069] The terms "first", "second", and the like, in the description of the present disclosure and the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can interchange depending on the context in which they are used. Embodiments of the present disclosure described herein are not limited to the specific embodiments described herein, but include parts of the specific arrangements described.

[0070] In addition, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a list of steps or units not necessarily limited to those listed, but can include other steps or units not expressly listed or inherent to such process, method, product, or apparatus.

[0071] Plural, including two or more.

[0072] And / or, it should be understood that for the term "and / or" used in the present disclosure, it is only a description of the association relationship of the associated objects, which means that there can be three relationships. For example, A and / or B can represent three cases of A existing alone, A and B existing together, and B existing alone.

[0073] The present application relates to a precise document image self-defined information extraction method. The so-called self-defined information is the information that is not in the pre-defined entity category trained before the model. Most neural network models do not do well for this self-defined information extraction. There are several reasons:

[0074] 1. The model is not good at learning images and texts in the document. In order to extract the pre-defined entities in the training set well, the model will pay more attention to the features around these entities in the learning process, and ignore the features of other texts and global features in the document. This will lead to the fact that if you want to extract self-defined entities, the model will not be able to accurately extract them because of the lack of learned features;

[0075] 2. The design of the output layer of the model. Most entity extraction models will classify the output according to the pre-defined categories, and the information not in the pre-defined categories will be classified as 'other'. This leads to the fact that when adding self-defined categories, the model will classify all self-defined categories into 'other' without modifying the output layer and retraining, and cannot accurately extract them.

[0076] The purpose of the present application is to accurately extract self-defined entity information without retraining the model or modifying the output layer. The two problems mentioned above are well solved by pre-training the model and the question and answer task.

[0077] 1. The language model trained through unsupervised task on large-scale corpus not only has a good understanding of grammar, but also has a deep understanding of the semantics inside and between sentences, so that the model will not be limited to a certain type of text or information, but will focus on overall understanding, which lays a solid foundation for the optimization of downstream tasks;

[0078] 2. Regarding the design of the output layer, the present application does not use classification tasks, but question and answer tasks. Each entity to be extracted is designed as one or more questions. For example Figure 1As shown, when the information of the "total amount" on the shopping receipt is to be extracted, the question can be designed as "How much money did you spend on this shopping?", and the article is all the text on the shopping receipt. The input of the model is the question and the article, and the output is the start and end positions of the answer interval of the question in the article. As long as the entity information appears in the article, there will be the start and end positions of the corresponding interval, which avoids the case that there is no category corresponding to the self-defined entity in the classification task output.

[0079] The pre-training model is obtained through a full-word masking training task on a large-scale corpus (Baidu Encyclopedia, news data, and the like, with a total word number of 5.4B). After the pre-training model is obtained, the following work is divided into two stages: 1. re-training the pre-training model on the MRC (machine reading comprehension) data collected by oneself (including medical, education, entertainment, encyclopedia, law, and the like) for the question and answer task, and the main purpose is to strengthen the understanding and solving ability of the model for general text question and answer tasks; 2. using the target sample to train the model after re-training in 1 for the question and answer task to obtain the final model. It should be noted that the image encoding layer and the relative position encoding layer are added in the encoding layer when the pre-training model is re-trained in 1, so as to fuse the multiple features of the image, the text and the position with each other, so that the model has a more comprehensive understanding of the document.

[0080] Embodiments

[0081] Pre-training model

[0082] 1) Feature extraction module

[0083] The input of the model is the image of the whole sample, the text in the text box recognized by OCR, the position coordinates of the text box, and the question composed of entity information. The main task of this module is to encode these inputs to generate feature vectors that can be input to the subsequent module.

[0084] This embodiment mainly extracts three aspects of features: text, image and position.

[0085] The text feature is extracted through the character encoding layer in this module. The text input is divided into two parts: the question and the article. The question is composed of the entity information to be extracted, and the form of the question here is not unique; the article is all the text content in the sample document. Each small square in the algorithm structure diagram represents a character, such as Figure 2As shown, the question consists of three characters q1, q2 and q3, and the article consists of 4 characters t1, t2, t3 and t4. The question and the article are separated by a special character [SEP], and a special character [CLS] is added at the beginning. The weights of the character encoding layer are first initialized using a Chinese BERT model pre-trained on a large amount of text such as Chinese Wikipedia, which has the ability to parse syntax and semantics after pre-training. The initialized encoding layer can make the training converge faster and more accurate. Assuming that the dictionary of the model has N characters, the dimension of the character encoding layer weight is N*D, where D is the input dimension size of the model. Assuming that the number of input characters is M, each input character finds the corresponding vector in the character encoding layer according to its position in the dictionary, so that a text feature matrix of M*D is obtained after passing through the character encoding layer.

[0086] For the input image, the most important thing is to do size normalization with constant aspect ratio and boundary 0, so that the size of the image can support the convolution and downsampling operations required by the neural network in the encoding module, and maximize the preservation of global and local feature information. Image feature encoding mainly uses a deep convolutional neural network to encode the image features of the text block and its surrounding area. This step uses ResNet-50 pre-trained on ImageNet to serve as the feature encoding network, which has powerful representation ability for images and can extract and represent the key features of images well. The goal of this step is to output the image feature encoding corresponding to each text box, so it is necessary to apply ROIAlign to the corresponding position of the network output feature map combined with the position of the text box to obtain the corresponding image feature encoding. The characters in the same text box share the image features of the text box, such as Figure 2 As shown in FIG. 3B, assuming that t1 and t2 are in different text boxes, so their corresponding image encoding layer features are different, v0 and v1 respectively; t3 and t4 are in the same text box, so they correspond to the same image feature v2. Here, the image features of the entire sample document are placed in the position of the first character [CLS]. Since the question and [SEP] do not appear in the document, the corresponding image features are set to zero matrices.

[0087] Position features are divided into absolute position and relative position. The absolute position is the position of a character in the input string, such as Figure 2 As shown in FIG. 3C, the absolute position encoding layer starts from 0 and is arranged in order down to 512 characters at most. The relative position here uses the coordinates of the top-left and bottom-right points of the character box recognized by OCR to convert them into vectors through the encoding layer, and then learns the relative position relationship between characters through the subsequent feature encoding module. As shown in FIG. 3D, the relative position encoding layer is a 2*2 matrix, and the input is the coordinates of the top-left and bottom-right points of the character box, which are converted into vectors through the encoding layer. Figure 2For example, t1.x1 represents the coordinate value of the upper left point of the character frame on the x-axis, t1.x2 represents the coordinate value of the lower right point of the character frame on the x-axis, t1.y1 represents the coordinate value of the upper left point of the character frame on the y-axis, and t1.y2 represents the coordinate value of the lower right point of the character frame on the y-axis. The x1 encoding layer and the x2 encoding layer are the same, and the y1 encoding layer and the y2 encoding layer are the same. Since the question and special characters do not appear in the document, the corresponding relative position coordinates are uniformly set to 0.

[0088] 2) Feature encoding module

[0089] This module is mainly composed of a Transformer encoder. Transformer is a seq2seq model proposed by Google Brain in the paper "Attention is all you need" in 2017. Its essence is composed of an encoder and a decoder. Here, the encoder is mainly used. The most core part of the encoder is the self-attention mechanism, which can be understood as calculating the correlation. It is a natural idea to pay more attention to things with high correlation. Here, the concepts of Query, Key and Value are introduced. Query means query, Key is used to compare with the Query to get a score (correlation or similarity) and multiply it by the Value to get the final result. Multiple sets of Query, Key and Value can be designed to extract different features, which is the multi-head self-attention mechanism. This module uses a 12-head self-attention mechanism to encode features. Of course, the more heads are not necessarily better. These parameters are selected through experiments. The feature vectors obtained by the feature encoding module not only contain image features, grammar and semantic information of the text itself, and sentence features specific to this type of sample, but also contain the mutual position relationship between text boxes, which enables the model to learn multiple types of information and better complete the subsequent tasks.

[0090] For example, in the multi-head attention mechanism, an input vector x passes through h sets of weight matrices to obtain h sets of query, key and value vectors. Each word has h sets of query, key and value vectors. The query vector of a word is multiplied by the key vector of each word to calculate the attention score (attention score) of the current word with all words. Divide the attention score by the square root of the first dimension dk of the weight matrix, and then perform a softmax operation to get a weight value for each word. Next, multiply the weight value obtained for each word with its own value vector and add them together, and finally calculate h output matrices Z1, Z2,... Zh. The h output matrices are spliced together to obtain the final output matrix Z.

[0091] In the algorithm, Query, Key and Value are represented by matrices Q, K and V, which respectively represent the mapping of the feature vectors of the input encoder in three different low-dimensional spaces. Q and K do the inner product of vectors, and then do the weighted sum with V to get the feature vector of self-attention, which is calculated as follows:

[0092]

[0093] where d k represents the dimension size of the feature vector.

[0094] 3) Task reasoning module

[0095] After the feature encoding module, each character corresponds to a vector containing rich information. This module is to find the start position (S) and end position (E) of the answer using these vectors. This is divided into two stages: the first stage is to construct a binary classifier to determine whether a character belongs to the start or end position or belongs to other (O). Each character vector is binary classified into meaningful characters (start or end) and meaningless characters. The second stage also constructs a binary classifier to distinguish between start and end positions for meaningful characters. Dividing into two stages can make the task of the classifier more accurate, and dividing a complex task into several simple tasks can improve reasoning speed and accuracy. With the start and end positions of the answer, the entity corresponding to the question is the text content from the start position to the end position.

[0096] As Figure 3 shown, the technical scheme of the present application has the following process:

[0097] In the first stage, the pre-trained model is retrained for the question and answer task using the MRC data collected by the applicant. The data set of this stage includes medical, education, entertainment, encyclopedia, law, etc. Various fields are used to enable the model to learn the ability required to solve the question and answer task from various types of text, such as the relevance between the question and the article, how to locate the answer in the article through the question, and how to quickly associate the relevant part in the article through the keyword in the question and the self-attention mechanism, etc.

[0098] The second stage can use the specified sample set to train the model to extract the desired entity type. Through experimental comparison, it is found that the retraining of the question and answer task of the general text in the first stage can effectively improve the extraction accuracy of the task in the second stage.

[0099] In the third stage, after careful analysis of the samples, some custom entities are proposed to test the generalization ability of the model. Through experiments, it is found that the model can also be well extracted for custom entities that have not been trained. The pre-training model and the retraining in the first stage cannot be ignored.

[0100] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0101] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0102] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned implementation methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a number of instructions to make a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0103] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims. These are all within the protection of the present application.

Claims

1. A method of determining self-defined information in a document image, characterized by, The method comprises the following steps: Step 1: using a machine reading comprehension dataset, combining text features, image features, absolute position features and relative position features of each character, performing first question and answer task training on a pre-trained model to obtain a retrained model; Step 2: using a specified sample dataset to perform second question and answer task training on the retrained model to obtain a final model; Step 3: inputting a document image to be determined for custom information, and outputting text content of the custom information; Wherein, the step 1 specifically comprises: Step 11: selecting an initial sample from the machine reading comprehension dataset, inputting an initial sample image, text in a text box identified from the initial sample image, text box position coordinates, and entity information to form a question; Step 12: based on a pre-trained model, performing feature extraction on text information and image information, and simultaneously dividing position information into absolute position and relative position and performing feature extraction respectively to obtain text features, image features, absolute position features and relative position features; Step 13: based on a multi-head self-attention mechanism, performing feature encoding on the text features, image features, absolute position features and relative position features to obtain an encoding feature vector of each character; Step 14: performing two-stage task reasoning on the encoding feature vector of each character to obtain a retrained model; Wherein, in the step 12, the pre-trained model comprises a pre-trained Chinese BERT model and a pre-trained ResNet-50 network; Wherein, the step 12 specifically comprises: A character encoding layer: inputting text information including a question and an article, performing text feature extraction through a pre-trained Chinese BERT model to obtain text features; An image encoding layer: inputting image information, performing image feature extraction through a pre-trained ResNet-50 network combined with ROIAlign to obtain image features; An absolute position encoding layer: for each character in the input string, the absolute position features are obtained by sorting from 0; A relative position encoding layer: the coordinates of the top-left vertex and the bottom-right vertex of the character box are converted into a feature vector to obtain the relative position features.

2. The determination method according to claim 1, characterized in that, In the character encoding layer, the question is the question composed of the entity information: q1…qi, and the article is the text in the text box identified from the initial sample image: t1…tj, i and j are positive integers.

3. The determination method according to claim 2, characterized in that, In the character encoding layer, the text features are a text feature matrix of M*D, where M is the number of input characters and M=i+j+3; D is the input dimension size of the pre-trained Chinese BERT model.

4. The determination method according to claim 1, characterized in that, The dimension of the character encoding layer weight is N*D, where D is the input dimension of the pre-trained Chinese BERT model, and N is the number of characters in the dictionary of the pre-trained Chinese BERT model, N and D are positive integers.

5. The determination method of claim 1, wherein, In the image encoding layer, all characters in the same text box share the image features of the text box.

6. The determination method of claim 1, wherein, The relative position encoding layer comprises: An x1 encoding layer, x1 being the coordinate value of the left top point of the character box of a certain character on the x-axis; An x2 encoding layer, x2 being the coordinate value of the right bottom point of the character box of a certain character on the x-axis; y1 is a coordinate value on the y-axis of the upper left point of a character frame of a certain character; y2 is a coordinate value on the y-axis of the lower right point of a character frame of a certain character.

7. The determination method of claim 1, wherein, In the step 13, a self-attention mechanism of 12 heads is used to encode the features.

8. The determination method according to claim 7, characterized in that, In the step 13, the calculation formula is as follows: wherein matrices Q, K and V represent Query, Key and Value respectively, representing the mapping of the text feature vector / image feature vector / absolute position feature vector / relative position feature vector of the input encoder in three different low-dimensional spaces, d k denotes the dimension size of the corresponding feature vector.

9. The determination method of claim 1, wherein, The step 14 specifically includes: The first stage: constructing a first binary classifier, and dividing all characters into meaningful characters related to start / terminal positions and meaningless characters related to other information according to the encoding feature vectors of each character; The second stage: constructing a second binary classifier, and dividing the meaningful characters related to the start / terminal positions into start position characters and terminal position characters according to the encoding feature vectors of each meaningful character, Thus, a retraining model is obtained.

10. The determination method of claim 1, wherein, In the step 2, the specified sample data set includes articles composed of self-defined questions and specified data.

11. The determination method of claim 1, wherein, The step 3 specifically includes: inputting a document image to be determined into the final model, outputting start position characters and terminal position characters of the self-defined information, and then determining the self-defined information.

12. A system for determining custom information in a document image, the system comprising: A processor and a memory for storing executable instructions; wherein the processor is configured to execute the executable instructions to perform the method for determining self-defined information in a document image according to any one of aspects 1 to 11.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the method for determining self-defined information in a document image according to any one of aspects 1 to 11.

Images