A fine-grained document image understanding method and system

Abstract

The application relates to a fine-grained document image understanding method and system, wherein the method performs text content recognition on an input document image, obtains the pixel coordinates of each character in the obtained text sequence, constructs a text coordinate mapping table, performs Token-level segmentation processing on the text sequence through a BER segmenter, performs visual coding and language semantic feature extraction on all Token-level texts obtained according to the text coordinate mapping table, obtains all Token-level visual coding and all Token-level language semantic features, performs patch segmentation on the document image, performs visual feature extraction on each obtained image patch, calculates the intersection of each obtained patch visual feature and all Token-level visual coding, maps the patch visual feature in the intersection to all Token-level language semantic features, and inputs the comparison model for alignment, so as to realize understanding of the document image according to the obtained alignment result. Therefore, the application realizes fine-grained document image understanding.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a fine-grained document image understanding method and system. BACKGROUND

[0002] The document image understanding technology refers to converting the image data of paper or electronic documents into structured text semantic information, and the core goal is to establish the association between "image visual information" and "text semantic information" to support information extraction, table parsing, multilingual translation and other downstream tasks, and is widely used in financial report processing, government form verification, medical prescription analysis and other fields.

[0003] The current mainstream document image understanding technology is to extract visual features of the document image through a convolutional neural network, extract language semantic features of the document image through a pre-trained language model, and align the extracted visual features and language semantic features to realize the understanding of the document image. However, due to the limited field of view of the convolutional neural network, it can only capture spatial correlations within a limited range and cannot capture fine-grained spatial distribution features, resulting in insufficient precision of the obtained visual features, affecting the accuracy of subsequent alignment, and affecting the accuracy of document image understanding. In addition, the pre-trained language model contains a large number of parameters, and the computational complexity is high when extracting language semantic features, which is difficult to meet the needs of real-time processing scenarios. SUMMARY

[0004] The technical problem to be solved by the present application is that the present application provides a fine-grained document image understanding method and system, which realizes fine-grained visual feature extraction while improving the efficiency of language semantic feature extraction, meets the needs of real-time processing scenarios, and improves the accuracy of alignment and the accuracy of document image understanding.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is:

[0006] In a first aspect, the present application provides a fine-grained document image understanding method, comprising:

[0007] obtaining an input document image, performing text content recognition on the document image to obtain a text sequence, obtaining the pixel coordinates of each character in the text sequence, mapping the pixel coordinates and the text sequence to obtain a text coordinate mapping table;

[0008] Token level text is obtained by performing Token level segmentation processing on the text sequence by the BPE tokenizer, visual encoding is performed on all Token level texts according to the text coordinate mapping table, and Token level visual encoding is obtained, and language semantic feature extraction is performed on all Token level texts, and Token level language semantic features are obtained.

[0009] The document image is segmented into K image patches, visual feature extraction is performed on each image patch, and patch level visual features are obtained, the intersection of each patch level visual feature and all Token level visual encodings is calculated, the patch visual features in the intersection are mapped with all Token level language semantic features, and the mapped patch level visual features and the mapped Token level language semantic features are obtained, the mapped patch level visual features and the mapped Token level language semantic features are input into the comparison model for alignment, and the alignment result is obtained, and the understanding of the document image is realized according to the alignment result.

[0010] The beneficial effects of the present application are that: the text sequence is processed by the BPE tokenizer Token level segmentation, that is, the text sequence is granulated into Token level segmentation, and each Token level segmentation only carries a single semantic unit in the form of the smallest effective unit, without redundancy, which not only improves the accuracy of language semantic feature extraction, but also reduces the computational complexity and improves the extraction efficiency, and can meet the needs of real-time processing scenarios, and the document image is segmented into patches, and visual feature extraction is performed on each image patch, the visual space is granulated, the accuracy of visual feature extraction is improved, the accuracy of subsequent alignment of patch level visual features and Token level language semantic features is improved, and the accuracy of document image understanding is improved. The visual encoding of Token level text is realized by constructing the text coordinate mapping table in advance, the visual anchoring of Token level text is realized, the misalignment of visual regions and text regions is avoided, the intersection of each patch level visual feature and all Token level visual encodings is calculated, that is, the patch level visual features related to Token level visual encoding are screened out, and the patch level visual features in the intersection are mapped with all Token level language semantic features, avoiding the participation of full patch level visual features in calculation, saving computing resources and improving mapping efficiency.

[0011] Optionally, the Token level segmentation processing on the text sequence by the BPE tokenizer to obtain all Token level texts comprises:

[0012] obtaining a basic vocabulary of a BPE tokenizer, performing an adding operation including mathematical symbols, multilingual characters and document structure markers on the basic vocabulary to obtain an added basic vocabulary, performing Token-level segmentation processing on the text sequence according to the added basic vocabulary to obtain all Token-level texts, the all Token-level texts including Token-level texts matched with the added basic vocabulary and Token-level texts not matched with the added basic vocabulary;

[0013] taking the Token-level texts not matched with the added basic vocabulary as unknown words, calculating the character length of each unknown word, performing optimization processing on the unknown words according to the character length, judging whether the character length is greater than a length threshold, if not, performing optimization processing on the unknown words with an unknown word label, obtaining the optimized unknown words, if yes, performing optimization processing on the unknown words with character-level coding and character decomposition, obtaining the optimized unknown words;

[0014] updating all Token-level texts according to the optimized unknown words to obtain updated all Token-level texts.

[0015] According to the above description, the adding operation is performed in the basic vocabulary of the BPE tokenizer, which overcomes the defects of being unable to process mathematical symbols and multilingual symbols while maintaining the basic segmentation ability, avoids the loss or error segmentation of important information, and differentiates the unknown words according to their character length. For unknown words greater than the length threshold, character-level coding is used for character decomposition, which can better preserve morphological information and semantic information, reduce semantic loss, and thus improve the accuracy and integrity of the obtained Token-level texts.

[0016] Optionally, the language semantic feature extraction is performed on all Token-level texts to obtain all Token-level language semantic features, including:

[0017] generating an embedded character level corresponding to each Token-level text through a character-level embedding formula;

[0018] generating an embedded subword level corresponding to each Token-level text through a subword-level embedding formula;

[0019] generating an embedded semantic level corresponding to each Token-level text through a semantic-level embedding formula;

[0020] The character-level embedding, the subword-level embedding and the semantic-level embedding of each Token-level text are dynamically fused by a gating fusion mechanism to generate Token-level language semantic features of each Token-level text, and all Token-level language semantic features are obtained.

[0021] The character-level embedding formula is:

[0022] ;

[0023] wherein, represents an embedded character level corresponding to the i-th Token-level text, represents a character set of the i-th Token-level text, represents a character embedding matrix, and c represents a single character;

[0024] The subword-level embedding formula is:

[0025] ;

[0026] wherein, represents an embedded subword level corresponding to the i-th Token-level text, represents a subword embedding matrix, represents a row index of the i-th Token-level text;

[0027] The semantic-level embedding formula is:

[0028] ;

[0029] wherein, represents an embedded semantic level corresponding to the i-th Token-level text, represents a multi-layer perception, represents a part-of-speech tag of the i-th Token-level text, represents an entity tag of the i-th Token-level text, represents a one-hot encoding function, represents a feature splicing symbol.

[0030] According to the above description, the character-level embedding, the subword-level embedding and the semantic-level embedding of each Token-level text are generated, the character-level embedding can capture the semantic information and character pattern inside the Token-level text, the subword-level embedding can capture the semantic similarity and grammatical relationship between different subwords, and the semantic-level embedding can understand the semantic information of the Token-level text at the syntactic and semantic levels. The character-level embedding, the subword-level embedding and the semantic-level embedding obtained are dynamically fused by a gating fusion mechanism, and the accuracy and integrity of the Token-level language semantic features obtained are improved.

[0031] Optionally, the patch segmentation of the document image to obtain K image patches comprises:

[0032] Obtaining the image width, image height and document type of the document image;

[0033] Inputting the image width and image height into a text density formula to calculate the text density of the document image, the text density formula being:

[0034]

[0035] wherein, denotes the text density of the document image I, H denotes the image height, W denotes the image width, denotes an indication function of whether the pixel (x, y) is a text pixel;

[0036] Calculating the optimal scaling ratio of the document image according to the text density, and scaling the document image according to the optimal scaling ratio to obtain a scaled document image;

[0037] According to the document type, dynamically adjusting the patch size to perform adaptive size patch segmentation on the scaled document image to obtain K image patches.

[0038] According to the above description, when performing patch segmentation on the document image, the optimal scaling ratio of the document image is first calculated according to the text density of the document image, and then the patch size is adaptively adjusted according to the document type to realize dynamic patch segmentation, which breaks the limitation of traditional fixed size scaling and fixed size segmentation. Driven by the document density and the document type, the obtained image patch retains fine-grained detail information.

[0039] Optionally, the visual feature extraction of each image patch to obtain all patch-level visual features comprises:

[0040] Using a convolutional neural network to extract internal local features of each image patch to obtain all patch-level local features;

[0041] Mapping all patch-level local features to a fixed-dimensional embedding space through a linear projection layer to obtain all mapped patch-level local features, and performing optimization processing on all mapped patch-level local features to obtain all optimized patch-level local features, the optimization processing comprising residual connection and normalization processing.

[0042] ​According to the above description, the internal spatial correlation of a single image patch is captured by the convolutional neural network, so that the obtained patch-level local features retain respective detailed features, which are mapped to a fixed-dimensional embedding space by linear projection, eliminating spatial dimension differences, achieving dimension standardization, and performing residual connection to retain the original patch-level local features, avoiding feature degradation, and stabilizing the stable distribution of patch-level local features in the embedding space through normalization processing.

[0043] Optionally, the calculating the intersection of each patch-level visual feature and all Token-level visual encodings, and the mapping of the patch visual feature in the intersection to all Token-level language semantic features comprises:

[0044] Obtaining the row index and the column index of each image patch in the document image, inputting the row index and the column index into an absolute position formula for absolute position encoding calculation to obtain the absolute position encoding of each image patch, and inputting the row index and the column index of each two image patches into a relative position formula for relative position encoding calculation to obtain the relative position encoding of each image patch;

[0045] Weighted fusion of the absolute position encoding and the corresponding relative position encoding of each image patch to obtain a two-dimensional hybrid position encoding of each image patch, and adding the two-dimensional hybrid position encoding to the patch-level visual feature corresponding to the image patch to realize feature enhancement of the patch-level visual feature to obtain all enhanced patch-level visual features;

[0046] Obtaining the sequence position information and the document structure type of the Token-level text corresponding to each Token-level language semantic feature in the text sequence;

[0047] Meanwhile, local context information of each Token-level language semantic feature is calculated, and local context encoding is performed within a preset association window with each Token-level language semantic feature as the center to obtain local context information of each Token-level language semantic feature;

[0048] Weighted fusion of the local context information, the sequence position information and the document structure type of each Token-level language semantic feature to obtain a context position encoding of each Token-level language semantic feature, and adding the context position encoding to the corresponding Token-level language semantic feature to realize feature enhancement of the Token-level language semantic feature to obtain all enhanced Token-level language semantic features;

[0049] mapping all enhanced patch-level visual features in the intersection to all enhanced token-level language semantic features.

[0050] According to the above description, the two-dimensional hybrid position coding composed of absolute position coding and relative position coding is added in the patch-level visual features, not only the position information of the image patch itself is established, but also the associated position information between different image patches is established, the spatial disorder of the image patch is solved, and the spatial correlation of the patch-level visual features is ensured, the context position coding composed of local context information, sequence position information and document structure type is added in the token-level language semantic features, the semantic ambiguity is eliminated, the Token-level language semantic feature sequence disorder is avoided, and the logical correlation of the token-level language semantic features is established, since the corresponding position information is added in the enhanced patch-level visual features and the enhanced token-level language semantic features, the cross-modal mapping can be realized, the multi-scene requirements can be met, and the adaptability and flexibility are improved.

[0051] Optionally, the mapping all enhanced patch-level visual features in the intersection to all enhanced token-level language semantic features comprises:

[0052] calculating an effective attention range of each enhanced patch-level visual feature through a sparse attention mechanism, and performing feature association on the enhanced patch-level visual features in the effective attention range through a multi-head attention mechanism to obtain all feature-associated patch-level visual features;

[0053] performing multi-scale interpolation fusion on all feature-associated patch-level visual features to obtain multi-scale interpolation fused patch-level visual features, and triggering different pooling strategies according to the spatial layout of all token-level visual encodings to perform pooling processing on the multi-scale interpolation fused patch-level visual features, so as to realize the mapping of the multi-scale interpolation fused patch-level visual features to all enhanced token-level language semantic features.

[0054] According to the above description, the multi-head attention mechanism is used to associate the enhanced patch-level visual features, solve the problem of feature isolation and lack of context of a single enhanced patch-level visual feature, and calculate the effective attention range by using sparse attention. The feature association is performed in the effective attention range to avoid the interference of invalid enhanced patch-level visual features, reduce the parameter size, improve the inference speed of feature association, realize lightweight, and perform multi-scale interpolation fusion and adaptive pooling processing on all patch-level visual features after feature association. The detail discrimination of the patch-level visual features is further improved, and the problem of important information loss or noise introduction existing in the traditional fixed pooling processing is overcome, thereby ensuring the accuracy of the patch-level visual features after multi-scale interpolation fusion in mapping with all enhanced Token-level language semantic features.

[0055] Optionally, the method further comprises:

[0056] calculating Token-level similarity, phrase-level similarity, and sentence-level similarity between each of the mapped patch-level visual features and each of the mapped Token-level language semantic features;

[0057] aligning the mapped patch-level visual features and the mapped Token-level language semantic features at multiple levels according to the Token-level similarity, the phrase-level similarity, and the sentence-level similarity to obtain a multi-level alignment result;

[0058] aligning and fusing the multi-level alignment result to obtain a final alignment result, and implementing understanding of the document image according to the final alignment result.

[0059] According to the above description, the Token-level similarity, the phrase-level similarity, and the sentence-level similarity are used for multi-level alignment, which realizes progressive alignment from fine-grained, medium-grained, and coarse-grained, and aligns and fuses the multi-level alignment result, so that the final alignment result takes into account the multi-granularity requirement, improves the alignment accuracy, and improves the accuracy of document image understanding.

[0060] Optionally, the method further comprises:

[0061] calculating the alignment quality degree of the alignment result by using an alignment quality formula, and calculating the alignment consistency degree of the alignment result by using an alignment consistency detection formula;

[0062] The alignment quality degree and the alignment consistency degree are input into an alignment scoring formula to obtain a scoring result, and it is determined whether the scoring result is lower than a scoring threshold, and if so, the alignment model is optimized according to the scoring result to obtain an optimized alignment model.

[0063] The alignment quality formula is:

[0064] ;

[0065] wherein, indicates an alignment quality degree, N indicates a total number of mapped patch-level visual features and mapped Token-level language semantic features, indicates an i-th mapped patch-level visual feature, indicates an i-th mapped Token-level language semantic feature, indicates a final similarity of the i-th mapped patch-level visual feature and the i-th mapped Token-level language semantic feature.

[0066] The consistency detection formula is:

[0067] ;

[0068] wherein, indicates an alignment consistency degree, Var indicates a variance, indicates a final similarity of the i-th mapped patch-level visual feature and the i-th mapped Token-level language semantic feature, indicates a maximum variance, N indicates a total number of mapped patch-level visual features and mapped Token-level language semantic features.

[0069] The alignment scoring formula is:

[0070] ;

[0071] wherein, indicates a scoring result, indicates a quality weight coefficient of the alignment quality degree, indicates a consistency weight coefficient of the alignment consistency degree, indicates an alignment quality degree, indicates an alignment consistency degree.

[0072] According to the above description, the alignment quality degree and the alignment consistency degree are calculated for the alignment result, so that the alignment result is scored according to the alignment quality degree and the alignment consistency degree, the alignment model is optimized according to the scoring result, and the alignment accuracy of the alignment model is improved.

[0073] In a second aspect, the present invention provides a fine-grained document image understanding system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fine-grained document image understanding method described in the first aspect.

[0074] The technical effects of the fine-grained document image understanding system provided in the second aspect are the same as those of the fine-grained document image understanding method provided in the first aspect. Attached Figure Description

[0075] Figure 1 A flowchart illustrating a fine-grained document image understanding method provided in this embodiment;

[0076] Figure 2 This is a schematic diagram of the overall process of a fine-grained document image understanding method provided in this embodiment;

[0077] Figure 3 This is a schematic diagram illustrating the process of mapping the visual features of the intersection of patches to all token-level linguistic semantic features involved in this embodiment;

[0078] Figure 4 This is a schematic diagram of the structure of a fine-grained document image understanding system provided in this embodiment.

[0079] [Explanation of Labels in the Attached Image]

[0080] 1. A fine-grained document image understanding system;

[0081] 2. Processor;

[0082] 3. Memory. Detailed Implementation

[0083] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0084] Example 1

[0085] Please refer to Figures 1 to 3 This invention provides a method for understanding fine-grained document images, comprising the following steps:

[0086] S1, obtain an input document image, perform text content recognition on the document image to obtain a text sequence, obtain pixel coordinates of each character in the text sequence, map the pixel coordinates and the text sequence to obtain a text coordinate mapping table;

[0087] In the embodiment, as shown in Figure 2 the input document image is subjected to text content recognition to obtain a text sequence. At this time, a lightweight OCR model can be used for text content recognition. Pixel coordinates of each character in the text sequence are obtained to realize mapping of the pixel coordinates and the text sequence, and a text coordinate mapping table is constructed and obtained.

[0088] S2, perform Token-level segmentation processing on the text sequence by a BPE segmenter to obtain all Token-level texts, perform visual coding on all Token-level texts according to the text coordinate mapping table to obtain all Token-level visual codes, and perform language semantic feature extraction on all Token-level texts to obtain all Token-level language semantic features;

[0089] In the embodiment, Token-level segmentation processing is performed on the text sequence obtained in step S1 by a BPE segmenter to obtain all Token-level texts. Visual coding is performed on all Token-level texts according to the text coordinate mapping table. At this time, pixel coordinates of all characters constituting each Token-level text are extracted from the text coordinate mapping table, a minimum enclosing matrix of the pixel coordinates is calculated, a binary matrix with consistent dimensions of the document image is constructed, pixels in the minimum enclosing matrix are marked as 1 and the rest are marked as 0, and visual coding is performed on each Token-level text to obtain all Token-level visual codes. The Token-level visual codes at this time only contain pixel range information and do not contain language semantic information, and are used for positioning patch visual features of an image patch. Language semantic feature extraction is performed on all Token-level texts to obtain all Token-level language semantic features.

[0090] At this time, the step of performing Token-level segmentation processing on the text sequence by a BPE segmenter to obtain all Token-level texts includes:

[0091] S21. Obtain the basic vocabulary of the BPE segmenter, perform addition operations on the basic vocabulary including mathematical symbols, multilingual characters and document structure tags to obtain the basic vocabulary after the addition operations, perform token-level segmentation on the text sequence according to the basic vocabulary after the addition operations to obtain all token-level texts, including token-level texts that match the basic vocabulary after the addition operations and token-level texts that do not match the basic vocabulary after the addition operations.

[0092] S22. Token-level texts that do not match the basic vocabulary after the addition operation are taken as out-of-vocabulary words. The character length of each out-of-vocabulary word is calculated. The out-of-vocabulary words are optimized according to the character length. It is determined whether the character length is greater than the length threshold. If not, the out-of-vocabulary words are optimized by unknown word tagging to obtain optimized out-of-vocabulary words. If so, the out-of-vocabulary words are optimized by character-level encoding and character decomposition to obtain optimized out-of-vocabulary words.

[0093] S23. Update all token-level texts based on the optimized out-of-vocabulary words to obtain all updated token-level texts.

[0094] In this embodiment, as Figure 2 As shown, the BPE tokenizer's original basic vocabulary was expanded by adding mathematical symbols, multilingual characters, and document structure tags, thereby increasing the tokenization capabilities of the BPE tokenizer. Based on the basic vocabulary after the additions, the text sequence was processed for token-level tokenization, resulting in all token-level texts. The token-level texts included both those that matched the basic vocabulary after the additions and those that did not. For Token-level texts that do not match the basic vocabulary after the addition operation, they will be treated as out-of-vocabulary words. Differential optimization processing is applied to out-of-vocabulary words. It is determined whether the character length of the out-of-vocabulary word is greater than the length threshold. Considering that words with a character length of no more than 3 are usually abbreviations, symbols or function words, the semantic information is relatively simple and will not cause serious information loss, while words with a character length greater than 3 usually contain rich semantic information, the length threshold is set to 3. That is, out-of-vocabulary words with a character length of no more than 3 are directly optimized by unknown word labeling, and out-of-vocabulary words with a character length greater than 3 are directly decomposed into character sequences to obtain optimized out-of-vocabulary words. The optimized out-of-vocabulary words replace the original Token-level text, that is, the Token-level text is updated to obtain all the updated Token-level texts.

[0095] At this point, step S2 involves simultaneously extracting linguistic semantic features from all token-level texts, resulting in all token-level linguistic semantic features including:

[0096] S24. Generate the embedded character level corresponding to each token-level text using the character-level embedding formula;

[0097] S25. Generate the embedded sub-words corresponding to each Token-level text using the sub-word-level embedding formula;

[0098] S26. Generate the semantic level of embedding for each token-level text using semantic-level embedding formulas;

[0099] S27. By using a gating fusion mechanism, the embedded character level, embedded sub-word level and embedded semantic level of each Token-level text are dynamically fused to generate the Token-level language semantic features of each Token-level text, thus obtaining all Token-level language semantic features.

[0100] The character-level embedding formula is:

[0101] ;

[0102] in, This represents the embedded character level corresponding to the i-th token-level text. This represents the set of characters for the i-th token-level text. This represents a character embedding matrix, where c represents a single character;

[0103] The sub-word-level embedding formula is as follows:

[0104] ;

[0105] in, This represents the embedded sub-word level corresponding to the i-th token-level text. Represents the subword embedding matrix, Represents the line index of the i-th token-level text;

[0106] The semantic-level embedding formula is:

[0107] ;

[0108] in, This represents the embedding semantic level corresponding to the i-th token-level text. This represents a multilayer perceptron. This represents the part-of-speech tag of the i-th token-level text. This represents the entity label of the i-th token-level text. This represents the one-hot encoding function. Symbols indicating feature splicing.

[0109] In this embodiment, as Figure 2 As shown, character-level, word-level, and semantic-level embeddings are generated for each token-level text using character-level, word-level, and semantic-level embedding formulas. A gating fusion mechanism dynamically fuses these embeddings. This mechanism concatenates the character-level, word-level, and semantic-level embeddings along the feature dimensions and inputs them into a gating weight calculation network to generate three-dimensional dynamic gating weights: character-level weights, word-level weights, and semantic-level weights. The specific formulas are as follows:

[0110] ;

[0111] in, This represents the embedding character-level weight of the i-th token-level text. This represents the weight of the embedded sub-words of the i-th token-level text. This represents the semantic weight of the embedding of the i-th token-level text. express function, This represents a bias of dimension d. Let represent the gate weight matrix, and the dimension of the gate weight matrix is ​​d×3d.

[0112] Then, the embedded character level, embedded sub-word level, and embedded semantic level are weighted and fused using dynamic gating weights across three dimensions to generate the token-level linguistic semantic features for each token-level text. The specific formula is as follows:

[0113] ;

[0114] in, Token-level language semantic features.

[0115] S3. Perform patch segmentation on the document image to obtain K image patches. Extract visual features from each image patch to obtain all patch-level visual features. Calculate the intersection of each patch-level visual feature with all token-level visual codes. Map the patch visual features in the intersection with all token-level linguistic semantic features to obtain mapped patch-level visual features and mapped token-level linguistic semantic features. Input the mapped patch-level visual features and mapped token-level linguistic semantic features into a comparison model for alignment to obtain the alignment result. Understand the document image based on the alignment result.

[0116] At this point, step S3, which involves patching the document image to obtain K image patches, includes:

[0117] S31. Obtain the image width, image height, and document type of the document image;

[0118] S32. Input the image width and image height into the text density formula to calculate the text density of the document image. The text density formula is:

[0119] ;

[0120] in, H represents the text density of document image I, H represents the image height, and W represents the image width. An indicator function that indicates whether a pixel (x, y) is a text pixel;

[0121] S33. Calculate the optimal scaling ratio of the document image based on the text density, and scale the document image according to the optimal scaling ratio to obtain the scaled document image;

[0122] S34. Dynamically adjust the patch size according to the document type to perform adaptive patch segmentation on the scaled document image, resulting in K image patches.

[0123] In this embodiment, as Figure 2As shown, the document image's width, height, and document type are obtained. The text density of the document image is calculated using the formula for image height, width, and text density. During the optimal scaling calculation, the document image is preprocessed according to dimensions from a preset size set, resulting in a preprocessed document image. The text density is then recalculated based on the preprocessed image's width and height, yielding the preprocessed document image's document density. The optimal scaling ratio is then calculated based on all text densities. The document image is scaled according to this optimal scaling ratio, resulting in a scaled document image. The patch size is dynamically adjusted based on the document type to adaptively segment the scaled document image. Document types include: natural scene text, tables and charts, dense documents, and GIU interfaces. Natural scene text uses 16×16 pixels, tables and charts use 14×14 pixels, dense documents use 12×12 pixels, and GIU interfaces use 18×18 pixels. The preset size set is {0.5, 0.75, 1.0, 1.25, 1.5}, and the formula for the optimal scaling ratio is as follows:

[0124] ;

[0125] in, This indicates the optimal scaling ratio. This represents the text density of a document image after preprocessing it according to size s from a preset set of sizes. This represents the optimal text density threshold, s represents the current size, and S represents the preset size set. express function.

[0126] At this point, the visual feature extraction performed on each image patch in step S3 yields all patch-level visual features, including:

[0127] S35. Use a convolutional neural network to extract the internal local features of each image patch to obtain all patch-level local features;

[0128] S36. All patch-level local features are mapped to a fixed-dimensional embedding space through a linear projection layer to obtain all mapped patch-level local features. All mapped patch-level local features are then optimized to obtain all optimized patch-level local features. The optimization process includes residual connection and normalization.

[0129] In this embodiment, a 3×3 convolutional neural network is used to extract internal features from each image patch to obtain all patch-level local features. First, all patch-level local features are flattened to transform them into patch-level local features of the same size as the image patch. Then, a linear projection layer is used to map all patch-level local features to a fixed-dimensional embedding space, where the fixed dimension is d. Finally, all mapped patch-level local features are optimized by residual connections and normalization.

[0130] At this point, in step S3, when mapping the patch visual features in the intersection to all token-level linguistic semantic features, feature enhancement was performed on both the patch visual features and the token-level linguistic semantic features. The specific steps are as follows:

[0131] Obtain the row index and column index of each image patch in the document image, input the row index and column index into the absolute position formula to calculate the absolute position encoding, so as to obtain the absolute position encoding of each image patch. At the same time, input the row index and column index of every two image patches into the relative position formula to calculate the relative position encoding, so as to obtain the relative position encoding of each image patch.

[0132] The absolute position code of each image patch is weighted and fused with the corresponding relative position code to obtain a two-dimensional hybrid position code for each image patch. The two-dimensional hybrid position code is added to the patch-level visual feature corresponding to the image patch to enhance the patch-level visual feature, and all enhanced patch-level visual features are obtained.

[0133] Obtain the sequence position information and document structure type of the token-level text corresponding to each token-level linguistic semantic feature in the text sequence;

[0134] Simultaneously, local context information is calculated for each Token-level linguistic semantic feature. Local context encoding is performed within a preset association window, centered on each Token-level linguistic semantic feature, to obtain the local context information of each Token-level linguistic semantic feature.

[0135] The local context information, sequence position information and document structure type of each token-level linguistic semantic feature are weighted and fused to obtain the context position code of each token-level linguistic semantic feature. The context position code is added to the corresponding token-level linguistic semantic feature to enhance the token-level linguistic semantic feature, and all enhanced token-level linguistic semantic features are obtained.

[0136] Calculate the intersection of each enhanced patch-level visual feature with all token-level visual codes, and map all enhanced patch-level visual features in the intersection to all enhanced token-level linguistic semantic features.

[0137] In this embodiment, as Figure 3 As shown, the row and column indices of each image patch in the document image are input into the absolute position formula for absolute position encoding calculation. Simultaneously, the row and column indices of every two image patches are input into the relative position formula for relative position encoding calculation, thus obtaining the absolute and relative position codes for each image patch. The absolute and relative position codes are then weighted and fused to obtain a two-dimensional hybrid position code for an image patch. The absolute position formula is:

[0138] ;

[0139] in, This represents the absolute position encoding of the i-th image patch. This represents the row index of the i-th image patch. The column index represents the i-th image patch, k represents the frequency index, and d represents the dimension;

[0140] The formula for relative position is:

[0141] ;

[0142] in, This represents the relative position encoding of the i-th image patch. This represents the row index of the i-th image patch. This represents the column index of the i-th image patch. This represents the row index of the j-th image patch. This represents the column index of the j-th image patch. Indicates the maximum range of relative positions. express function;

[0143] Two-dimensional hybrid positional encoding is added to the patch-level visual features corresponding to the image patch to enhance the features of the patch-level visual features, resulting in all enhanced patch-level visual features. The sequence position of the token-level text in the text sequence and the document structure type corresponding to each token-level linguistic semantic feature are obtained. Simultaneously, local context information is calculated for each token-level linguistic semantic feature. During the calculation, local context encoding is performed within a preset association window centered on each token-level linguistic semantic feature to obtain the local context information of each token-level linguistic semantic feature. The local context information, sequence position information, and document structure type of each token-level linguistic semantic feature are weighted and fused to obtain the context position encoding of each token-level linguistic semantic feature. This context position encoding is added to the corresponding token-level linguistic semantic feature to enhance the features of the token-level linguistic semantic feature, resulting in all enhanced token-level linguistic semantic features. The intersection of each enhanced patch-level visual feature and all token-level visual encodings is calculated, and all enhanced patch-level visual features in the intersection are mapped to all enhanced token-level linguistic semantic features.

[0144] At this point, when mapping all enhanced patch-level visual features to all enhanced token-level linguistic semantic features, a sparse attention mechanism and a multi-head attention mechanism are used. The specific steps are as follows:

[0145] The effective attention range of each enhanced patch-level visual feature is calculated by a sparse attention mechanism, and the enhanced patch-level visual features are associated within the effective attention range by a multi-head attention mechanism to obtain the patch-level visual features after all features are associated.

[0146] The patch-level visual features, after associating all features, are subjected to multi-scale interpolation fusion to obtain multi-scale interpolated patch-level visual features. Different pooling strategies are triggered based on the spatial layout of all token-level visual codes to perform pooling processing on the multi-scale interpolated patch-level visual features, so as to realize the mapping of the multi-scale interpolated patch-level visual features with all enhanced token-level linguistic semantic features.

[0147] In this embodiment, as Figure 3As shown, the effective attention range of each enhanced patch-level visual feature is calculated through a sparse attention mechanism. The effective attention and range include attention within a preset local window and attention corresponding to the globally key token text. The globally key token text is selected from all token-level texts according to preset key rules. The enhanced patch-level visual features are then associated within the effective attention range through a multi-head attention mechanism to obtain the patch-level visual features after all feature associations. The patch-level visual features after all feature associations are then fused using multi-scale interpolation, with the multi-scale values ​​being 0.5, 1.0, and 2.0. Different pooling strategies are triggered based on the spatial layout of all token-level visual codes to pool the multi-scale interpolated patch-level visual features. For example, when the spatial layout of all token-level visual codes is compact, an average pooling strategy is used; when the spatial layout of all token-level visual codes is complex, an attention pooling strategy is used; and when the spatial layout of all token-level visual codes is dense, a max pooling strategy is adopted.

[0148] At this point, step S3, which involves aligning the mapped patch-level visual features with the mapped token-level linguistic semantic features in the comparison model to obtain the alignment result, and then using the alignment result to understand the document image, includes:

[0149] S37. Calculate the token-level similarity, phrase-level similarity, and sentence-level similarity between each mapped patch-level visual feature and each mapped token-level linguistic semantic feature.

[0150] S38. Align the mapped patch-level visual features with the mapped token-level linguistic semantic features at multiple levels based on the token-level similarity, the phrase-level similarity, and the sentence-level similarity to obtain a multi-level alignment result.

[0151] S39. Align and merge the multi-level alignment results to obtain the final alignment result, and understand the document image based on the final alignment result.

[0152] In this embodiment, the token-level similarity, phrase-level similarity, and sentence-level similarity between each mapped patch-level visual feature and each mapped token-level linguistic semantic feature are calculated. When calculating token-level similarity, the token-level complexity of the current token-level linguistic semantic feature is first calculated, and then the similarity formula is adaptively selected based on the token-level complexity. Similarly, when calculating phrase-level similarity, the phrase-level complexity of the current token-level linguistic semantic feature is first calculated, and then the similarity formula is adaptively selected based on the phrase-level complexity. Likewise, when calculating sentence-level similarity, the sentence-level complexity of the current token-level linguistic semantic feature is first calculated, and then the similarity formula is adaptively selected based on the sentence-level complexity. The following principles are followed when selecting the similarity formula: when the calculated complexity is less than a first complexity threshold, the cosine similarity formula is used; when the calculated complexity is between the first and second complexity thresholds, the bilinear similarity formula is used; and when the calculated complexity is greater than or equal to the second threshold, the MLP similarity formula is used.

[0153] Therefore, based on token-level similarity, phrase-level similarity, and sentence-level similarity, the mapped patch-level visual features and the mapped token-level linguistic semantic features are aligned at multiple levels to obtain multi-level alignment results. The multi-level alignment results are then fused to obtain the final alignment result, and the document image is understood based on the final alignment result.

[0154] In this embodiment, the final alignment result is also evaluated to optimize the alignment model in reverse. The specific steps are as follows:

[0155] The alignment quality score of the alignment result is calculated using the alignment quality formula, and the alignment consistency score of the alignment result is calculated using the alignment consistency detection formula.

[0156] The alignment quality score and the alignment consistency score are input into the alignment scoring formula for scoring to obtain a scoring result. It is then determined whether the scoring result is lower than the scoring threshold. If so, the alignment model is optimized based on the scoring result to obtain an optimized alignment model.

[0157] The alignment quality formula is:

[0158] ;

[0159] in, This represents the alignment quality score, where N represents the total number of mapped patch-level visual features and mapped token-level linguistic semantic features. This represents the patch-level visual feature after the i-th mapping. This represents the token-level linguistic semantic features after the i-th mapping. This represents the final similarity between the patch-level visual features after the i-th mapping and the token-level linguistic semantic features after the i-th mapping;

[0160] The consistency detection formula is as follows:

[0161] ;

[0162] in, Indicates alignment consistency; Var represents variance. This represents the final similarity between the patch-level visual features after the i-th mapping and the token-level linguistic semantic features after the i-th mapping. denoted by , where represents the maximum variance, and N represents the total number of mapped patch-level visual features and mapped token-level linguistic semantic features;

[0163] The alignment scoring formula is as follows:

[0164] ;

[0165] in, Indicates the scoring result. The quality weighting coefficient represents the alignment quality score. The consistency weighting coefficient represents the degree of alignment consistency. Indicates alignment quality. Indicates the degree of alignment consistency.

[0166] In this embodiment, as Figure 2As shown, the alignment quality score of the alignment result is calculated using the alignment formula, and the alignment consistency score is calculated using the alignment consistency detection formula. The alignment result is scored based on the obtained alignment quality score, alignment consistency score, and alignment scoring formula. If the score is lower than the scoring threshold, the comparison model is optimized based on the score. During the optimization process, the comparison model calculates the comparison loss for token-level similarity, phrase-level similarity, and sentence-level similarity, and dynamically fuses them through a loss dynamic fusion mechanism to achieve joint optimization. This optimization is then traced back to the pre-trained visual feature extraction model used for visual feature extraction and the pre-trained language semantic feature extraction model used for language semantic feature extraction, resulting in an optimized comparison model. It should be noted that the pre-trained visual feature extraction model and the pre-trained language semantic feature extraction model are used for visual feature extraction and language semantic feature extraction, respectively, and they are jointly trained in the early stages. Furthermore, an intelligent negative sample mining strategy is used to dynamically construct high-quality negative samples for the construction of training samples.

[0167] Example 2

[0168] Please refer to Figure 4 The present invention provides a fine-grained document image understanding system 1, including a memory 3, a processor 2, and a computer program stored on the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, it implements the steps in Embodiment 1.

[0169] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0170] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0172] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0173] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0174] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0175] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A fine particle document image understanding method characterized by, The method comprises the following steps: obtaining an input document image, performing text content recognition on the document image to obtain a text sequence, obtaining pixel coordinates of each character in the text sequence, mapping the pixel coordinates and the text sequence to obtain a text coordinate mapping table; performing Token-level segmentation processing on the text sequence by using a BPE tokenizer to obtain all Token-level texts, performing visual coding on all Token-level texts according to the text coordinate mapping table to obtain all Token-level visual codes, and performing language semantic feature extraction on all Token-level texts to obtain all Token-level language semantic features, wherein the Token-level visual codes are obtained based on the following steps: extracting pixel coordinates of all characters constituting each Token-level text from the text coordinate mapping table, calculating a minimum enclosing matrix of the pixel coordinates, constructing a binary matrix with consistent dimensions of the document image, marking the pixel coordinates in the minimum enclosing matrix as 1 and marking the remaining pixels as 0, thereby realizing visual coding of each Token-level text to obtain all Token-level visual codes; Token-level visual codes only contain pixel range information and do not contain language semantic information, and are used for positioning patch visual features of an image patch; performing patch segmentation on the document image to obtain K image patches, performing visual feature extraction on each image patch to obtain all patch-level visual features, calculating a spatial position intersection of each patch-level visual feature and all Token-level visual codes, mapping patch visual features in the spatial position intersection and all Token-level language semantic features to obtain mapped patch-level visual features and mapped Token-level language semantic features, inputting the mapped patch-level visual features and the mapped Token-level language semantic features into a comparison model for alignment, obtaining an alignment result based on Token-level similarity, phrase-level similarity and sentence-level similarity, and realizing understanding of the document image according to the alignment result.

2. The fine particle document image understanding method of claim 1, wherein, The Token-level segmentation processing on the text sequence by using the BPE tokenizer to obtain all Token-level texts comprises the following steps: obtaining a basic vocabulary table of the BPE tokenizer, performing addition operation including mathematical symbols, multi-language characters and document structure markers on the basic vocabulary table to obtain an added basic vocabulary table, performing Token-level segmentation processing on the text sequence according to the added basic vocabulary table to obtain all Token-level texts, and all Token-level texts include Token-level texts matched with the added basic vocabulary table and Token-level texts not matched with the added basic vocabulary table; The Token-level text that does not match the basic vocabulary table after the addition operation is taken as an out-of-vocabulary word, the character length of each out-of-vocabulary word is calculated, the out-of-vocabulary word is optimized according to the character length, it is judged whether the character length is greater than a length threshold, if not, the out-of-vocabulary word is optimized for unknown word marking to obtain an optimized out-of-vocabulary word, and if yes, the out-of-vocabulary word is optimized for character-level coding and character decomposition to obtain an optimized out-of-vocabulary word; All Token-level texts are updated according to the optimized out-of-vocabulary word to obtain updated Token-level texts.

3. The fine particle document image understanding method of claim 1, wherein, The language semantic feature extraction of all Token-level texts simultaneously obtains all Token-level language semantic features, which includes: An embedded character level corresponding to each Token-level text is generated through a character-level embedding formula; An embedded subword level corresponding to each Token-level text is generated through a subword-level embedding formula; An embedded semantic level corresponding to each Token-level text is generated through a semantic-level embedding formula; Token-level language semantic features of each Token-level text are dynamically fused through a gating fusion mechanism to generate an embedded character level, an embedded subword level and an embedded semantic level of each Token-level text, and all Token-level language semantic features are obtained; The character-level embedding formula is: ； wherein, represents the embedding character level corresponding to the i-th Token level text, represents the character set of the i-th Token level text, represents the character embedding matrix, c represents a single character; The subword-level embedding formula is: ； wherein, represents the embedding subword level corresponding to the i-th Token level text, represents the subword embedding matrix, represents the row index of the i-th Token level text; The semantic-level embedding formula is: ； wherein, denotes the embedding semantic level corresponding to the i-th Token level text, denotes a multi-layer perceptron, denotes the part-of-speech tag of the i-th Token level text, denotes the entity tag of the i-th Token level text, denotes a one-hot encoding function, denotes a feature concatenation symbol.

4. The fine particle document image understanding method of claim 1, wherein, The patch segmentation of the document image to obtain K image patches includes: An image width, an image height and a document type of the document image are obtained; The image width and the image height are input into a text density formula to calculate a text density of the document image, and the text density formula is: ； wherein, denotes the text density of the document image I, H denotes the image height, W denotes the image width, denotes an indicator function that indicates whether a pixel (x, y) is a text pixel or not. An optimal scaling ratio of the document image is calculated according to the text density, and the document image is scaled according to the optimal scaling ratio to obtain a scaled document image; The patch size is dynamically adjusted according to the document type to perform adaptive-size patch segmentation on the scaled document image to obtain K image patches.

5. The fine particle document image understanding method of claim 1, wherein, The visual feature extraction of each image patch to obtain all patch-level visual features includes: An internal local feature of each image patch is extracted by using a convolutional neural network to obtain all patch-level local features; All patch-level local features are mapped to an embedded space with a fixed dimension through a linear projection layer to obtain all mapped patch-level local features, and the mapped patch-level local features are optimized to obtain all optimized patch-level local features, and the optimization includes residual connection and normalization processing.

6. The fine particle document image understanding method of claim 1, wherein, The calculation of the spatial position intersection of each patch-level visual feature and all Token-level visual encodings, and the mapping of the patch visual features in the spatial position intersection to all Token-level language semantic features include: obtaining the row index and the column index of each image patch in the document image, inputting the row index and the column index into an absolute position formula to perform absolute position encoding calculation to obtain the absolute position encoding of each image patch, and inputting the row index and the column index of each two image patches into a relative position formula to perform relative position encoding calculation to obtain the relative position encoding of each image patch; performing weighted fusion on the absolute position encoding and the corresponding relative position encoding of each image patch to obtain the two-dimensional hybrid position encoding of each image patch, adding the two-dimensional hybrid position encoding to the patch-level visual feature corresponding to the image patch to implement feature enhancement on the patch-level visual feature, and obtaining all enhanced patch-level visual features; obtaining the sequence position information and the document structure type of each Token-level language semantic feature in the text sequence; performing local context information calculation on each Token-level language semantic feature, respectively performing local context encoding within a preset correlation window with each Token-level language semantic feature as the center to obtain the local context information of each Token-level language semantic feature; performing weighted fusion on the local context information, the sequence position information and the document structure type of each Token-level language semantic feature to obtain the context position encoding of each Token-level language semantic feature, adding the context position encoding to the corresponding Token-level language semantic feature to implement feature enhancement on the Token-level language semantic feature, and obtaining all enhanced Token-level language semantic features; calculating the spatial position intersection of each enhanced patch-level visual feature and all Token-level visual encodings, and mapping all enhanced patch-level visual features in the spatial position intersection and all enhanced Token-level language semantic features.

7. A fine particle document image understanding method according to claim 6, wherein, The mapping of all enhanced patch-level visual features in the spatial position intersection and all enhanced Token-level language semantic features comprises: calculating the effective attention range of each enhanced patch-level visual feature through a sparse attention mechanism, and performing feature association on the enhanced patch-level visual feature within the effective attention range through a multi-head attention mechanism to obtain all feature-associated patch-level visual features; performing multi-scale interpolation fusion on all feature-associated patch-level visual features to obtain multi-scale interpolation fused patch-level visual features, triggering different pooling strategies according to the spatial layout of all Token-level visual encodings to perform pooling processing on the multi-scale interpolation fused patch-level visual features, so as to map the multi-scale interpolation fused patch-level visual features and all enhanced Token-level language semantic features.

8. The fine particle document image understanding method of claim 1, wherein, The mapped patch-level visual features and the mapped Token-level language semantic features are input into a comparison model for alignment, Token-level similarity, phrase-level similarity, and sentence-level similarity are obtained based on the alignment results, and understanding of the document image is realized according to the alignment results, which includes: Token-level similarity, phrase-level similarity, and sentence-level similarity between each mapped patch-level visual feature and each mapped Token-level language semantic feature are calculated; Multi-level alignment results are obtained by aligning the mapped patch-level visual features and the mapped Token-level language semantic features according to the Token-level similarity, the phrase-level similarity, and the sentence-level similarity; The final alignment results are obtained by aligning and fusing the multi-level alignment results, and understanding of the document image is realized according to the final alignment results.

9. A fine particle document image understanding method according to claim 8, wherein, Further comprising: The alignment quality degree of the alignment results is calculated by an alignment quality formula, and the alignment consistency degree of the alignment results is calculated by an alignment consistency detection formula; The alignment quality degree and the alignment consistency degree are input into an alignment scoring formula for scoring to obtain a scoring result, it is judged whether the scoring result is lower than a scoring threshold, if yes, the comparison model is optimized according to the scoring result to obtain an optimized comparison model; The alignment quality formula is: ； wherein, represents the alignment quality degree, N represents the total number of the mapped patch-level visual features and the mapped Token-level language semantic features, represents the i-th mapped patch-level visual feature, represents the i-th mapped Token-level language semantic feature, represents the final similarity between the i-th mapped patch-level visual feature and the i-th mapped Token-level language semantic feature; The consistency detection formula is: ； wherein, represents the alignment consistency degree, Var represents the variance, represents the final similarity between the i-th mapped patch-level visual feature and the i-th mapped Token-level language semantic feature, represents the maximum variance, N represents the total number of the mapped patch-level visual features and the mapped Token-level language semantic features; The alignment scoring formula is: ； wherein, represents a score result, represents a quality weight coefficient of the alignment quality degree, represents a consistency weight coefficient of the alignment consistency degree, represents an alignment quality degree, represents an alignment consistency degree.

10. A fine particle document image understanding system comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to realize the method of any one of claims 1 to 9.

Images