An irregularly arranged invoice document layout prediction method and device and a storage medium

Abstract

The application discloses an irregular layout invoice document layout prediction method, a prediction device and a storage medium, and the irregular layout invoice document layout prediction method comprises the following steps: collecting irregular layout document information by using a scanner or a camera; labeling the collected irregular layout document information and dividing the labeled data into a training set and a test set; obtaining a layout prediction model by training the training set; testing the effect of the layout prediction model by using the test set, adjusting the hyperparameters of the model according to the test effect, and obtaining an ideal model until the irregular layout invoice document layout is predicted by the prediction model. The application has the advantages that the high-dimensional visual features, low-dimensional visual random changes and semantic features of the irregular layout invoice document are combined to predict the layout, the problems of less semantic information and simple visual semantic fusion in traditional layout analysis are solved, and the accuracy of layout analysis of irregular invoices is improved.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, and in particular relates to a method for analyzing and predicting the layout of irregularly formatted invoices and documents. Background Technology

[0002] Documents are among the most information-rich data formats found in many real-world documents, such as financial statements, scientific literature, and purchase lists. Beyond text content, the table structure is crucial for extracting key information. Furthermore, document layout analysis is a key step in automatically understanding documents, supporting many important applications such as document retrieval, digitization, and editing. Its goal is to identify areas of interest in unstructured documents and determine the function of each area. Therefore, document structure recognition has become one of the important technologies in current document understanding systems. However, the diversity and complexity of invoice document layouts make the task of analyzing irregularly formatted invoice document layouts challenging.

[0003] Recent researchers have focused solely on visual or semantic features, neglecting the fact that combining information from both models can help in better identifying document layouts, especially for irregularly formatted invoices and invoices. This is because certain structured areas, such as graphics and tables, can be easily identified through macroscopic visual features, while semantic features are crucial for visually similar areas, such as abstracts and paragraphs. Therefore, some recent studies have attempted to combine these two models.

[0004] Natural Language Processing (NLP)-based methods model layout analysis as a sequence labeling task and apply a bottom-up strategy. They first serialize the text into a one-dimensional sequence of labels. Then, using the semantic and visual features of each label (such as coordinates and image embeddings), they sequentially determine the label order through a sequence labeling model. However, NLP-based methods have shown shortcomings in layout modeling.

[0005] Computer vision (CV)-based methods model layout analysis as an object detection or segmentation task and apply a top-down strategy. They first extract visual features using convolutional neural networks and then introduce sentence-level or character-level semantic features through text embedding maps, which are directly concatenated as a document representation. Then, detection or segmentation models are used to generate coordinates and semantic labels for layout components. Compared to NLP-based methods, CV-based methods capture spatial information better. Summary of the Invention

[0006] This invention provides a method, apparatus, and storage medium for predicting irregularly formatted invoice document layouts by obtaining precise coordinates and semantic tags, thereby improving the prediction accuracy of irregularly formatted invoice document layout analysis.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0008] A method for analyzing the layout of irregularly formatted invoices and invoices, characterized by comprising:

[0009] Step 1: Collect information from irregularly formatted documents;

[0010] Step 2: Label the collected irregularly formatted document information and divide the labeled data into training set and test set; each set of labeled data includes text annotation information, text selection information and image information;

[0011] Step 3: Train the layout prediction model using the training set: Use document images and text embeddings as inputs to the visual module and semantic module, respectively. The visual module uses convolutional neural network backbones of different sizes to extract high- and low-dimensional visual features; the semantic module extracts multi-scale document semantic features; and through the relation feature aggregation module, combine the visual features and semantic features to learn a multimodal feature map to obtain semantic labels and coordinates.

[0012] Step 4: Use the test set to test the effect of the layout prediction model, adjust the hyperparameters of the layout prediction model according to the test results, and repeat Step 3 until a layout prediction model that meets the test results is obtained.

[0013] Step 5: Predict the semantic tags of irregularly formatted invoices and documents using the layout prediction model to obtain the corresponding layout.

[0014] Preferably, the irregular document information annotation in step 2 includes the document RGB image, and the text positioning and annotation includes character-level and sentence-level annotation.

[0015] Preferably, step 3 includes the following:

[0016] Step 3-1: Use the document images in the training set as input to the visual module. Multi-dimensional visual features are extracted using convolutional neural networks (CNNs) at different scales. These features are represented as follows: Where H represents the x-height of the input image, and W represents the x-width of the input image. Representing feature map S i Channel dimension;

[0017] Step 3-2: The semantic module embeds the text into a mapping. As input, document semantics are introduced, and T0 has the same spatial size as the document image x. Indicates the initial channel size. The characters on the documentation page represent this. The sentence is represented as Where n and m are the total number of characters and the total number of sentences, respectively, and c k and It represents the k-th character and its associated bounding box, where (x0, y0) and (x1, y1) are the top-left and bottom-right pixel coordinates, respectively. Similarly, s can be obtained. k and Obtain multi-dimensional document semantic features W1, W2, W3, ...;

[0018] Step 3-3: Connect the visual features S1, S2, S3, ... and the semantic features W1, W2, W3, ... using an attention mechanism, and use them as input to the convolutional layer to learn the aggregated visual and semantic multimodal features MF. i The standard object detection or segmentation model Mask R-CNN is used to generate candidate components in the document, a graph neural network (GNN) is used to model the relationships between the components, and this is used to optimize the prediction results;

[0019] Steps 3-4: Calculate the two-part loss L(Θ) = L cls +λL gen The cross-entropy loss of semantic labels is used as the classification loss L. cls Smoothing L1 loss is used as the coordinate regression loss L gen ;

[0020] Steps 3-5: Backpropagation, using stochastic gradient descent (SGD) to optimize the model parameters Θ;

[0021] Steps 3-6: Read the document image and text embedding information of the next set of sequences, then jump to step 3-1 until all the data in the training set has been read;

[0022] Step 3-7: Repeat steps 3-1 to 3-6 and continue to optimize the model parameter Θ until L(Θ) drops to a reasonable order of magnitude.

[0023] More preferably, step 3-1 includes:

[0024] The document image's high- and low-dimensional features are extracted using convolutional kernels of different sizes through the Residual Block module, and the output features are... The calculation formulas are as follows: (1) (2):

[0025] S0=x#(1)

[0026]

[0027] H represents the x-height of the input image, and W represents the x-width of the input image. Representing feature map S i The channel dimension, f i，j(·) represents the Residual block(·) module, where c and l are hyperparameters.

[0028] More preferably, step 3-2 includes:

[0029] Use two mapping functions to and Construct character embedding maps for input and sentence embedding mapping The calculation formulas are shown in (3) and (4):

[0030]

[0031]

[0032] E c (·) is a word embedding layer, E s (·) The pre-trained language model Bert is used, and then the text embedding mapping W0 is obtained by applying LayerNormal(·) to the sum of Char and Sent. The calculation formula is shown in (5):

[0033] W0=LayerNormal(Char+Sent)#(5)

[0034] More preferably, step 3-3 includes:

[0035] Different dimensions of visual features and semantic features S i and W i As input to the attention mechanism module, the two modal features are concatenated and fed into the convolutional layer to obtain multimodal features (MF). i The calculation formulas are shown in (6) and (7):

[0036] Att i =MultiAtt([S i W i ])#(6)

[0037] MF i =Att i ⊙S i +(1-Att i )⊙W i #(7)

[0038] Where [·, ·] denote cascaded operations, MultiAtt(·) denotes a multi-head attention module, and ⊙ denotes element-wise product. A set of fused multimodal features MF is generated by aggregating visual and semantic features. iThen, a graphical neural network (GNN) is used to model the component relationships.

[0039] The document is represented as a graph G = (O, E), where O = {o1, o2, ..., o3}. n Let} represent the set of nodes, and E represent the set of edges. Each node represents a candidate component previously generated by the object detection model, and each edge represents the relationship between two candidate components. Different regions in the document may have close dependencies, so all regions constitute neighbor relationships, meaning the document graph is a fully connected graph.

[0040] Each node is represented as o j =(l j f j ), including position coordinates l j and depth features f j =RoIAlign(MF, l j Construct a new node feature representation z that integrates the two types of information. j The calculation formula is shown in (8):

[0041]

[0042] in This represents the embedding vector of the j-th node. Simultaneously, a self-attention mechanism is applied to the nodes to learn the relationships between them, and the updated node features are z′. j The detection result of the j-th node is optimized to obtain The calculation formulas are shown in (9) and (10):

[0043]

[0044]

[0045] In the formula Let be the probability of belonging to class c. It is its precise regression coordinates, Linear reg and Linear cls It is the projection layer. GNN models the pairwise relationships between tags and predicts their semantic labels.

[0046] More preferably, in steps 3-4, the loss function consists of two parts, L. cls and L gen The composition and calculation formula are shown in (11):

[0047] L(Θ)=L cls +λL gen #(11)

[0048] The cross-entropy loss of semantic labels is used as the classification loss L. cls Smoothing L1 loss is used as the coordinate regression loss L gen λ is a hyperparameter.

[0049] The present invention also provides an irregular layout prediction device for invoices and documents, including a processor and a memory; the memory stores a program or instructions, which are loaded and executed by the processor to implement the regular layout prediction method for invoices and documents.

[0050] The present invention also provides a computer-readable storage medium storing a program or instructions, which, when executed by a processor, implement the rule-based invoice document layout prediction method.

[0051] Beneficial effects

[0052] This invention introduces high- and low-dimensional visual information through document images and semantic information through text embedding mapping. By fusing visual and semantic features using a relational feature aggregation module and a component relational module based on a graph neural network, this model ultimately achieves layout analysis and prediction for irregularly formatted invoices and documents.

[0053] (1) This invention extracts the visual and semantic features of irregularly laid out invoices and introduces macroscopic and microscopic visual features reflected by different granularities of visual features.

[0054] (2) In the relation feature aggregation module, the present invention adopts a self-attention mechanism to fuse multimodal features and uses graph neural network to model the relationship between irregularly formatted invoice document components to generate accurate coordinates and semantic labels.

[0055] (3) This invention combines classification and generation errors in the loss function to improve the accuracy of analysis and prediction of irregularly formatted invoice document layout. Attached Figure Description

[0056] Figure 1 This is a network framework diagram of the prediction model of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] This embodiment provides a method for predicting the layout of irregularly formatted invoices and invoices, based on... Figure 1 The prediction model shown makes predictions, and the prediction method includes the following steps:

[0059] Step 1: Use a scanner or camera to collect information from irregularly formatted documents; the document information data format can be pdf, jpg, etc., to obtain clear document text and spatial information;

[0060] Step 2: Annotate the collected irregularly formatted document information using PPOCR, and divide the data into training set and test set; one set of data includes text annotation information, text selection information, and image information;

[0061] Step 3: Train the layout prediction model using the training set: Use document images and text embeddings as inputs to the visual module and semantic module, respectively. The visual module uses convolutional neural network (CNN) backbones of different sizes to extract high- and low-dimensional visual features; the semantic module uses a BERT pre-trained language model to extract multi-scale document semantics using a text embedding map; and the relation feature aggregation module aggregates different morphological features, i.e., visual and semantic features, to learn a multimodal feature map, ultimately obtaining semantic labels and coordinates.

[0062] Step 4: Use the test set to test the effect of the layout prediction model, adjust the hyperparameters of the model according to the test results, and repeat Step 3 until the ideal model is obtained.

[0063] Step 5: Predict the semantic tags of irregularly formatted invoices using a predictive model to obtain their layout.

[0064] To facilitate public understanding, the technical solution of the present invention will be described in detail below through a preferred embodiment and in conjunction with the accompanying drawings.

[0065] In step 1, use a scanner or camera to collect information from irregularly formatted documents; the document information data format can be pdf, jpg, etc., which can produce clear document text and spatial information.

[0066] In step 2, the collected irregularly formatted document information is annotated using PPOCR (see details: https: / / github.com / PaddlePaddle / PaddleOCR), and the data is divided into training set and test set; one set of data includes text annotation information, text selection information, and image information.

[0067] First, the collected irregular layout information is used to obtain text bounding boxes in COCO dataset format and their corresponding character-level and sentence-level text annotations using PPOCR.

[0068] Secondly, count the number of entries and divide the dataset into training and test sets in an 8:2 ratio.

[0069] Finally, the RGB information of the overall image of the document is obtained, which is then used to facilitate training and conversion to grayscale values.

[0070] Step 3: Use the training set data to train and optimize the prediction model.

[0071] The document image and text embedding are used as inputs to the visual and semantic modules, respectively. The visual module uses convolutional neural networks (CNNs) of different sizes to extract high- and low-dimensional visual features. The semantic module uses a BERT pre-trained language model to extract multi-scale document semantics using a text embedding map. A relational feature aggregation module combines different morphological features, i.e., visual and semantic features, to learn a multimodal feature map. Finally, semantic labels and coordinates are obtained. The specific training steps of the model are as follows:

[0072] Step 3-1: The vision module uses convolutional neural networks (CNNs) of different scales to extract multi-dimensional visual features from document images in the training set.

[0073] Input: Images from the training set

[0074] Processing: Convolutional kernels process the image;

[0075] Output: Features in different dimensions

[0076] Where H represents the height of the input image x, and W represents the width of the input image x. Representing feature map S i Channel dimension;

[0077] The document image's high- and low-dimensional features are extracted using convolutional kernels of different sizes through the Residual Block module, and the output features are... The calculation formulas are as follows: (1) (2):

[0078] S0=x (1)

[0079]

[0080] Among them, f i，j (·) represents the Residual block(·) module; i = 1, 2, ..., l; c and l are hyperparameters.

[0081] Step 3-2: The semantic module processes text embedding mappings to obtain multi-dimensional document semantic features;

[0082] Input: Text embedding map Indicates the initial channel size;

[0083] Processing: Utilize a word embedding layer and a pre-trained language model BERT, combined with LayerNormal(·);

[0084] Output: Text semantic features W i ;

[0085] The semantic module embeds text into a mapping. Document semantics are introduced as input, having the same spatial size as the document image x. The characters of the document page are represented as follows: The sentence is represented as Where n and m are the total number of characters and sentences, respectively; c k and It represents the k-th character and its associated bounding box, where (x0, y0) and (x1, y1) are the top-left and bottom-right pixel coordinates; s k and Given the k-th sentence and its associated bounding box, we obtain the multi-dimensional document semantic features W1, W2, W3, ...;

[0086] Use two mapping functions to and Construct a character embedding map for the input. and sentence embedding mapping The calculation formulas are shown in (3) and (4):

[0087]

[0088]

[0089] Among them, E c (·) is a word embedding layer, E s (·) The pre-trained language model Bert is used, and then LayerNormal(·) is applied to the sum of Char and Sent to obtain the text semantic features W0. The calculation formula is shown in (5):

[0090] W0=LayerNormal(Char+Sent) (5)

[0091] Step 3-3: Connect the visual features S1, S2, S3, ... and the semantic features W1, W2, W3, ... using an attention mechanism, and use this connection as input to the convolutional layer to learn the aggregated visual and semantic multimodal features MF. i The standard object detection or segmentation model Mask R-CNN is used to generate candidate components in the document, a graph neural network (GNN) is used to model the relationships between the components, and this is used to optimize the prediction results.

[0092] Different dimensions of visual features and semantic features S i and W i As input to the attention mechanism module, the two modal features are concatenated and fed into the convolutional layer to obtain multimodal features (MF). i The calculation formulas are shown in (6) and (7):

[0093] Att i =MultiAtt([S i W i (6)

[0094] MF i =Att i ⊙S i +(1-Att i )⊙W i (7)

[0095] Where [·, ·] denote cascaded operations, MultiAtt(·) denotes a multi-head attention module, and ⊙ denotes element-wise product. A set of fused multimodal features MF is generated by aggregating visual and semantic features. i Then, a graphical neural network (GNN) is used to model the component relationships.

[0096] The document is represented as a graph G = (O, E), where O = {o1, o2, ..., o3}. n Let} represent the set of nodes, and E represent the set of edges. Each node represents a candidate component previously generated by the object detection model, and each edge represents the relationship between two candidate components. Different regions in the document may have close dependencies, so all regions constitute neighbor relationships, meaning the document graph is a fully connected graph.

[0097] Each node is represented as o j =(l j f j ), including position coordinates l j and depth features f j =RoIAlign(MF, l j Construct a new node feature representation z that integrates the two types of information. j The calculation formula is shown in (8):

[0098]

[0099] in, This represents the embedding vector of the j-th node. Simultaneously, a self-attention mechanism is applied to the nodes to learn the relationships between them, and the updated node features are z′. j The detection result of the j-th node is optimized to obtain The calculation formulas are shown in (9) and (10):

[0100]

[0101]

[0102] In the formula, Let be the probability of belonging to class c. It is its precise regression coordinates, Linear reg and Linear cls It is the projection layer. GNN models the pairwise relationships between tags and predicts their semantic labels.

[0103] Steps 3-4: Calculate the loss function, which consists of two parts, L. cls and L gen The composition and calculation formula are shown in (11):

[0104] L(Θ)=L cls +λL gen (11)

[0105] The cross-entropy loss of semantic labels is used as the classification loss L. cls Smoothing L1 loss as coordinate regression loss L gen λ is a hyperparameter;

[0106] Steps 3-5: Backpropagation, using stochastic gradient descent (SGD) to optimize the model parameters Θ, the calculation formula is shown in (12):

[0107]

[0108] Steps 3-6: Read the document image and text embedding information of the next set of sequences, then jump to step 3-1 until all the data in the training set has been read;

[0109] Step 3-7: Repeat steps 3-1 to 3-6 and continue to optimize the model parameter Θ until L(Θ) drops to a reasonable order of magnitude.

[0110] Step 4: Use the test set to test the effect of the layout prediction model, adjust the hyperparameters of the model according to the test results, and repeat Step 3 until the ideal model is obtained.

[0111] The test set is deployed on the layout test model for testing to check the model's performance. Based on the model's performance, hyperparameter information is adjusted, such as the learning rate, the total number of characters and sentences (n and m), c and l from step 3-1, and λ for adjusting the error from step 3-4, etc. Then, proceed to step 3 for retraining until the model achieves satisfactory results on the test set. This step yields the final layout prediction model.

[0112] Step 5: Predict the semantic tags of irregularly formatted invoices using a predictive model to obtain their layout.

[0113] Once the prediction result is obtained, further in-depth analysis can be conducted on this irregularly formatted invoice document.

[0114] This embodiment provides a device for predicting the layout of irregularly formatted invoices and documents, including a processor and a memory; the memory stores programs or instructions, which are loaded and executed by the processor to implement the method for predicting the layout of regularly formatted invoices and documents in this embodiment.

[0115] This embodiment provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the rule-based invoice document layout prediction method of this embodiment.

[0116] The irregular layout prediction method, device, and storage medium for invoices provided by this invention can be implemented in many ways. The above description is only a preferred embodiment of this invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technology.

Claims

1. An irregularly laid out invoice document layout prediction method, characterized by, include: Step 1: Collect information from irregularly formatted documents; Step 2: Label the collected irregularly formatted document information and divide the labeled data into training set and test set; each set of labeled data includes text annotation information, text selection information and image information; Step 3: Train the layout prediction model using the training set: Use document images and text embeddings as inputs to the visual module and semantic module, respectively. The visual module uses convolutional neural network backbones of different sizes to extract high- and low-dimensional visual features; the semantic module extracts multi-scale document semantic features; and through the relation feature aggregation module, combine the visual features and semantic features to learn a multimodal feature map to obtain semantic labels and coordinates. Step 4: Use the test set to test the effect of the layout prediction model, adjust the hyperparameters of the layout prediction model according to the test results, and repeat Step 3 until a layout prediction model that meets the test results is obtained. Step 5: Predict the semantic tags of irregularly formatted invoices and documents using the layout prediction model to obtain the corresponding layout; Step 3 includes: Step 3-1: The training set Chinese document image is input as a visual module, and the input image A multi-dimensional visual feature is extracted using a convolutional neural network with different scales, and different dimensional features are represented as S1, S2, S3, … S i …, where i denotes the ith dimension, H denotes the input image x height, W denotes the input image x width, denotes the channel dimension of the feature map S i and S0= x; Step 3-2: The semantic module embeds the text into a mapping. As input, document semantics are introduced, and T0 has the same spatial size as the document image x. Indicates the initial channel size; The characters on the document page are represented as The sentence is represented as Where n and m are the total number of characters and sentences, respectively; c k and It represents the k-th character and its associated bounding box, where (x0, y0) and (x1, y1) are the top-left and bottom-right pixel coordinates; s k and It represents the k-th sentence and its associated bounding box, obtained from the multi-dimensional document semantic features W1, W2, W3, ... W1 in the ConvNet convolutional neural network. i …; Step 3-3: Visual features S1, S2, S3, … and semantic features W1, W2, W3, … are connected using an attention mechanism as a convolutional layer input to learn aggregated multi-modal features MF i ; using a standard object detection or segmentation model Mask RCNN to generate candidate components in the document; using a graph neural network to model component relationships and optimize prediction results; Step 3-4: Compute two-part loss L(0) = L cls + λL gen , cross-entropy loss of semantic labels as classification loss L cls , and smooth L1 loss as coordinate regression loss L gen ; Steps 3-5: Backpropagation, using stochastic gradient descent to optimize the model parameters Θ; Steps 3-6: Read the document image and text embedding information of the next set of sequences, then jump to step 3-1 until all the data in the training set has been read; Step 3-7: Repeat steps 3-1 to 3-6 and continue to optimize the model parameter Θ until L(Θ) drops to the set order of magnitude; Step 3-3 includes: Different dimensional visual features S i and semantic features and W i As input to the attention mechanism module, the two modal features are concatenated and fed into the convolutional layer to obtain multimodal features (MF). i The calculation formulas are shown in (6) and (7): Att i = MultiAtt([S i , W i ]) (6) MF i = Att i ⊙S i +(1-Att i )⊙W i (7) Where [·,·] represents cascaded operations, MultiAtt(·) represents a multi-head attention module, and ⊙ represents element-pair multiplication; generating a set of fused multi-modal features MF by aggregating visual and semantic features i ; then using a graph neural network to model component relationships; The document is represented as a graph G = (O, E), where O = {o1, o2, ..., o}. n } represents the set of points, E represents the set of edges, each node represents a candidate component previously generated by the object detection model, and each edge represents the relationship between two candidate components. Each node is represented as o j =(l j ,f j ), including position coordinates l j and depth features f j =RoIAlign(MF,l j Construct a new node feature representation z that integrates the two types of information. j The calculation formula is shown in (8): wherein, represents the position embedding vector of the jth node; The self-attention mechanism is applied to the nodes to learn the relationship between the nodes, and the updated node features are z′ j The detection result of the jth node is optimized to obtain The calculation formulas are shown in equations (9) and (10): wherein is the probability of belonging to the c-th class, is its exact regression coordinate, Linear reg and Linear cls is the projection layer, GNN models the pairwise relationships between tokens and predicts their semantic labels.

2. The method of claim 1, wherein, Step 3-1 includes: The high and low dimensional features of the document image are extracted by the Residual block module with different size convolution kernels, and output features S1, S2, S3, … S i …, the calculation formula is as (1) (2): S0=x (1) where f i,j (·) denotes the Residual block (·) module; i = 1, 2, …, l; c, l are hyperparameters.

3. The irregular layout prediction method for invoices and documents according to claim 2, characterized in that, Step 3-2 includes: Use two mapping functions to and Construct character embedding maps for input and sentence embedding mapping The calculation formulas are shown in (3) and (4): Among them, E c (·) is a word embedding layer, E s (·) Employ the pre-trained language model Bert; Based on Char and Sent, the text embedding map is obtained, and the calculation formula is shown in (5): W0=LayerNormal(Char+Sent) (5) LayerNormal(·) is the mapping function.

4. The method of claim 1, wherein, The irregular document information in step 2 includes the document's RGB image, and the text positioning and annotation include character-level and sentence-level.

5. An irregularly formatted invoice document layout prediction device, comprising a processor and a memory; the memory stores a program or instructions, which are loaded and executed by the processor to implement the regularly formatted invoice document layout prediction method as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the rule-based invoice document layout prediction method as described in any one of claims 1 to 4.

Images