Document processing and document model training method and device, equipment and storage medium

Abstract

The present disclosure provides a document processing method and device, a document model training method and device, equipment and a storage medium, relates to the technical field of computers, and in particular to the fields of natural language processing, computer vision, deep learning and the like of artificial intelligence. The document processing method comprises: obtaining information of at least one modality of a to-be-processed document, wherein the information of each modality in the at least one modality comprises at least one processing unit, the information of the at least one modality comprises a text sequence, the processing unit comprises a text unit, and the text units under the same layout are arranged in a preset order in the text sequence; obtaining a representation vector of each processing unit in the at least one processing unit; and obtaining a processing result of the to-be-processed document based on the representation vector of each processing unit. The present disclosure can improve the document processing effect.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to artificial intelligence fields such as natural language processing, computer vision, and deep learning, and particularly to a document processing and document model training method, device, and storage medium. Background Technology

[0002] With the advent of the digital age, documents have gradually shifted from traditional paper documents to electronic documents. To understand electronic documents, document models can be used to process them.

[0003] As electronic documents contain an increasing variety of information, improving document processing efficiency has become an urgent issue. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and storage medium for document processing and training document models.

[0005] According to one aspect of this disclosure, a document processing method is provided, comprising: acquiring information of at least one modality of a document to be processed, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence; acquiring a representation vector of each processing unit in the at least one processing unit; and obtaining a processing result of the document to be processed based on the representation vectors of each processing unit.

[0006] According to another aspect of this disclosure, a method for training a document model is provided, comprising: acquiring information of at least one modality of a document sample, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the at least one processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence; acquiring a representation vector of each processing unit in the at least one processing unit; obtaining a prediction result of the document sample based on the representation vector of each processing unit; constructing a loss function based on the prediction result; and training a document model based on the loss function.

[0007] According to another aspect of this disclosure, a document processing apparatus is provided, comprising: a first acquisition module, configured to acquire information of at least one modality of a document to be processed, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence; a second acquisition module, configured to acquire a representation vector of each processing unit in the at least one processing unit; and a third acquisition module, configured to obtain a processing result of the document to be processed based on the representation vectors of each processing unit.

[0008] According to another aspect of this disclosure, a training apparatus for a document model is provided, comprising: a first acquisition module, configured to acquire information of at least one modality of a document sample, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the at least one processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence; a second acquisition module, configured to acquire representation vectors of each processing unit in the at least one processing unit; a third acquisition module, configured to obtain a prediction result of the document sample based on the representation vectors of each processing unit; a construction module, configured to construct a loss function based on the prediction result; and a training module, configured to train a document model based on the loss function.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.

[0011] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.

[0012] The technical solution disclosed herein can improve document processing efficiency.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0016] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0018] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0019] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0020] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;

[0021] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure;

[0022] Figure 8 This is a schematic diagram according to the eighth embodiment of the present disclosure;

[0023] Figure 9 This is a schematic diagram according to the ninth embodiment of the present disclosure;

[0024] Figure 10 This is a schematic diagram of an electronic device used to implement the document processing method or document model training method of the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1 Based on the schematic diagram of the first embodiment of this disclosure, this embodiment provides a method for training a document pre-training model, including:

[0027] 101. Obtain information of at least one modality of the document to be processed, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the at least one processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence.

[0028] 102. Obtain the representation vector of each processing unit in the at least one processing unit.

[0029] 103. Based on the representation vectors of each processing unit, obtain the processing result of the document to be processed.

[0030] The execution entity in this embodiment can be referred to as a document processing device. The document processing device can be software, hardware, or a combination of both, and it can be located in an electronic device. The electronic device can be located on a server or a terminal device. The server can be a local server or a cloud device, and the terminal device can include: a personal computer (PC), a portable computer, a mobile device (such as a mobile phone or tablet computer), an in-vehicle terminal (such as a car infotainment system), a wearable device (such as a smartwatch or smart bracelet), a smart home device (such as a smart TV or smart speaker), etc.

[0031] Document processing methods can be applied to various scenarios, such as information extraction and document classification. Information extraction includes extracting contract numbers, dates, and product information from electronic invoices; document classification includes categorizing electronic documents into technical documents, legal documents, and contract documents.

[0032] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] For example, regarding information extraction, see Figure 2 Users can upload documents to be processed through terminal device 201. Terminal device 201 can send the documents to be processed to server 202, and the server can extract information from the documents to be processed.

[0034] Understandably, if the terminal device has sufficient performance, the document processing process can also be performed locally on the terminal device.

[0035] The document to be processed is an electronic document, which can be in the format of Word, PDF, image, etc.

[0036] Modal refers to the form in which information is represented, such as text, image, and video.

[0037] In this embodiment, the information of at least one modality includes a text sequence, that is, it includes a text modality.

[0038] Furthermore, the information of at least one modality can also be an image, that is, it can also include an image modality.

[0039] For a text sequence, the text sequence can consist of at least one text unit.

[0040] Text units can also be called subwords, words, tokens, etc.

[0041] Taking Chinese as an example, a text unit can be specifically a character.

[0042] For example, the text sequence “The weather is nice today” consists of 6 text units (specifically, words).

[0043] In related technologies, when extracting text from a document, it is generally done in a preset order, such as from top to bottom or from left to right.

[0044] However, in some cases, because the layout information of the text in the document may be different, errors may occur if the extraction is performed in the order of top to bottom and left to right.

[0045] For example, see Figure 3 If the electronic document includes columns 301, then text 1 and text 2 are consecutive text descriptions. If the column information is ignored and the text is extracted directly from left to right, text 1, text 3, text 2, and text 4 will be extracted, which will disrupt the actual reading order.

[0046] Therefore, in this embodiment, when extracting text, the layout information is referenced, and the text under the same layout is sorted in a preset order, while the text under different layouts is arranged sequentially.

[0047] For example, if text 1 and text 2 belong to the same layout (let's call it the first layout), and text 3 and text 4 belong to the same layout (let's call it the second layout), then the text in the first layout is arranged in a preset order, and the text in the second layout is arranged in a preset order. Text from different layouts is then concatenated. For example, extracting [text 1, text 2] and [text 3, text 4], and then concatenating these two sets, which can also be called merging, results in the text sequence [text 1, text 2, text 3, text 4].

[0048] Among them, layout refers to the format that affects the reading order, such as columns, tables, paragraphs, etc.

[0049] After obtaining information about at least one modality of the document to be processed, a representation vector of the processing unit can be obtained for the processing unit of information of each modality in the at least one modality.

[0050] After obtaining the representation vector of the processing unit, the processing result of the document to be processed can be obtained based on the representation vector of the processing unit.

[0051] Taking information extraction as an example, the processing result can be information extracted from the document to be processed; or, taking document classification as an example, the processing result can be the document category.

[0052] In this embodiment of the disclosure, by arranging the text units under the same layout in the text sequence in a preset order, the order of the text units can conform to the actual reading order, improve the semantic coherence of the text sequence, and thus improve the document processing effect.

[0053] The above explanation pertains to text sequences. In reality, documents may contain not only text but also information in other modalities.

[0054] In practice, a document can be an image. For example, a paper document can be scanned or photographed to obtain an image of the paper document. In this case, the document can be an image document.

[0055] For an image document, at least one modality of information may include, in addition to the text (e.g., the text sequence mentioned above), the image of the document itself.

[0056] Accordingly, if at least one modality of information is text, the text processing unit can be called a text unit, such as Chinese characters; if at least one modality of information is an image, the image processing unit can be called an image unit (or image block).

[0057] Whether it is a text unit or an image unit, each can obtain its corresponding representation vector.

[0058] To obtain the representation vector of a text unit, the text unit can be extracted first, and then an embedding layer can be used to obtain the representation vector.

[0059] For the representation vector of an image unit, a visual encoder can be used to obtain the representation vector. That is, it is not necessary to divide the image into multiple image units first. The visual encoder can encode the whole image and obtain the encoding vector of each image unit in the image. Then, the representation vector of the image unit can be obtained based on the encoding vector of the image unit.

[0060] Among these methods, optical character recognition (OCR) can be used to extract text units, a layout parser can be used to obtain layout information, and the text units can be arranged based on the layout information.

[0061] That is, in some embodiments, for the text sequence included in the information of the at least one modality, obtaining the information of the at least one modality of the document to be processed includes: performing OCR on the document to be processed to obtain the text units within the document to be processed; performing layout parsing on the text units to obtain layout information of the text units; sequentially concatenating the text units under different layouts based on the layout information; and arranging the text units under the same layout in a preset order.

[0062] For example, see Figure 4 Taking an image document as an example, the image document can be input into the OCR module to perform OCR on the image document. The OCR output can include: text units in the image document, and the two-dimensional (2D) position information of the text units.

[0063] The text cells output by OCR can be input into the layout parser. The layout parser can obtain the layout information of the text cells and, based on the layout information, arrange the text cells under the same layout in a preset order, and arrange the text cells of different layouts in sequence.

[0064] For example, as described above, the text unit under the first layout is [text1, text2], the text unit under the second layout is [text3, text4], and the text sequence is [text1, text2, text3, text4].

[0065] In addition, the layout parser can also output one-dimensional (1D) position information of text cells, segment information of the segment in which the text cell is located, and layout information of the layout in which the text cell is located.

[0066] Taking text units as an example, 2D position information can be the information of the detection box of the text. For example, the detection box is generally a rectangle, which can be represented by four elements: the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2).

[0067] 1D location information can be a serial number of text, such as 0, 1, 2, etc.

[0068] Segments can be specified according to actual needs. For example, each line can be a segment, each sentence can be a segment, etc. Segment information can be represented by segment numbers, such as 0, 1, 2, etc.

[0069] Layout refers to the format that affects the reading order, such as columns, tables, etc. Layout information can also be represented by layout numbers, such as 0, 1, 2, etc.

[0070] The layout parser can then input the obtained text units and their formatting information into the text embedding layer. The formatting information may include one or more of the following: 1D position information, 2D position information, fragment information, and layout information.

[0071] By performing OCR and layout parsing on the document to be processed, text units arranged in sequence under the same layout can be obtained, improving the semantic coherence of the text sequence.

[0072] Additionally, see Figure 4 The image document can also be input into a visual encoder, which encodes the image and outputs the encoded vector of the image unit, as well as the format information of the image unit. Similar to the format information of text units, the format information of image units can include one or more of the following: 1D position information, 2D position information, fragment information, and layout information of the image unit.

[0073] Then, the visual encoder can input the encoding vector of the image unit and the format of the image unit into the image embedding layer.

[0074] In some embodiments, obtaining the representation vector of each processing unit in the at least one processing unit includes: obtaining the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; and obtaining the representation vector of each processing unit based on the semantic representation vector and the format representation vector.

[0075] The position representation vector may include a 1D position representation vector and / or a 2D position representation vector.

[0076] The format representation vectors include: position representation vectors, fragment representation vectors, and layout representation vectors. For example, position representation vectors include 1D position representation vectors and 2D position representation vectors. (See [link to documentation]). Figure 5 The embedding layer (text embedding layer and image embedding layer) can include semantic embedding layer, 1D position embedding layer, 2D position embedding layer, fragment embedding layer and layout embedding layer.

[0077] The processing procedures for text units and image units are similar. The following explanation uses text units as an example.

[0078] For text units, a semantic embedding layer can be used to convert text units into semantic representation vectors, a 1D position embedding layer can be used to convert the 1D position information of text units into 1D position representation vectors, a 2D position embedding layer can be used to convert the 2D position information of text units into 2D position representation vectors, a fragment embedding layer can be used to convert the fragment information of text units into fragment representation vectors, and a layout embedding layer can be used to convert the layout information of text units into fragment representation vectors.

[0079] Each embedding layer (semantic embedding layer, 1D position embedding layer, 2D position embedding layer, fragment embedding layer, layout embedding layer) can be implemented using a deep neural network, and the specific structure can be set as needed.

[0080] Specifically, for text units, the semantic embedding layer can convert text units in text form into semantic representation vectors of text units in vector form.

[0081] For image units, since the visual encoder obtains the encoding vector, which is already in vector form, but generally speaking, the encoding vector has a different dimension than the semantic representation vector of the text unit, a transformation network (such as a fully connected layer) can be used to convert the encoding vector into a vector with the same dimension as the semantic representation vector of the text unit. This transformed vector can be called the semantic representation vector of the image unit.

[0082] Since the semantic embedding layers of text units and image units do not have completely identical functions, the semantic embedding layers of text units and image units can each have their own model parameters.

[0083] For other format-related embedding layers, such as 1D position embedding layers, 2D position embedding layers, fragment embedding layers, and layout embedding layers, text units and image units can share model parameters.

[0084] After obtaining the representation vectors of each embedding layer, as follows: Figure 5 As shown, the various representation vectors can be added together to obtain the representation vector of the text unit and the representation vector of the image unit.

[0085] For example, taking a text unit as an example, the semantic representation vector of the text unit + the 1D position representation vector of the text unit + the 2D position representation vector of the text unit + the fragment representation vector of the text unit + the layout representation vector of the text unit are added together to obtain the representation vector of the text unit.

[0086] By introducing fragment embedding layers and layout embedding layers into the embedding layer, more granular formatting information can be incorporated into document processing. This allows for the learning of document formatting content in a more flexible manner, thereby improving the effectiveness of document processing.

[0087] In some embodiments, the information of the at least one modality further includes: an image corresponding to the document to be processed; obtaining the semantic representation vector of each processing unit includes: if the information of the at least one modality is a text sequence, performing semantic embedding processing on each text unit in the text sequence to obtain the semantic representation vector of each text unit; and / or, if the information of the at least one modality is the image, performing visual encoding on the image to obtain the semantic representation vector of each image unit in the image.

[0088] Among them, such as Figure 4 As shown, different methods can be used to obtain semantic representation vectors for information of different modalities (text and images). For example, for text, text units can be obtained first, and then the semantic representation vector of each text unit can be obtained. For images, it is not necessary to divide them into image units first. A visual encoder can be used to process the whole image to obtain the semantic representation vector of each image unit.

[0089] By using different methods to obtain semantic representation vectors corresponding to different modalities, more effective semantic representation vectors can be obtained.

[0090] In some embodiments, obtaining the processing result of the document to be processed based on the representation vectors of each processing unit includes: processing the representation vectors of each processing unit based on a spatially aware self-attention network to obtain a hidden layer encoding vector; and decoding the hidden layer encoding vector to obtain the processing result of the document to be processed.

[0091] Among them, see Figure 4 Based on text embedding layers and image embedding layers, representation vectors of text units and image units can be obtained. Then, the representation vectors of text units and image units can be concatenated, and the concatenated vector is used as the input of the pre-trained model. Vector concatenation is similar to text concatenation, that is, merging them together. For example, if one vector is [0,1] and another vector is [1,1], then the concatenated vector is [0,1,1,1].

[0092] Figure 4 The pre-trained model in the code is represented by a Transformer layer. It's understandable that a typical Transformer includes an encoder and a decoder; a Transformer layer specifically refers to the encoder part of the Transformer.

[0093] The output vector of the Transformer layer (which can be called the hidden layer encoding vector) can be input into the decoding layer, and the output of the decoding layer is the processing result of the text to be processed.

[0094] The decoding layer can be structured according to the specific task, and the corresponding model parameters can also be determined during the training phase.

[0095] Accordingly, the processing results can also vary depending on the task. For example, if the task is information extraction, the processing result can be the information extracted from the image document; or if the task is document classification, the processing result can be the classification result of the image document.

[0096] In this embodiment, the self-attention network of the Transformer layer can be a spatial-aware self-attention mechanism / network.

[0097] In particular, spatially aware self-attention networks explicitly introduce spatial relationships between processing units (tokens).

[0098] For example, the attention score of the traditional self-attention mechanism is calculated using α. ij If we express this as an example, then the attention score based on the spatial perception-based self-attention mechanism can be represented as α′. ij , b (1D) , These are the learnable relative positional deviation values, (x i ,y i α is the top-left corner coordinate of the 2D bounding box for the i-th token. ij This represents the attention score between the i-th token and the j-th token. In determining the 2D relative position deviation value, x and y are both based on the top-left corner. Alternatively, x can be the x-coordinate of the top-left corner and y can be the y-coordinate of the top-right corner.

[0099] By employing spatially aware self-attention networks, we can obtain hidden layer encoding vectors that contain more accurate spatial location information, thereby improving document processing performance.

[0100] The above describes the model application process, namely, processing documents based on the model.

[0101] The training process of the model will be explained below.

[0102] Figure 6 Based on the schematic diagram of the sixth embodiment of this disclosure, this embodiment provides a method for training a document model. The method of this embodiment includes:

[0103] 601. Obtain information of at least one modality of a document sample, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the at least one processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence.

[0104] 602. Obtain the representation vector of each processing unit in the at least one processing unit.

[0105] 603. Based on the representation vectors of each processing unit, obtain the prediction results of the document sample.

[0106] 604. Based on the prediction results, construct a loss function.

[0107] 605. Based on the loss function, train the document model.

[0108] The model results can be consistent in the training and application phases. In the training phase, the documents processed can be called document samples, and the processing results corresponding to the document samples can be called prediction results. The processing process from document samples to prediction results is consistent with the principle in the application phase of the model, and will not be described in detail here.

[0109] After obtaining the prediction results, a loss function can be constructed using the prediction results and the label values ​​of the document samples.

[0110] Document samples can be obtained from existing datasets or through collection, and the label values ​​of document samples can be manually labeled or obtained from existing datasets.

[0111] In some embodiments, obtaining a prediction vector based on the representation vectors of each processing unit includes: performing multiple tasks based on the representation vectors of each processing unit to obtain prediction results corresponding to each of the multiple tasks, wherein the multiple tasks include: text tasks, image-text tasks, and layout tasks. The image-text task includes a fine-grained image-text matching task, and the processing unit includes an image unit. Any image unit in the image unit is randomly replaced. For the fine-grained image-text matching task, performing multiple tasks based on the representation vectors of each processing unit to obtain prediction results corresponding to each of the multiple tasks includes: obtaining a prediction result corresponding to the fine-grained image-text matching task based on the representation vector of the image unit, wherein the prediction result corresponding to the fine-grained image-text matching task is used to predict the replaced image unit.

[0112] In some embodiments, the image-text task includes a fine-grained image-text matching task, the processing unit includes image units, any image unit in the image units is randomly replaced, and for the fine-grained image-text matching task, the step of performing multiple tasks based on the representation vectors of each processing unit to obtain prediction results corresponding to each of the multiple tasks includes: obtaining prediction results corresponding to the fine-grained image-text matching task based on the representation vectors of the image units, and the prediction results corresponding to the fine-grained image-text matching task are used to predict the replaced image units.

[0113] In some embodiments, obtaining the representation vector of each processing unit in the at least one processing unit includes: obtaining the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; and obtaining the representation vector of each processing unit based on the semantic representation vector and the format representation vector.

[0114] In some embodiments, obtaining the prediction result of the document sample based on the representation vectors of each processing unit includes: processing the representation vectors of each processing unit based on a spatially aware self-attention network to obtain a hidden layer encoding vector; and decoding the hidden layer encoding vector to obtain the prediction result of the document sample.

[0115] In some embodiments, for the information of the at least one modality including the text sequence, obtaining the information of the at least one modality of the document sample includes: performing OCR on the document sample to obtain the text units within the document sample; performing layout parsing on the text units to obtain layout information of the text units; sequentially concatenating the text units under different layouts based on the layout information; and arranging the text units under the same layout in a preset order.

[0116] In some embodiments, the information of the at least one modality further includes: an image corresponding to the document sample; the step of obtaining the semantic representation vector of each processing unit includes: if the information of the at least one modality is a text sequence, performing semantic embedding processing on each text unit in the text sequence to obtain the semantic representation vector of each text unit; and / or, if the information of the at least one modality is the image, performing visual encoding on the image to obtain the semantic representation vector of each image unit in the image.

[0117] For details regarding the process of predicting document samples from document samples, please refer to the above embodiments.

[0118] For details on the embedding layer, please refer to [link / reference].Figure 7 Where Ti (i = 1, 2...) are text units, and Vi (i = 1, 2...) are image units. 511 represents a text sequence of length 512, 48 represents dividing the image document into 49 image units, and 128 represents the maximum value of the segment encoding and layout number. It is understood that these specific values ​​are examples, and other values ​​can also be used. Furthermore, Figure 7 The addition in this context refers to vector addition. Taking the semantic embedding layer as an example, the shown Ti needs to be converted into the corresponding semantic representation vector. Taking the 1D position embedding layer as an example, the shown 1D position numbers (0,1...) need to be converted into the corresponding 1D position representation vector. The other embedding layers are processed similarly.

[0119] During the training phase, tasks need to be set, such as... Figure 7 As shown, the tasks in this embodiment may include: Masked Visual-Language Model task, Fine-grained Text-Image Matching task, Text-Image Alignment task, and WordPositionPrediction task.

[0120] The contents of each of the above tasks are as follows:

[0121] Masked Visual-Language Model: This task belongs to the text task category. Based on the information from the document input, characters on the text side are masked to minimize the difference between the predicted results and the actual values ​​of the masked characters in the document. In the document image scenario, to avoid label leakage on the image side, the words that need to be masked are blacked out in the document image.

[0122] Fine-grained Text-Image Matching: This task belongs to the category of text-image interaction tasks. Given a small patch of an image, the task is to predict which patch will be replaced.

[0123] Text-Image Alignment: This task also falls under the category of text-image interaction tasks. It involves randomly blacking out text in an image and predicting which words will be blacked out.

[0124] Word Position Prediction: This task belongs to the layout task category. It predicts which image block a word belongs to within a document.

[0125] When constructing the loss function, you can construct a loss function for each task, for example, construct loss functions for 4 tasks, then add the loss functions of these 4 tasks together to get the total loss function, and then adjust the model parameters by minimizing the total loss function.

[0126] In this context, the loss function used in training the document model is the overall loss function mentioned above. The process of training the model based on this loss function can be implemented using relevant techniques. Specifically, the model parameters can be adjusted through backpropagation until a preset termination condition is met. The model parameters that meet the termination condition are then used as the final model parameters, thus obtaining the trained model. The termination condition can be a preset number of iterations, a preset time, or the convergence of the loss function, etc.

[0127] The document model obtained in this embodiment can be called a document pre-trained model. For a specific task, the document pre-trained model can be fine-tuned to generate a model corresponding to the specific task. Then, the model corresponding to the specific task can be used to process the document to be processed.

[0128] In this embodiment, a layout parsing module is used to ensure that the concatenated text conforms to the actual reading order, improving the semantic coherence of the text. The purpose of text-image interaction in document scenarios is to enable the text to learn the layout features in the image and the correspondence between the text and the image. Traditional text-image matching only needs to roughly determine whether the text has appeared in the document, which is too simplistic. Fine-grained text-image matching is better able to learn fine-grained text-image matching relationships than traditional text-image matching algorithms, and it also greatly helps the model's layout learning ability. For different levels of format information, segment information (segment ID) and layout information (layout ID) are added to help the model better learn format features. It also includes pre-training tasks related to text, text-image, and layout, comprehensively improving the model's analytical capabilities in these three aspects.

[0129] Figure 8 This is a schematic diagram based on the eighth embodiment of the present disclosure, which provides a document processing apparatus. For example... Figure 8 As shown, the document processing device 800 includes: a first acquisition module 801, a second acquisition module 802, and a third acquisition module 803.

[0130] The first acquisition module 801 is used to acquire information of at least one modality of the document to be processed. The information of each modality includes at least one processing unit. The information of the at least one modality includes a text sequence. The processing unit includes text units. The text units under the same layout are arranged in a preset order within the text sequence. The second acquisition module 802 is used to acquire the representation vector of each processing unit in the at least one processing unit. The third acquisition module 803 is used to obtain the processing result of the document to be processed based on the representation vector of each processing unit.

[0131] In some embodiments, the second acquisition module 802 is further configured to: acquire the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; and obtain the representation vector of each processing unit based on the semantic representation vector and the format representation vector.

[0132] In some embodiments, the third acquisition module 803 is further configured to: process the representation vectors of each processing unit based on a spatially aware self-attention network to obtain a hidden layer encoding vector; and decode the hidden layer encoding vector to obtain the processing result of the document to be processed.

[0133] In some embodiments, for the text sequence included in the information of the at least one modality, the first acquisition module 801 is further configured to: perform OCR on the document to be processed to obtain the text units within the document to be processed; perform layout parsing on the text units to obtain layout information of the text units; based on the layout information, sequentially concatenate the text units under different layouts; and arrange the text units under the same layout in a preset order.

[0134] In some embodiments, the information of the at least one modality further includes: an image corresponding to the document to be processed, and the second acquisition module 802 is further configured to: if the information of the at least one modality is the text sequence, perform semantic embedding processing on each text unit in the text sequence to obtain the semantic representation vector of each text unit; and / or, if the information of the at least one modality is the image, perform visual encoding on the image to obtain the semantic representation vector of each image unit in the image.

[0135] In this embodiment of the disclosure, by arranging the text units under the same layout in the text sequence in a preset order, the order of the text units can conform to the actual reading order, improve the semantic coherence of the text sequence, and thus improve the document processing effect.

[0136] Figure 9 This is a schematic diagram based on the ninth embodiment of the present disclosure, which provides a training apparatus for a document model. For example... Figure 9 As shown, the document model training device 900 includes: a first acquisition module 901, a second acquisition module 902, a third acquisition module 903, a construction module 904, and a training module 905.

[0137] The first acquisition module 901 is used to acquire information of at least one modality of a document sample, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence, the at least one processing unit includes text units, and the text units under the same layout are arranged in a preset order within the text sequence; the second acquisition module 902 is used to acquire the representation vector of each processing unit in the at least one processing unit; the third acquisition module 903 is used to obtain the prediction result of the document sample based on the representation vector of each processing unit; the construction module 904 is used to construct a loss function based on the prediction result; and the training module 905 is used to train a document model based on the loss function.

[0138] In some embodiments, the third acquisition module 903 is further configured to: perform multiple tasks based on the representation vectors of each processing unit to obtain prediction results corresponding to each of the multiple tasks, wherein the multiple tasks include: text tasks, image and text tasks, and layout tasks.

[0139] In some embodiments, the image-text task includes a fine-grained image-text matching task, the processing unit includes image units, and any image unit in the image units is randomly replaced. For the fine-grained image-text matching task, the third acquisition module 903 is further configured to: obtain the prediction result corresponding to the fine-grained image-text matching task based on the representation vector of the image unit, and the prediction result corresponding to the fine-grained image-text matching task is used to predict the replaced image unit.

[0140] In some embodiments, the second acquisition module 902 is further configured to: acquire the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; and obtain the representation vector of each processing unit based on the semantic representation vector and the format representation vector.

[0141] In some embodiments, the third acquisition module 903 is further configured to: process the representation vectors of each processing unit based on a spatially aware self-attention network to obtain a hidden layer encoding vector; and decode the hidden layer encoding vector to obtain the prediction result of the document sample.

[0142] In some embodiments, for the text sequence included in the information of the at least one modality, the first acquisition module 901 is further configured to: perform OCR on the document sample to obtain the text units within the document sample; perform layout parsing on the text units to obtain layout information of the text units; based on the layout information, sequentially concatenate the text units under different layouts; and arrange the text units under the same layout in a preset order.

[0143] In some embodiments, the information of the at least one modality further includes: an image corresponding to the document sample, and the third acquisition module 903 is further configured to: if the information of the at least one modality is the text sequence, perform semantic embedding processing on each text unit in the text sequence to obtain the semantic representation vector of each text unit; and / or, if the information of the at least one modality is the image, perform visual encoding on the image to obtain the semantic representation vector of each image unit in the image.

[0144] In this embodiment of the disclosure, by arranging the text units under the same layout in the text sequence in a preset order, the order of the text units can conform to the actual reading order, improve the semantic coherence of the text sequence, and thus improve the processing effect of the document model.

[0145] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0146] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.

[0147] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.

[0148] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0151] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as document processing methods or document model training methods. For example, in some embodiments, the document processing methods or document model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the document processing methods or document model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform a document processing method or a document model training method.

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0159] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A document processing method, comprising: Information of at least one modality of a document to be processed is obtained. The information of each modality includes at least one processing unit. The information of the at least one modality includes a text sequence and an image. The processing unit includes a text unit and an image unit. The text units under the same layout are arranged in a preset order within the text sequence, and the text units under different layouts are arranged sequentially. Obtain the representation vector of each processing unit in the at least one processing unit; the representation vector is obtained based on the semantic representation vector and the format representation vector, the format representation vector including the layout representation vector; Based on the representation vectors of each processing unit, the processing result of the document to be processed is obtained, including: concatenating the representation vectors of the text units and the representation vectors of the image units, inputting the concatenated vector into the pre-trained model, inputting the output vector of the pre-trained model into the decoding layer, and the output of the decoding layer being the processing result; The pre-trained model employs a spatially aware self-attention network, which explicitly incorporates the spatial positional relationships between processing units. The semantic representation vector is obtained in the following way: If the information of the at least one modality is the text sequence, semantic embedding processing is performed on each text unit in the text sequence to obtain the semantic representation vector of each text unit; if the information of the at least one modality is the image, visual encoding is performed on the image to obtain the semantic representation vector of each image unit in the image.

2. The method according to claim 1, wherein, The step of obtaining the representation vector of each processing unit in the at least one processing unit includes: Obtain the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; Based on the semantic representation vector and the format representation vector, the representation vector of each processing unit is obtained.

3. The method according to claim 1 or 2, wherein, The process of obtaining the processing result of the document to be processed based on the representation vectors of each processing unit includes: A spatially aware self-attention network processes the representation vectors of each processing unit to obtain the hidden layer encoding vector. The hidden layer encoding vector is decoded to obtain the processing result of the document to be processed.

4. The method according to claim 1 or 2, wherein, The information for at least one modality includes a text sequence, and obtaining information for at least one modality of the document to be processed includes: Perform OCR on the document to be processed to obtain the text units within the document to be processed; The layout of the text unit is parsed to obtain the layout information of the text unit; Based on the layout information, the text units under different layouts are sequentially spliced ​​together, and the text units under the same layout are arranged in a preset order.

5. A method for training a document model, comprising: Information of at least one modality of a document sample is obtained, wherein the information of each modality includes at least one processing unit, the information of the at least one modality includes a text sequence and an image, the at least one processing unit includes a text unit and an image unit, and the text units under the same layout are arranged in a preset order within the text sequence, and the text units under different layouts are arranged sequentially. Obtain the representation vector of each processing unit in the at least one processing unit; the representation vector is obtained based on the semantic representation vector and the format representation vector, the format representation vector including the layout representation vector; Based on the representation vectors of each processing unit, the prediction result of the document sample is obtained, including: concatenating the representation vectors of the text unit and the image unit, inputting the concatenated vector into the pre-trained model, inputting the output vector of the pre-trained model into the decoding layer, and the output of the decoding layer being the prediction result; the pre-trained model adopts a spatially aware self-attention network, which explicitly introduces the spatial positional relationship between processing units; Based on the prediction results, a loss function is constructed; Based on the loss function, train the document model; The semantic representation vector is obtained in the following way: If the information of the at least one modality is the text sequence, semantic embedding processing is performed on each text unit in the text sequence to obtain the semantic representation vector of each text unit; if the information of the at least one modality is the image, visual encoding is performed on the image to obtain the semantic representation vector of each image unit in the image.

6. The method according to claim 5, wherein, The process of obtaining the prediction vector based on the representation vectors of each processing unit includes: Based on the representation vectors of each processing unit, multiple tasks are executed to obtain the prediction results corresponding to each of the multiple tasks, including: text tasks, image and text tasks, and layout tasks.

7. The method according to claim 6, wherein, The image-text task includes a fine-grained image-text matching task. The processing unit includes an image unit, where any image unit is randomly replaced. For the fine-grained image-text matching task, multiple tasks are executed based on the representation vectors of each processing unit to obtain prediction results for each of the multiple tasks, including: Based on the representation vector of the image unit, the prediction result corresponding to the fine-grained image-text matching task is obtained, and the prediction result corresponding to the fine-grained image-text matching task is used to predict the image unit to be replaced.

8. The method according to any one of claims 5-7, wherein, The step of obtaining the representation vector of each processing unit in the at least one processing unit includes: Obtain the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; Based on the semantic representation vector and the format representation vector, the representation vector of each processing unit is obtained.

9. The method according to any one of claims 5-7, wherein, The step of obtaining the prediction result of the document sample based on the representation vectors of each processing unit includes: A spatially aware self-attention network processes the representation vectors of each processing unit to obtain the hidden layer encoding vector. The hidden layer coding vector is decoded to obtain the prediction result of the document sample.

10. The method according to any one of claims 5-7, wherein, The acquisition of information about at least one modality of a document sample, including the text sequence of the information for the at least one modality, includes: Perform OCR on the document sample to obtain the text units within the document sample; The layout of the text unit is parsed to obtain the layout information of the text unit; Based on the layout information, the text units under different layouts are sequentially spliced ​​together, and the text units under the same layout are arranged in a preset order.

11. A document processing apparatus, comprising: The first acquisition module is used to acquire information of at least one modality of the document to be processed. The information of each modality includes at least one processing unit. The information of the at least one modality includes a text sequence and an image. The processing unit includes a text unit and an image unit. The text units under the same layout are arranged in a preset order within the text sequence, and the text units under different layouts are arranged sequentially. The second acquisition module is used to acquire the representation vector of each processing unit in the at least one processing unit; the representation vector is obtained based on the semantic representation vector and the format representation vector, and the format representation vector includes the layout representation vector; The third acquisition module is used to obtain the processing result of the document to be processed based on the representation vectors of each processing unit, including: concatenating the representation vectors of the text unit and the representation vectors of the image unit, inputting the concatenated vector into the pre-trained model, inputting the output vector of the pre-trained model into the decoding layer, and the output of the decoding layer being the processing result; the pre-trained model adopts a spatially aware self-attention network, which explicitly introduces the spatial positional relationship between processing units; The semantic representation vector is obtained in the following way: If the information of the at least one modality is the text sequence, semantic embedding processing is performed on each text unit in the text sequence to obtain the semantic representation vector of each text unit; if the information of the at least one modality is the image, visual encoding is performed on the image to obtain the semantic representation vector of each image unit in the image.

12. The apparatus according to claim 11, wherein, The second acquisition module is further used for: Obtain the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; Based on the semantic representation vector and the format representation vector, the representation vector of each processing unit is obtained.

13. The apparatus according to claim 11 or 12, wherein, The third acquisition module is further used for: A spatially aware self-attention network processes the representation vectors of each processing unit to obtain the hidden layer encoding vector. The hidden layer encoding vector is decoded to obtain the processing result of the document to be processed.

14. The apparatus according to claim 11 or 12, wherein, Regarding the text sequence included in the information of the at least one modality, the first acquisition module is further configured to: Perform OCR on the document to be processed to obtain the text units within the document to be processed; The layout of the text unit is parsed to obtain the layout information of the text unit; Based on the layout information, the text units under different layouts are sequentially spliced ​​together, and the text units under the same layout are arranged in a preset order.

15. A training device for a document model, comprising: The first acquisition module is used to acquire information of at least one modality of a document sample. The information of each modality includes at least one processing unit. The information of the at least one modality includes a text sequence and an image. The at least one processing unit includes a text unit and an image unit. The text units under the same layout are arranged in a preset order within the text sequence, and the text units under different layouts are arranged sequentially. The second acquisition module is used to acquire the representation vector of each processing unit in the at least one processing unit; the representation vector is obtained based on the semantic representation vector and the format representation vector, and the format representation vector includes the layout representation vector; The third acquisition module is used to obtain the prediction result of the document sample based on the representation vectors of each processing unit, including: concatenating the representation vectors of the text unit and the image unit, inputting the concatenated vector into the pre-trained model, inputting the output vector of the pre-trained model into the decoding layer, and the output of the decoding layer being the prediction result; the pre-trained model adopts a spatially aware self-attention network, which explicitly introduces the spatial positional relationship between processing units; The construction module is used to construct a loss function based on the prediction results; The training module is used to train the document model based on the loss function; The semantic representation vector is obtained in the following way: If the information of the at least one modality is the text sequence, semantic embedding processing is performed on each text unit in the text sequence to obtain the semantic representation vector of each text unit; if the information of the at least one modality is the image, visual encoding is performed on the image to obtain the semantic representation vector of each image unit in the image.

16. The apparatus according to claim 15, wherein, The third acquisition module is further used for: Based on the representation vectors of each processing unit, multiple tasks are executed to obtain the prediction results corresponding to each of the multiple tasks, including: text tasks, image and text tasks, and layout tasks.

17. The apparatus according to claim 16, wherein, The image-text task includes a fine-grained image-text matching task. The processing unit includes an image unit, where any image unit is randomly replaced. For the fine-grained image-text matching task, the third acquisition module is further configured to: Based on the representation vector of the image unit, the prediction result corresponding to the fine-grained image-text matching task is obtained, and the prediction result corresponding to the fine-grained image-text matching task is used to predict the image unit to be replaced.

18. The apparatus according to any one of claims 15-17, wherein, The second acquisition module is further used for: Obtain the semantic representation vector of each processing unit and the format representation vector of each processing unit, wherein the format representation vector includes at least one of the following: position representation vector, fragment representation vector, and layout representation vector; Based on the semantic representation vector and the format representation vector, the representation vector of each processing unit is obtained.

19. The apparatus according to any one of claims 15-17, wherein, The third acquisition module is further used for: A spatially aware self-attention network processes the representation vectors of each processing unit to obtain the hidden layer encoding vector. The hidden layer coding vector is decoded to obtain the prediction result of the document sample.

20. The apparatus according to any one of claims 15-17, wherein, Regarding the text sequence included in the information of the at least one modality, the first acquisition module is further configured to: Perform OCR on the document sample to obtain the text units within the document sample; The layout of the text unit is parsed to obtain the layout information of the text unit; Based on the layout information, the text units under different layouts are sequentially spliced ​​together, and the text units under the same layout are arranged in a preset order.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images