Document layout detection model training method and device, and electronic equipment

Abstract

The present disclosure provides a document layout detection model training method and device and electronic equipment, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and large model. The specific implementation scheme is: obtaining first training data, an initial first document layout detection model, and a second document layout detection model; the parameters of the second document layout detection model are determined according to second training data including true labels of a plurality of sample document images; inputting the sample document images in the first training data into the second document layout detection model to obtain layout prediction results; determining pseudo labels according to the layout prediction results and weak labels of the sample document images in the first training data; and then training and processing the first document layout detection model in combination with the pseudo labels.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to scenarios such as image processing. In particular, it relates to a training method, apparatus, and electronic device for a document layout detection model. Background Technology

[0002] Current document layout detection methods involve inputting the document to be detected into a document layout detection model to obtain the results. When the document layout detection model is a large language model, the training data volume is large, the annotation cost is high, and the annotation time is long, resulting in high cost of obtaining training data and poor model training efficiency. Summary of the Invention

[0003] This disclosure provides a training method, apparatus, and electronic device for a document layout detection model.

[0004] According to one aspect of this disclosure, a method for training a document layout detection model is provided. The method includes: acquiring first training data, an initial first document layout detection model, and a second document layout detection model; the first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on second training data including true labels of multiple sample document images; for the sample document images in the first training data, inputting the sample document images into the second document layout detection model to obtain the layout prediction result output by the second document layout detection model; determining the pseudo label of the sample document images based on the layout prediction result and the weak labels; and training the first document layout detection model based on the sample document images in the first training data and the pseudo labels of the sample document images to obtain a trained document layout detection model.

[0005] According to another aspect of this disclosure, a training apparatus for a document layout detection model is provided. The apparatus includes: a first acquisition module, configured to acquire first training data, an initial first document layout detection model, and a second document layout detection model; the first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on second training data including true labels of multiple sample document images; a second acquisition module, configured to input the sample document images in the first training data into the second document layout detection model and acquire the layout prediction result output by the second document layout detection model; a determination module, configured to determine the pseudo labels of the sample document images based on the layout prediction result and the weak labels; and a training module, configured to train the first document layout detection model based on the sample document images in the first training data and the pseudo labels of the sample document images to obtain a trained document layout detection model.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the training method for the document layout detection model proposed above in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the training method for the document layout detection model proposed in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the training method for the document layout detection model proposed above in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a block diagram of an electronic device used to implement the training method of the document layout detection model in the embodiments of this disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Current document layout detection methods involve inputting the document to be detected into a document layout detection model to obtain the results. When the document layout detection model is a large language model, it requires training or fine-tuning with a large amount of documents and corresponding layout annotation data. This results in a large amount of training data, high annotation costs, and long annotation time, leading to high costs in obtaining training data and poor model training efficiency.

[0018] To address the aforementioned issues, this disclosure proposes a training method, apparatus, and electronic device for a document layout detection model.

[0019] Figure 1 The diagram is based on the first embodiment of this disclosure. It should be noted that the training method for the document layout detection model in this embodiment can be applied to a training device for the document layout detection model. This device can be configured in an electronic device so that the electronic device can perform the training function of the document layout detection model. The following embodiments use an electronic device as an example for illustration.

[0020] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, and other hardware devices with various operating systems, touch screens, and / or displays.

[0021] like Figure 1 As shown, the training method for this document layout detection model may include the following steps:

[0022] Step 101: Obtain first training data, an initial first document layout detection model, and a second document layout detection model; the first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on the second training data, which includes true labels of multiple sample document images.

[0023] In this embodiment of the disclosure, the initial first document layout detection model can be a model that has not been trained using the second training data, or it can be a model trained using the second training data.

[0024] In this embodiment of the disclosure, the initial first document layout detection model can be a model that has not been trained using the second training data. Correspondingly, the electronic device performing step 101 may, for example, acquire the first training data, the second training data, the initial first document layout detection model, and the initial second document layout detection model; and train the initial second document layout detection model based on the sample document images in the second training data and the true labels of the sample document images to obtain a trained second document layout detection model.

[0025] The true labels for the sample document images can be bounding boxes obtained through annotation or other methods, along with their corresponding layout categories. The bounding boxes can be represented by the coordinates of their center point and the coordinates of their four corners. Layout categories, such as text paragraphs, tables, table titles, paragraph titles, figures, figure titles, headers, footers, and footnotes, can be set according to actual needs.

[0026] The weak labels for sample document images can be information related to bounding boxes and / or layout categories within the image, but they cannot completely determine the bounding boxes and layout categories. Weak labels can be any of the following: no label, number of layout categories, number of bounding boxes under each layout category, center point of a sample bounding box, center point of a sample bounding box combined with layout category and sample bounding box. This variety of weak labels allows annotators to choose the appropriate annotation method based on actual needs, further reducing annotation costs and model training costs.

[0027] In this process, the electronic device first uses second training data containing true labels to train the initial second document layout detection model. Then, it can combine the trained second document layout detection model with the first training data containing weak labels to determine the pseudo labels of the sample document images in the first training data. This avoids the need to fully annotate all sample document images required for training the first document layout detection model, shortens the annotation time, and reduces the annotation cost.

[0028] In this embodiment of the disclosure, the initial first document layout detection model can be a model trained using the second training data. Correspondingly, the electronic device performing step 101 may, for example, acquire the first training data, the initial first document layout detection model, and the initial second document layout detection model; and perform parameter initialization processing on the initial second document layout detection model according to the parameters of the initial first document layout detection model to obtain the second document layout detection model.

[0029] The electronic device initializes the parameters of the initial second document format detection model based on the parameters of the first document format detection model trained with the second training data. This improves the detection accuracy of the second document format detection model. The initialized second document format detection model is then used to determine pseudo-labels, avoiding full annotation of all sample document images, shortening annotation time, and reducing annotation costs.

[0030] In the case where the initial first document layout detection model is a model that has not been trained using the second training data, as an alternative to the first example, the initial first document layout detection model can be trained using the second training data. Then, based on the parameters of the trained first document layout detection model, the parameters of the initial second document layout detection model can be initialized to obtain the second document layout detection model.

[0031] In this embodiment, to facilitate the extraction of global and local features from sample document images by the first and second document layout detection models, and to consider more features to improve the training accuracy of the first document layout detection model, both the first and second document layout detection models can include a convolutional neural network and a Transformer network connected in sequence. The convolutional neural network (CNN) is used to extract local features; the encoding layer in the Transformer network is used to extract global features.

[0032] Taking the first document layout detection model as an example, the first document layout detection model specifically includes a convolutional neural network, an encoding layer of a Transformer network, a decoding layer of a Transformer network, and a feature prediction layer connected in sequence. The convolutional neural network extracts local features from the sample document image and provides them to the encoding layer; the encoding layer extracts global features from the sample document image and provides them to the decoding layer; the decoding layer decodes the features provided by the encoding layer to obtain decoded features; and the feature prediction layer predicts the decoded features to obtain detection boxes and the corresponding layout categories.

[0033] Step 102: For the sample document images in the first training data, input the sample document images into the second document layout detection model and obtain the layout prediction results output by the second document layout detection model.

[0034] In one embodiment of this disclosure, when the second document layout detection model is trained based on the second training data, the first document layout detection model and the second document layout detection model may have the same structure or different structures. When the structures are the same, the parameter precision of the first document layout detection model and the second document layout detection model may be different. For example, the parameter precision of the second document layout detection model may be higher than that of the first document layout detection model.

[0035] In another example of this disclosure, when the second document layout detection model is obtained by initializing the parameters of the first document layout detection model, the first document layout detection model and the second document layout detection model have the same structure but different parameter precision.

[0036] The first document layout detection model and the second document layout detection model have different parameter precision or different structures. Therefore, combining the layout prediction results output by the second document layout detection model with weak tags to determine pseudo tags can improve the accuracy of the determined pseudo tags.

[0037] Step 103: Determine the pseudo-labels of the sample document images based on the layout prediction results and weak labels.

[0038] In this embodiment of the disclosure, the pseudo-labels of the sample document image may include detection boxes and the layout categories corresponding to the detection boxes. The detection boxes are determined based on the detection boxes in the layout prediction results of the sample document image, and / or, relevant information about the detection boxes in the weak labels; the layout categories are determined based on the categories in the layout prediction results of the sample document image, and / or, relevant information about the layout categories in the weak labels.

[0039] Step 104: Based on the sample document images in the first training data and the pseudo-labels of the sample document images, train the first document layout detection model to obtain the trained document layout detection model.

[0040] In this embodiment of the disclosure, the initial first document layout detection model can be a model that has not been trained using the second training data, or it can be a model trained using the second training data.

[0041] In this embodiment of the disclosure, if the initial first document layout detection model is a model that has not been trained using the second training data, the electronic device may perform step 104 as follows: train the first document layout detection model based on the sample document images in the second training data and the true labels of the sample document images to obtain a trained first document layout detection model; and retrain the trained first document layout detection model based on the sample document images in the first training data and the pseudo labels of the sample document images to obtain a trained document layout detection model.

[0042] The process of retraining the first document layout detection model based on the first training data by the electronic device can be as follows: inputting sample document images from the first training data into the trained first document layout detection model to obtain the output predicted detection boxes and corresponding predicted layout categories; determining the value of the loss function based on the predicted detection boxes and corresponding predicted layout categories, the detection boxes and corresponding layout categories in the pseudo-labels, and the loss function; and adjusting the parameters of the first document layout detection model based on the value of the loss function to obtain the trained document layout detection model.

[0043] In the initial first document layout detection model, which is a model that has not been trained using the second training data, the accuracy of the trained document layout detection model can be improved by combining the pseudo-labels of the sample document images in the first training data and the true labels of the sample document images in the second training data.

[0044] The document layout detection model training method of this disclosure embodiment involves acquiring first training data, an initial first document layout detection model, and a second document layout detection model. The first training data includes sample document images and weak labels for the sample document images. The parameters of the second document layout detection model are determined based on the second training data, which includes true labels for multiple sample document images. For the sample document images in the first training data, the sample document images are input into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model. Based on the layout prediction results and weak labels of the sample document images, pseudo labels for the sample document images are determined. Based on the sample document images in the first training data and the pseudo labels of the sample document images, the first document layout detection model is trained to obtain a trained document layout detection model. The setting of weak labels and the determination of pseudo labels can reduce the annotation cost of sample document images, shorten the annotation time of sample document images, and improve the model training speed and efficiency.

[0045] To accurately combine weak labels and layout prediction results of sample document images to determine pseudo labels and improve the accuracy of the determined pseudo labels, when the weak label is the number of layout categories, the detection box and the corresponding layout category can be selected from the layout prediction results according to the number of layout categories, and used as pseudo labels. For example... Figure 2 As shown, Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure. Figure 2 The illustrated embodiment may include the following steps:

[0046] Step 201: Obtain first training data, an initial first document layout detection model, and a second document layout detection model; the first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on the second training data, which includes true labels of multiple sample document images; the weak label of the sample document image is the number of layout categories.

[0047] Step 202: For the sample document images in the first training data, input the sample document images into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model; the layout prediction results include multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category.

[0048] In this embodiment, the layout category of the detection box is the layout category corresponding to the category with the highest probability among all layout categories to which the detection box belongs. The detection box can be represented by the coordinates of its center point and the coordinates of its four corners. The layout category, such as text paragraph, table, table title, paragraph title, figure, figure title, header, footer, footnote, etc., can be set according to actual needs.

[0049] Step 203: Sort the multiple detection boxes in the layout prediction results in descending order according to their category probabilities to obtain the sorting results.

[0050] As an alternative to step 203, the electronic device can perform the following process to obtain the sorting result: sort the multiple detection boxes in the layout prediction result in ascending order according to the category probability to obtain the sorting result.

[0051] Step 204: Select the detection boxes with the highest category probabilities from the sorting results in sequence as target detection boxes, until the total number of layout categories of target detection boxes is consistent with the total number of layout categories.

[0052] In this embodiment of the disclosure, when the detection boxes in the sorting results are sorted in descending order of category probability, the electronic device may perform step 204 as follows: select the first detection box from the sorting results as the target detection box; delete the first detection box from the sorting results; count the total number of layout categories of the target detection boxes; if the total number of layout categories is less than the number of layout categories, continue to select the first detection box from the sorting results as the target detection box; repeat the above steps until the total number of layout categories of the target detection boxes is equal to the number of layout categories, that is, the total number of layout categories is consistent with the number of layout categories.

[0053] In order to avoid having only one detection box under a certain layout category and to avoid missing detection boxes, when the total number of layout categories of the target detection boxes is equal to the number of layout categories, the electronic device can continue to obtain the corresponding detection boxes with a category probability greater than or equal to the probability threshold from the sorting results, and use them as target detection boxes.

[0054] In this embodiment of the disclosure, it should also be noted that, in one example, when the weak label of the sample document image is unlabeled, the process of the electronic device determining the target detection box can be, for example, obtaining the target detection box in the layout prediction result, wherein the category probability of the target detection box is greater than or equal to the probability threshold; and using the target detection box and the layout category of the target detection box as pseudo-labels of the sample document image.

[0055] In the case where the weak label of the sample document image is unlabeled, the electronic device combines the category probability of the detection box and the probability threshold to select the target detection box to determine the pseudo label, which can improve the accuracy of the pseudo label in the case of no label.

[0056] In this embodiment of the disclosure, it should also be noted that, in another example, when the weak label of the sample document image is the number of detection boxes under the layout category, the process of the electronic device determining the target detection box can be as follows: for each layout category in the layout prediction result, the detection boxes with layout categories are sorted in descending order according to the category probability to obtain the sorting result; the detection boxes with the highest category probability are selected from the sorting result in turn as the target detection boxes, until the total number of target detection boxes is consistent with the number of detection boxes under the layout category; the target detection boxes and their layout categories are used as pseudo-labels of the sample document image.

[0057] In the case where the weak label of the sample document image is the number of detection boxes under the layout category, the electronic device combines the detection boxes and the corresponding category probabilities, as well as the number of detection boxes under each layout category, to select the target detection box and determine the pseudo label. This can improve the accuracy of the pseudo label when the weak label is the number of detection boxes under the layout category.

[0058] In this embodiment of the disclosure, it should also be noted that, in another example, when the weak label of the sample document image is the center point of the sample detection box, the process of the electronic device determining the target detection box can be, for example, obtaining the target detection box in the layout prediction result; the target detection box includes the center point of the sample detection box in the sample document image, and the class probability of the target detection box is greater than or equal to the probability threshold; the target detection box and the layout class of the target detection box are used as pseudo-labels of the sample document image.

[0059] The detection box can be represented by the coordinates of its center point and the coordinates of its four corners. The electronic device can determine the area of ​​the detection box based on the coordinates of its four corners; then, by combining this with the coordinates of the center points of each sample's detection box, it can determine whether the detection box includes the center point of that sample's detection box.

[0060] In the case where the weak label of the sample document image is the center point of the sample detection box, the electronic device selects the target detection box from the detection box that includes the center point of the sample detection box to determine the pseudo label, which can improve the accuracy of the pseudo label when the weak label is the center point of the sample detection box.

[0061] Step 205: Use the object detection bounding box and its layout category as pseudo-labels for the sample document image.

[0062] Step 206: Based on the sample document images in the first training data and the pseudo-labels of the sample document images, train the first document layout detection model to obtain the trained document layout detection model.

[0063] It should be noted that for details regarding steps 201 to 202, and step 206, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 to 102 and step 104 in the illustrated embodiment will not be described in detail here.

[0064] The training method for the document layout detection model in this embodiment of the present disclosure involves acquiring first training data, an initial first document layout detection model, and a second document layout detection model. The first training data includes sample document images and weak labels for the sample document images. The parameters of the second document layout detection model are determined based on the second training data, which includes true labels for multiple sample document images. The weak labels for the sample document images are the number of layout categories. For the sample document images in the first training data, the sample document images are input into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model. The layout prediction results include multiple detection boxes, the layout categories of the detection boxes, and the probability of the detection boxes belonging to the layout categories. For the multiple detection boxes in the layout prediction results, according to... The category probabilities are sorted in descending order to obtain the sorting results. The detection boxes with the highest category probabilities are selected from the sorting results as target detection boxes until the total number of layout categories for the target detection boxes matches the total number of layout categories. The target detection boxes and their layout categories are used as pseudo-labels for the sample document images. Based on the sample document images in the first training data and their pseudo-labels, the first document layout detection model is trained to obtain a trained document layout detection model. Selecting detection boxes and their corresponding layout categories from the layout prediction results as pseudo-labels according to the number of layout categories in the weak labels reduces the annotation cost of sample document images, shortens the annotation time, and improves the model training speed and efficiency.

[0065] To accurately combine weak labels and layout prediction results of sample document images to determine pseudo labels and improve the accuracy of the determined pseudo labels, when the weak labels are the center point of the sample detection box and the layout category, the detection box with the smaller difference in distance and layout category can be selected from the layout prediction results as the target detection box to determine the pseudo label. For example... Figure 3 As shown, Figure 3 This is a schematic diagram based on the third embodiment of the present disclosure. Figure 3 The illustrated embodiment may include the following steps:

[0066] Step 301: Obtain first training data, an initial first document layout detection model, and a second document layout detection model; the first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on the second training data, which includes true labels of multiple sample document images; the weak labels of the sample document images are the center point of the sample detection box and the layout category.

[0067] Step 302: For the sample document images in the first training data, input the sample document images into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model; the layout prediction results include multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category.

[0068] Step 303: For each sample detection box center point in the sample document image, determine the distance between the sample detection box center point and the center points of each detection box in the layout prediction result, as well as the class difference between the layout category corresponding to the sample detection box center point and the layout category of each detection box.

[0069] In this embodiment of the disclosure, the process by which the electronic device determines the category difference can be, for example, as follows: when the layout category corresponding to the center point of the sample detection frame is the same as the layout category of the detection frame, the category difference is determined to be a first value; when the layout category corresponding to the center point of the sample detection frame is different from the layout category of the detection frame, the category difference is determined to be a second value. The first value is less than the second value.

[0070] Step 304: Determine the matching degree of the detection boxes in the layout prediction results based on the distance and category difference of the detection boxes in the layout prediction results and the Hungarian matching algorithm.

[0071] In this embodiment, the Hungarian matching algorithm is a combinatorial optimization algorithm for solving the task allocation problem in polynomial time. In this embodiment, it can refer to a combinatorial optimization algorithm combining distance and class difference. For example, the process by which the electronic device determines the matching degree of the detection boxes can be as follows: the distance and class difference of the detection boxes in the layout prediction result are weighted and summed according to the Hungarian matching algorithm, and the matching degree is determined based on the processing result. The larger the processing result, the smaller the matching degree; the smaller the processing result, the larger the matching degree.

[0072] Step 305: Based on the matching degree, select the target detection box that matches the center point of the sample detection box from each detection box in the layout prediction result.

[0073] In this embodiment of the disclosure, for each sample detection box center point in the sample document image, the electronic device can select the detection box with the highest matching degree from multiple detection boxes determined by combining the sample detection box center point, as the target detection box that matches the sample detection box center point.

[0074] In this embodiment of the disclosure, it should also be noted that, when the weak label of the sample document image is a sample detection box, as an alternative to steps 303 to 305, the process of the electronic device determining the target detection box can be, for example, as follows: for each sample detection box in the sample document image, determine the distance between the sample detection box and each detection box in the layout prediction result, as well as the IOU loss; determine the matching degree of the detection boxes in the layout prediction result based on the distance and IOU loss of the detection boxes in the layout prediction result, and the Hungarian matching algorithm; select the target detection box that matches the sample detection box from each detection box in the layout prediction result based on the matching degree; and use the target detection box and the layout category of the target detection box as the pseudo label of the sample document image.

[0075] The process by which the electronic device determines the IOU loss can be, for example, as follows: for each sample detection box in the sample document image and for each detection box in the layout prediction result, determine the intersection between the sample detection box and the detection box, i.e., the size of the overlapping region; determine the union between the sample detection box and the detection box, i.e., the size of the total region occupied; determine the ratio between the size of the overlapping region and the size of the total region; and determine the difference between 1 and the ratio as the matching degree between the sample detection box and the detection box.

[0076] In the case where the weak label of the sample document image is the sample detection box, the electronic device selects the detection box with smaller difference from the layout prediction result based on the IOU loss and distance difference as the target detection box to determine the pseudo label, which can improve the accuracy of the pseudo label when the weak label is the sample detection box.

[0077] Step 306: Use the object detection bounding box and its layout category as pseudo-labels for the sample document image.

[0078] Step 307: Based on the sample document images in the first training data and the pseudo-labels of the sample document images, train the first document layout detection model to obtain the trained document layout detection model.

[0079] It should be noted that for details regarding steps 301 to 302 and step 307, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 to 102 and step 104 in the illustrated embodiment will not be described in detail here.

[0080] The training method for the document layout detection model in this embodiment of the present disclosure involves acquiring first training data, an initial first document layout detection model, and a second document layout detection model. The first training data includes sample document images and weak labels for the sample document images. The parameters of the second document layout detection model are determined based on the second training data, which includes true labels for multiple sample document images. The weak labels for the sample document images are the center point of the sample detection boxes and the layout category. For the sample document images in the first training data, the sample document images are input into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model. The layout prediction results include multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category. For each center point of the sample detection boxes in the sample document images, the distance between the center point of the sample detection box and the center points of each detection box in the layout prediction results, and the distance between the center point of the sample detection box and the center points of each detection box in the layout prediction results are determined. The document layout detection model is trained by considering the class difference between the layout category corresponding to the center point and the layout categories of each detection box; the matching degree of the detection boxes in the layout prediction results is determined based on the distance and class difference of the detection boxes in the layout prediction results and the Hungarian matching algorithm; based on the matching degree, target detection boxes that match the center point of the sample detection boxes are selected from each detection box in the layout prediction results; the target detection boxes and their layout categories are used as pseudo-labels for the sample document images; the first document layout detection model is trained based on the sample document images in the first training data and the pseudo-labels of the sample document images to obtain a trained document layout detection model. The method of selecting detection boxes and their corresponding layout categories from the layout prediction results as pseudo-labels according to the center point and layout category of the sample detection boxes in the weak labels can reduce the annotation cost of sample document images, shorten the annotation time of sample document images, and improve the model training speed and efficiency.

[0081] To implement the above embodiments, this disclosure also provides a training apparatus for a document layout detection model. For example... Figure 4 As shown, Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. The training device 40 for the document layout detection model may include: a first acquisition module 401, a second acquisition module 402, a determination module 403, and a training module 404.

[0082] The system includes a first acquisition module 401, which acquires first training data, an initial first document layout detection model, and a second document layout detection model. The first training data includes sample document images and weak labels for the sample document images. The parameters of the second document layout detection model are determined based on the second training data, which includes true labels for multiple sample document images. The second acquisition module 402 is used to input the sample document images from the first training data into the second document layout detection model and acquire the layout prediction results output by the second document layout detection model. The determination module 403 is used to determine the pseudo labels for the sample document images based on the layout prediction results and weak labels. The training module 404 is used to train the first document layout detection model based on the sample document images from the first training data and the pseudo labels for the sample document images to obtain a trained document layout detection model.

[0083] As one possible implementation of this disclosure, the initial first document layout detection model is a model that has not been trained using the second training data; the first acquisition module 401 is specifically used to acquire the first training data, the second training data, the initial first document layout detection model, and the initial second document layout detection model; and to train the initial second document layout detection model based on the sample document images in the second training data and the true labels of the sample document images to obtain a trained second document layout detection model.

[0084] As one possible implementation of this disclosure, the initial first document layout detection model is a model trained using the second training data; the first acquisition module 401 is specifically used to acquire the first training data, the initial first document layout detection model, and the initial second document layout detection model; and to perform parameter initialization processing on the initial second document layout detection model according to the parameters of the initial first document layout detection model to obtain the second document layout detection model.

[0085] As one possible implementation of this disclosure, the weak label of the sample document image can be any of the following: no label, number of layout categories, number of detection boxes under layout category, center point of sample detection box, center point of sample detection box, layout category, and sample detection box.

[0086] As one possible implementation of this disclosure, the weak label of the sample document image is no label; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to obtain the target detection boxes in the layout prediction result, wherein the category probability of the target detection boxes is greater than or equal to a probability threshold; and use the target detection boxes and the layout category of the target detection boxes as pseudo-labels of the sample document image.

[0087] As one possible implementation of this disclosure, the weak label of the sample document image is the number of layout categories; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to: sort the multiple detection boxes in the layout prediction result in descending order according to the category probability to obtain a sorting result; sequentially select the detection box with the highest category probability from the sorting result as the target detection box, until the total number of layout categories of the target detection boxes is consistent with the number of layout categories; and use the target detection box and the layout category of the target detection box as the pseudo label of the sample document image.

[0088] As one possible implementation of this disclosure, the weak label of the sample document image is the number of detection boxes under the layout category; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to, for each layout category in the layout prediction result, sort the detection boxes with the layout category in descending order according to the category probability to obtain a sorting result; sequentially select the detection box with the highest category probability from the sorting result as the target detection box, until the total number of the target detection boxes is consistent with the number of detection boxes under the layout category; and use the target detection box and the layout category of the target detection box as the pseudo label of the sample document image.

[0089] In one possible implementation of this disclosure, the weak label of the sample document image is the center point of the sample detection box; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to obtain the target detection box in the layout prediction result; the target detection box includes the center point of the sample detection box in the sample document image, and the category probability of the target detection box is greater than or equal to a probability threshold; the target detection box and the layout category of the target detection box are used as pseudo-labels of the sample document image.

[0090] As one possible implementation of this disclosure, the weak labels of the sample document image are the center point of the sample detection box and the layout category; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to: for each center point of the sample detection box in the sample document image, determine the distance between the center point of the sample detection box and the center points of each detection box in the layout prediction result, and the category difference between the layout category corresponding to the center point of the sample detection box and the layout categories of each detection box; determine the matching degree of the detection boxes in the layout prediction result based on the distance and category difference of the detection boxes in the layout prediction result and the Hungarian matching algorithm; select a target detection box that matches the center point of the sample detection box from each detection box in the layout prediction result based on the matching degree; and use the target detection box and the layout category of the target detection box as the pseudo label of the sample document image.

[0091] As one possible implementation of this disclosure, the weak label of the sample document image is a sample detection box; the layout prediction result includes multiple detection boxes, the layout category of the detection boxes, and the category probability of the detection boxes belonging to the layout category; the determining module 403 is specifically used to: for each sample detection box in the sample document image, determine the distance and IOU loss between the sample detection box and each detection box in the layout prediction result; determine the matching degree of the detection boxes in the layout prediction result based on the distance and IOU loss of the detection boxes in the layout prediction result and the Hungarian matching algorithm; select a target detection box that matches the sample detection box from each detection box in the layout prediction result based on the matching degree; and use the target detection box and the layout category of the target detection box as the pseudo label of the sample document image.

[0092] As one possible implementation of this disclosure, the initial first document layout detection model is a model that has not been trained using the second training data; the training module 404 is specifically used to: train the first document layout detection model based on the sample document images in the second training data and the true labels of the sample document images to obtain a trained first document layout detection model; and retrain the trained first document layout detection model based on the sample document images in the first training data and the pseudo labels of the sample document images to obtain a trained document layout detection model.

[0093] As one possible implementation of this disclosure, the first document layout detection model and the second document layout detection model may have the same structure or different structures; when the structures are the same, the parameter precision of the first document layout detection model and the second document layout detection model is different.

[0094] As one possible implementation of this disclosure, the first document layout detection model and the second document layout detection model include a convolutional neural network and a Transformer network connected in sequence.

[0095] The document layout detection model training apparatus of this disclosure acquires first training data, an initial first document layout detection model, and a second document layout detection model. The first training data includes sample document images and weak labels for the sample document images. The parameters of the second document layout detection model are determined based on the second training data, which includes true labels for multiple sample document images. For the sample document images in the first training data, the sample document images are input into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model. Based on the layout prediction results and weak labels of the sample document images, pseudo labels for the sample document images are determined. Based on the sample document images in the first training data and the pseudo labels of the sample document images, the first document layout detection model is trained to obtain a trained document layout detection model. The setting of weak labels and the determination of pseudo labels can reduce the annotation cost of sample document images, shorten the annotation time of sample document images, and improve the model training speed and efficiency.

[0096] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information are all carried out with the consent of the users, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0097] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0098] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0099] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0100] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the training method for a document layout detection model. For example, in some embodiments, the training method for a document layout detection model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the training method for the document layout detection model described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for the document layout detection model.

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0107] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0108] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a document layout detection model, the method comprising: Acquire the first training data, the initial first document layout detection model, and the second document layout detection model; The first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on the second training data including true labels of multiple sample document images; the weak labels include information related to detection boxes in the sample document images, and / or information related to layout categories in the sample document images; For the sample document images in the first training data, the sample document images are input into the second document layout detection model to obtain the layout prediction results output by the second document layout detection model; Based on the layout prediction results and weak labels of the sample document images, determine the pseudo labels of the sample document images; Based on the sample document images in the first training data and the pseudo-labels of the sample document images, the first document layout detection model is trained to obtain a trained document layout detection model. The layout prediction result includes multiple bounding boxes, the layout category of the bounding boxes, and the probability that the bounding boxes belong to the layout category; when the weak label of the sample document image is the center point of the sample bounding box and the layout category, determining the pseudo label of the sample document image based on the layout prediction result and the weak label includes: For each sample detection box center point in the sample document image, determine the distance between the sample detection box center point and the center points of each detection box in the layout prediction result, as well as the category difference between the layout category corresponding to the sample detection box center point and the layout category of each detection box. Based on the distance and category difference of the detection boxes in the layout prediction results, and the Hungarian matching algorithm, the matching degree of the detection boxes in the layout prediction results is determined. Based on the matching degree, select the target detection box that matches the center point of the sample detection box from each detection box in the layout prediction result; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

2. The method according to claim 1, wherein, The initial first document layout detection model is a model that was not trained using the second training data; The acquisition of the first training data, the initial first document layout detection model, and the second document layout detection model includes: Acquire the first training data, the second training data, the initial first document layout detection model, and the initial second document layout detection model; Based on the sample document images in the second training data and the true labels of the sample document images, the initial second document layout detection model is trained to obtain a trained second document layout detection model.

3. The method according to claim 1, wherein, The initial first document layout detection model is a model trained using the second training data; The acquisition of the first training data, the initial first document layout detection model, and the second document layout detection model includes: Obtain the first training data, the initial first document layout detection model, and the initial second document layout detection model; Based on the parameters of the initial first document layout detection model, the parameters of the initial second document layout detection model are initialized to obtain the second document layout detection model.

4. The method according to claim 1, wherein, The weak labels of the sample document images are any one of the following: number of layout categories, number of detection boxes under layout categories, center point of sample detection boxes, center point of sample detection boxes, layout category, and sample detection boxes.

5. The method according to claim 1 or 4, wherein, When the weak label of the sample document image is the number of layout categories, determining the pseudo label of the sample document image based on the layout prediction result and the weak label includes: The multiple detection boxes in the layout prediction result are sorted in descending order according to their category probabilities to obtain the sorting result; Select the detection box with the highest category probability from the sorting results in sequence as the target detection box, until the total number of layout categories of the target detection box is consistent with the number of layout categories; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

6. The method according to claim 1 or 4, wherein, When the weak labels of the sample document image are the number of detection boxes under the layout category, the step of determining the pseudo-label of the sample document image based on the layout prediction result and the weak labels includes: For each layout category in the layout prediction result, the detection boxes with the layout category are sorted in descending order according to the category probability to obtain the sorting result. Select the detection box with the highest category probability from the sorting results in sequence as the target detection box, until the total number of the target detection boxes is consistent with the number of detection boxes under the layout category; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

7. The method according to claim 1 or 4, wherein, When the weak label of the sample document image is the center point of the sample detection box, the step of determining the pseudo label of the sample document image based on the layout prediction result and the weak label includes: Obtain the target detection box in the layout prediction result; the target detection box includes the center point of the sample detection box in the sample document image, and the class probability of the target detection box is greater than or equal to the probability threshold; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

8. The method according to claim 1 or 4, wherein, When the weak label of the sample document image is a sample detection box, the step of determining the pseudo label of the sample document image based on the layout prediction result and the weak label includes: For each sample detection box in the sample document image, determine the distance between the sample detection box and each detection box in the layout prediction result, as well as the IOU loss. Based on the distance of the detection boxes in the layout prediction results, the IOU loss, and the Hungarian matching algorithm, the matching degree of the detection boxes in the layout prediction results is determined. Based on the matching degree, select the target detection box that matches the sample detection box from each detection box in the layout prediction result; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

9. The method according to claim 1, wherein, The initial first document layout detection model is a model that has not been trained using the second training data; the step of training the first document layout detection model based on sample document images in the first training data and the pseudo-labels of the sample document images to obtain a trained document layout detection model includes: Based on the sample document images in the second training data and the true labels of the sample document images, the first document layout detection model is trained to obtain the trained first document layout detection model. Based on the sample document images in the first training data and the pseudo-labels of the sample document images, the trained first document layout detection model is retrained to obtain a trained document layout detection model.

10. The method according to any one of claims 2 to 3, wherein, The first document layout detection model and the second document layout detection model may have the same structure or different structures; When the structures are the same, the parameter precision of the first document layout detection model and the second document layout detection model are different.

11. The method according to claim 1, wherein, The first document layout detection model and the second document layout detection model each include a convolutional neural network and a Transformer network connected in sequence.

12. A training device for a document layout detection model, the device comprising: The first acquisition module is used to acquire the first training data, the initial first document layout detection model, and the second document layout detection model; The first training data includes sample document images and weak labels of the sample document images; the parameters of the second document layout detection model are determined based on the second training data including true labels of multiple sample document images; the weak labels include information related to detection boxes in the sample document images, and / or information related to layout categories in the sample document images; The second acquisition module is used to input the sample document images in the first training data into the second document layout detection model and obtain the layout prediction results output by the second document layout detection model. The determination module is used to determine the pseudo-labels of the sample document image based on the layout prediction results and weak labels of the sample document image; The training module is used to train the first document layout detection model based on the sample document images in the first training data and the pseudo-labels of the sample document images, so as to obtain a trained document layout detection model. The layout prediction result includes multiple bounding boxes, the layout category of each bounding box, and the probability that each bounding box belongs to a particular layout category. When the weak label of the sample document image is the center point of the sample bounding box and the layout category, the determination module is specifically used for... For each sample detection box center point in the sample document image, determine the distance between the sample detection box center point and the center points of each detection box in the layout prediction result, as well as the category difference between the layout category corresponding to the sample detection box center point and the layout category of each detection box. Based on the distance and category difference of the detection boxes in the layout prediction results, and the Hungarian matching algorithm, the matching degree of the detection boxes in the layout prediction results is determined. Based on the matching degree, select the target detection box that matches the center point of the sample detection box from each detection box in the layout prediction result; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

13. The apparatus according to claim 12, wherein, The initial first document layout detection model is a model that was not trained using the second training data; the first acquisition module is specifically used for, Acquire the first training data, the second training data, the initial first document layout detection model, and the initial second document layout detection model; Based on the sample document images in the second training data and the true labels of the sample document images, the initial second document layout detection model is trained to obtain a trained second document layout detection model.

14. The apparatus according to claim 12, wherein, The initial first document layout detection model is a model trained using the second training data; the first acquisition module is specifically used for, Obtain the first training data, the initial first document layout detection model, and the initial second document layout detection model; Based on the parameters of the initial first document layout detection model, the parameters of the initial second document layout detection model are initialized to obtain the second document layout detection model.

15. The apparatus according to claim 12, wherein, The weak labels of the sample document images are any one of the following: number of layout categories, number of detection boxes under layout categories, center point of sample detection boxes, center point of sample detection boxes, layout category, and sample detection boxes.

16. The apparatus according to claim 12 or 15, wherein, When the weak label of the sample document image is the number of layout categories, the determining module is specifically used for, The multiple detection boxes in the layout prediction result are sorted in descending order according to their category probabilities to obtain the sorting result; Select the detection box with the highest category probability from the sorting results in sequence as the target detection box, until the total number of layout categories of the target detection box is consistent with the number of layout categories; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

17. The apparatus according to claim 12 or 15, wherein, When the weak label of the sample document image is the number of detection boxes under the layout category, the determining module is specifically used for: For each layout category in the layout prediction result, the detection boxes with the layout category are sorted in descending order according to the category probability to obtain the sorting result. Select the detection box with the highest category probability from the sorting results in sequence as the target detection box, until the total number of the target detection boxes is consistent with the number of detection boxes under the layout category; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

18. The apparatus according to claim 12 or 15, wherein, When the weak label of the sample document image is the center point of the sample detection box, the determining module is specifically used for, Obtain the target detection box in the layout prediction result; the target detection box includes the center point of the sample detection box in the sample document image, and the class probability of the target detection box is greater than or equal to the probability threshold; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

19. The apparatus according to claim 12 or 15, wherein, When the weak label of the sample document image is a sample detection box, the determining module is specifically used for, For each sample detection box in the sample document image, determine the distance between the sample detection box and each detection box in the layout prediction result, as well as the IOU loss. Based on the distance of the detection boxes in the layout prediction results, the IOU loss, and the Hungarian matching algorithm, the matching degree of the detection boxes in the layout prediction results is determined. Based on the matching degree, select the target detection box that matches the sample detection box from each detection box in the layout prediction result; The target detection bounding box and its layout category are used as pseudo-labels for the sample document image.

20. The apparatus according to claim 12, wherein, The initial first document layout detection model is a model that was not trained using the second training data; the training module is specifically used for, Based on the sample document images in the second training data and the true labels of the sample document images, the first document layout detection model is trained to obtain the trained first document layout detection model. Based on the sample document images in the first training data and the pseudo-labels of the sample document images, the trained first document layout detection model is retrained to obtain a trained document layout detection model.

21. The apparatus according to any one of claims 13 to 14, wherein, The first document layout detection model and the second document layout detection model may have the same structure or different structures; When the structures are the same, the parameter precision of the first document layout detection model and the second document layout detection model are different.

22. The apparatus according to claim 12, wherein, The first document layout detection model and the second document layout detection model each include a convolutional neural network and a Transformer network connected in sequence.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images