A method for processing a field in an image and a related device

Abstract

The application provides a field processing method in an image and a related device. The method can accurately identify the categories of all second fields in a to-be-processed image based on the matching results between a plurality of first fields in a reference image and a plurality of second fields in the to-be-processed image. The method comprises the following steps: obtaining a plurality of first fields in a reference image and a plurality of second fields in a to-be-processed image; obtaining a plurality of first features corresponding to the plurality of first fields and a plurality of second features corresponding to the plurality of second fields through a field processing model; processing the plurality of first features and the plurality of second features based on a graph matching algorithm through the field processing model to obtain the matching results between the plurality of first fields and the plurality of second fields; and determining the categories of the plurality of second fields based on the matching results and the categories of the plurality of first fields.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method for processing fields in an image and related equipment. Background Technology

[0002] To improve office efficiency, automatic recognition technologies can be used to extract information from documents and receipts. For example, an invoice image containing fields such as "Zhang XX" and "19XX year X month X day" can be processed to identify fields such as "Zhang XX" belonging to the name category and "19XX year X month X day" belonging to the date category, thus completing the extraction of key information from the invoice.

[0003] Currently, when it is necessary to determine the category of multiple fields in an image to be processed, a reference image can be obtained first. The reference image contains multiple fields and the category of each field. Since the reference image and the image to be processed are images of the same type of document or ticket (for example, both are images of the same type of ticket), the category of each field in the image to be processed can be determined based on the correspondence between the positions of each field in the reference image and the positions of each field in the image to be processed.

[0004] However, if there is an error in the printing of the absolute position of each field in the image to be processed (for example, field drift), the absolute position of some fields in the reference image cannot accurately correspond to the absolute position of some fields in the image to be processed, which will lead to the misclassification of the category of some fields in the image to be processed. Summary of the Invention

[0005] This application provides a method and related device for processing fields in an image, which can accurately identify the category of all second fields in the image to be processed based on the matching results between multiple first fields in a reference image and multiple second fields in the image to be processed.

[0006] A first aspect of this application provides a method for processing fields in an image, the method comprising:

[0007] When it is necessary to determine the category of a field in an image to be processed, the image to be processed and a corresponding reference image can be obtained first. It should be noted that the reference image presents multiple first fields and multiple categories, and the correspondence between these multiple first fields and multiple categories is determined; that is, the categories of the multiple first fields are determined. The image to be processed also presents multiple second fields and multiple categories, but the correspondence between these multiple second fields and multiple categories is undetermined; that is, the categories of the multiple second fields are undetermined.

[0008] After obtaining the reference image and the image to be processed, information of multiple first fields can be obtained from the reference image and information of multiple second fields can be obtained from the image to be processed. The information of multiple first fields and multiple second fields is then input into the field processing model to process the information and obtain multiple first features and multiple second features. The multiple first fields correspond one-to-one with the multiple first features, and the multiple second fields correspond one-to-one with the multiple second features.

[0009] After obtaining multiple first features and multiple second features, the field processing model can also process the multiple first features and multiple second features based on the graph matching algorithm to obtain the matching results between multiple first fields and multiple second fields. The matching results include the matching degree between any first field and any second field.

[0010] After obtaining the matching results output by the field processing model, since the categories of multiple first fields are known, the categories of multiple second fields can be determined based on the matching results and the categories of multiple first fields.

[0011] As can be seen from the above method: after obtaining multiple first fields from the reference image and multiple second fields from the image to be processed, the multiple first fields and multiple second fields are first processed using a field processing model to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the multiple first features and multiple second features are processed using the field processing model to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0012] Furthermore, the field processing model in the above method introduces a graph matching algorithm. In the process of processing the second field of the image to be processed, the position of the second field is not sensitive. Even if some second fields in the image to be processed drift, the category of each second field can be accurately identified.

[0013] In one possible implementation, obtaining matching results between multiple first features and multiple second features by processing multiple first features and multiple second features using a graph matching algorithm through a field processing model includes: processing multiple first features, multiple second features, multiple first edges, and multiple second edges using a graph matching algorithm through a field processing model to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features, and the second edge is the distance between two second features. In the aforementioned implementation, the field processing model can calculate the pairwise distances between multiple first features (i.e., calculate the distance between any two first features) to obtain multiple first edges, and calculate the pairwise distances between multiple second features (i.e., calculate the distance between any two second features) to obtain multiple second edges. After obtaining multiple first edges and multiple second edges, the field processing model can process the multiple first features, multiple second features, multiple first edges, and multiple second edges to obtain matching results between multiple first fields and multiple second fields. When calculating the matching results between multiple first fields and multiple second fields, the field processing model introduces multiple first edges and multiple second edges in addition to multiple first features and multiple second features. Therefore, the field processing model not only considers the influence of the features themselves on the matching results, but also considers the influence of the relationship between features on the matching results, which helps to improve the accuracy of the matching results obtained by the model.

[0014] In one possible implementation, the matching results between multiple first features, multiple second features, multiple first edges, and multiple second edges are processed by a field processing model based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields. This includes: processing the multiple first features and multiple second features by the field processing model to obtain a first matrix, which contains the similarity between any first feature and any second feature; processing the multiple first edges and multiple second edges by the field processing model to obtain a second matrix, which contains the similarity between any first edge and any second edge; and processing the first matrix and the second matrix by the field processing model based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields. In the aforementioned implementation, for a given first feature, the field processing model can calculate the similarity between the first feature and each second feature based on machine learning (e.g., attention mechanisms), and use the similarity between the first feature and each second feature as each element of a row in the first matrix. For other first features, the field processing model can perform the same process as described above, thus generating a complete first matrix. Each element in a row of the first matrix represents the similarity between a first feature and each second feature, and each element in a column of the first matrix represents the similarity between a second feature and each first feature. Similarly, for a given first edge, the field processing model can calculate the similarity between that first edge and each second edge based on machine learning (e.g., attention mechanisms), and use this similarity as each element in a row of the second matrix. For other first edges, the field processing model can perform the same process as described above, thus generating a complete second matrix. Each element in a row of the second matrix represents the similarity between a first edge and each second edge, and each element in a column of the second matrix represents the similarity between a second edge and each first edge. After obtaining the first matrix and the second matrix, the field processing model can process the first matrix and the second matrix based on the graph matching algorithm, thereby accurately obtaining the matching results between multiple first fields and multiple second fields. Based on the matching results, even if some second fields in the image to be processed show a drift phenomenon, the category of these second fields can be accurately identified.

[0015] In one possible implementation, the matching result includes the matching degree between any first field and any second field. Based on the matching result and the categories of multiple first fields, determining the categories of multiple second fields includes: for any given second field, based on the matching degree between that second field and each first field, determining the category of the first field with the highest matching degree as the category of that second field. In the aforementioned implementation, among multiple second fields, for any given second field, based on the matching degree between that second field and each first field, the category of the first field with the highest matching degree is determined as the category of that second field. Therefore, for other second fields, the same process can be performed, thus determining the category of each of the multiple second fields.

[0016] In one possible implementation, before determining the category of the first field with the highest matching degree between any given second field and each first field, the method further includes: for any given second field, if the matching degree between the second field and each first field is less than a preset threshold, then the second field is removed from the multiple second fields. In the aforementioned implementation, among multiple second fields, if the matching degree between any given second field and each first field is less than a preset threshold, then the second field is considered an abnormal field and removed from the multiple second fields; if the matching degree between the second field and a certain first field is greater than or equal to a preset threshold, then the second field is considered a normal field and retained. Therefore, the same process can be performed for other second fields. This completes the filtering of multiple second fields, detecting abnormal fields among them, thus preventing the assignment of any category to abnormal fields and only assigning categories to normal fields, further improving the accuracy of field category recognition.

[0017] In one possible implementation, the first feature is generated based on at least one of the following: the content indicated by the first field, the center point of the region where the first field is located, and the size of the region where the first field is located. In the aforementioned implementation, for any one of the multiple first fields, the field processing model can process the information of that first field to obtain its first feature. For example, the field processing model can use the center point of the region where the first field is located as its spatial feature, the size (also known as the aspect ratio) of the region where the first field is located as its shape feature, and perform feature extraction processing on the content indicated by the first field to obtain its textual features. Then, the field processing model can perform linear transformation processing on the spatial, shape, and textual features of the first field to map them to a higher-dimensional space, obtaining new spatial, shape, and textual features for the first field. These three features can then serve as the first feature of the first field. In this way, the field processing model can obtain multiple first features corresponding one-to-one with the multiple first fields.

[0018] In one possible implementation, the second feature is generated based on at least one of the following: the content indicated by the second field, the center point of the region containing the second field, and the size of the region containing the second field. In the aforementioned implementation, for any one of the multiple second fields, the field processing model can process the information of that second field to obtain its second feature. For example, the field processing model can use the center point of the region containing the second field as its spatial feature, the size of the region as its shape feature, and perform feature extraction processing on the content indicated by the second field to obtain its textual feature. Then, the field processing model can perform linear transformation processing on the spatial, shape, and textual features of the second field to map them to a higher-dimensional space, obtaining new spatial, shape, and textual features for the second field. These three features can then serve as the second feature of the second field. In this way, the field processing model can obtain multiple second features corresponding one-to-one with the multiple second fields.

[0019] In one possible implementation, the first field indicates editable information related to a specific user. For example, the first field might indicate a user's (e.g., Zhang XX's) name, birthday, gender, or other information relevant to that user. If the first field indicates the name, birthday, gender, or other information of another user (e.g., Wang X), then the content indicated by the first field changes. Therefore, the content indicated by the first field is editable and not fixed. Similarly, the second field also indicates editable information related to a specific user.

[0020] A second aspect of this application provides a model training method, which includes: acquiring multiple first fields of a reference image and multiple second fields of an image to be processed; acquiring multiple first features corresponding one-to-one with the multiple first fields and multiple second features corresponding one-to-one with the multiple second fields through a model to be trained; processing the multiple first features and multiple second features using a graph matching algorithm based on the model to be trained to obtain matching results between the multiple first fields and multiple second fields; determining the predicted categories of the multiple second fields based on the matching results and the true categories of the multiple first fields; and training the model to be trained based on the true categories and the predicted categories of the multiple second fields to obtain a field processing model.

[0021] The field processing model obtained by the above method has the ability to classify fields in the image to be processed. Specifically, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the field processing model is first used to process the multiple first fields and multiple second fields to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the field processing model is used to process the multiple first features and multiple second features to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0022] In one possible implementation, the matching results between multiple first features and multiple second features are obtained by processing multiple first fields and multiple second fields using a graph matching algorithm based on the model to be trained. This includes processing multiple first features, multiple second features, multiple first edges, and multiple second edges using a graph matching algorithm based on the model to be trained, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0023] In one possible implementation, the matching results between multiple first features, multiple second features, multiple first edges, and multiple second edges are processed by the model to be trained using a graph matching algorithm. This process includes: processing the multiple first features and multiple second features by the model to be trained to obtain a first matrix, which contains the similarity between any one first feature and any one second feature; processing the multiple first edges and multiple second edges by the model to be trained to obtain a second matrix, which contains the similarity between any one first edge and any one one second edge; and processing the first matrix and the second matrix by the model to be trained using a graph matching algorithm to obtain the matching results between multiple first fields and multiple second fields.

[0024] In one possible implementation, the matching result includes the matching degree between any first field and any second field. Based on the matching result and the true categories of multiple first fields, determining the predicted categories of multiple second fields includes: for any second field, based on the matching degree between the second field and each first field, determining the true category of the first field with the highest matching degree as the predicted category of the second field.

[0025] In one possible implementation, before determining the true category of the first field with the highest matching degree as the predicted category of the second field based on the matching degree between the second field and each first field for any second field, the method further includes: for any second field, if the matching degree between the second field and each first field is less than a preset threshold, then the second field is removed from the multiple second fields.

[0026] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0027] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0028] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0029] A third aspect of this application also provides a method for processing fields in an image. This method can be implemented by a cloud server in a cloud service system. The method includes: obtaining a target instruction, which is used to indicate a plurality of first fields in a reference image and a plurality of second fields in an image to be processed; processing the plurality of first fields and the plurality of second fields indicated by the target instruction based on a graph matching algorithm to obtain a recognition result, which is used to indicate the category of the plurality of second fields.

[0030] As can be seen from the above method, after obtaining the target instruction, the cloud server can determine multiple first fields in the reference image indicated by the target instruction and multiple second fields in the image to be processed. Then, the cloud server can process the first fields and multiple second fields based on a graph matching algorithm to obtain recognition results indicating the categories of the multiple second fields. In the aforementioned process, the cloud server uses a graph matching algorithm when processing the second fields of the image to be processed. Therefore, the cloud server is not sensitive to the position of the second fields. Even if some second fields in the image to be processed drift, the cloud server can still accurately identify the category of each second field.

[0031] In one possible implementation, the recognition result is obtained by processing multiple first fields and multiple second fields indicated by the target instruction based on a graph matching algorithm, including: obtaining multiple first features corresponding one-to-one with the multiple first fields, and multiple second features corresponding one-to-one with the multiple second fields; processing the multiple first features and multiple second features to obtain a first matrix, which contains the similarity between any one first feature and any one second feature; processing multiple first edges and multiple second edges to obtain a second matrix, which contains the similarity between any one first edge and any one second edge; and processing the first matrix and the second matrix based on the graph matching algorithm to obtain the recognition result.

[0032] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0033] A fourth aspect of this application provides an apparatus for processing fields in an image. The apparatus includes: an acquisition module for acquiring a plurality of first fields of a reference image and a plurality of second fields of an image to be processed; a first processing module for acquiring a plurality of first features corresponding one-to-one with the plurality of first fields and a plurality of second features corresponding one-to-one with the plurality of second fields through a field processing model; a second processing module for processing the plurality of first features and the plurality of second features based on a graph matching algorithm using the field processing model to obtain a matching result between the plurality of first fields and the plurality of second fields; and a determination module for determining the category of the plurality of second fields based on the matching result and the category of the plurality of first fields.

[0034] As can be seen from the above apparatus: after acquiring multiple first fields of the reference image and multiple second fields of the image to be processed, the multiple first fields and multiple second fields are first processed through a field processing model to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the multiple first features and multiple second features are processed through the field processing model to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0035] Furthermore, the field processing model in the above-mentioned device introduces a graph matching algorithm. In the process of processing the second field of the image to be processed, the position of the second field is not sensitive. Even if some second fields in the image to be processed drift, the category of each second field can be accurately identified.

[0036] In one possible implementation, the second processing module is used to process multiple first features, multiple second features, multiple first edges, and multiple second edges based on a graph matching algorithm using a field processing model to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0037] In one possible implementation, the second processing module is configured to: process multiple first features and multiple second features using a field processing model to obtain a first matrix, the first matrix containing the similarity between any first feature and any second feature; process multiple first edges and multiple second edges using a field processing model to obtain a second matrix, the second matrix containing the similarity between any first edge and any second edge; and process the first matrix and the second matrix using a graph matching algorithm based on the field processing model to obtain matching results between multiple first fields and multiple second fields.

[0038] In one possible implementation, the matching result includes the matching degree between any first field and any second field. The determining module is used to determine the category of the first field with the highest matching degree as the category of the second field for any second field, based on the matching degree between the second field and each first field.

[0039] In one possible implementation, the device further includes a rejection module, which rejects any second field from a plurality of second fields if the matching degree between the second field and each first field is less than a preset threshold.

[0040] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0041] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0042] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0043] A fifth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring a plurality of first fields of a reference image and a plurality of second fields of an image to be processed; a first processing module for acquiring a plurality of first features corresponding one-to-one with the plurality of first fields and a plurality of second features corresponding one-to-one with the plurality of second fields through a model to be trained; a second processing module for processing the plurality of first features and the plurality of second features through a graph matching algorithm based on the model to be trained to obtain matching results between the plurality of first fields and the plurality of second fields; a determination module for determining the predicted categories of the plurality of second fields based on the matching results and the true categories of the plurality of first fields; and a training module for training the model to be trained based on the true categories and the predicted categories of the plurality of second fields to obtain a field processing model.

[0044] The field processing model obtained by the aforementioned device has the ability to classify fields in the image to be processed. Specifically, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the field processing model is first used to process the multiple first fields and multiple second fields to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the field processing model is used to process the multiple first features and multiple second features to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of the second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0045] In one possible implementation, the second processing module is used to process multiple first features, multiple second features, multiple first edges, and multiple second edges using a graph matching algorithm based on the model to be trained, to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0046] In one possible implementation, the second processing module is used to: process multiple first features and multiple second features using the model to be trained to obtain a first matrix, the first matrix containing the similarity between any first feature and any second feature; process multiple first edges and multiple second edges using the model to be trained to obtain a second matrix, the second matrix containing the similarity between any first edge and any second edge; and process the first matrix and the second matrix using the model to be trained based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields.

[0047] In one possible implementation, the matching result includes the matching degree between any first field and any second field. The determination module is used to determine the true category of the first field with the highest matching degree as the predicted category of the second field for any second field, based on the matching degree between the second field and each first field.

[0048] In one possible implementation, the device further includes a rejection module, which rejects any second field from a plurality of second fields if the matching degree between the second field and each first field is less than a preset threshold.

[0049] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0050] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0051] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0052] A sixth aspect of this application also provides an image field processing apparatus, which can be set in a cloud server in a cloud service system. The apparatus includes: an acquisition module for acquiring a target instruction, the target instruction indicating a plurality of first fields in a reference image and a plurality of second fields in an image to be processed; and a processing module for processing the plurality of first fields and the plurality of second fields indicated by the target instruction based on a graph matching algorithm to obtain a recognition result, the recognition result indicating the category of the plurality of second fields.

[0053] As can be seen from the above device, after obtaining the target instruction, the cloud server can determine multiple first fields in the reference image indicated by the target instruction and multiple second fields in the image to be processed. Then, the cloud server can process the first fields and multiple second fields based on a graph matching algorithm to obtain recognition results indicating the categories of the multiple second fields. In the aforementioned process, the cloud server uses a graph matching algorithm when processing the second fields of the image to be processed. Therefore, the cloud server is not sensitive to the position of the second fields. Even if some second fields in the image to be processed drift, the cloud server can still accurately identify the category of each second field.

[0054] In one possible implementation, the processing module is configured to: process multiple first fields and multiple second fields indicated by the target instruction based on a graph matching algorithm to obtain a recognition result, including: acquiring multiple first features corresponding one-to-one with the multiple first fields, and multiple second features corresponding one-to-one with the multiple second fields; processing the multiple first features and multiple second features to obtain a first matrix, the first matrix containing the similarity between any one of the first features and any one of the second features; processing multiple first edges and multiple second edges to obtain a second matrix, the second matrix containing the similarity between any one of the first edges and any one of the second edges; and processing the first matrix and the second matrix based on the graph matching algorithm to obtain the recognition result.

[0055] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0056] A seventh aspect of this application provides an apparatus for processing fields in an image, the apparatus including a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the apparatus for processing fields in an image performs the method described in the first aspect or any possible implementation thereof.

[0057] An eighth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0058] A ninth aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0059] A tenth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory, such that the processor executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0060] In one possible implementation, the processor is coupled to the memory via an interface.

[0061] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0062] The eleventh aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0063] The twelfth aspect of this application provides a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0064] In this embodiment, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the multiple first fields and multiple second fields are first processed using a field processing model to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the multiple first features and multiple second features are processed using the field processing model to obtain matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0065] Furthermore, the field processing model in the above-mentioned device introduces a graph matching algorithm. In the process of processing the second field of the image to be processed, the position of the second field is not sensitive. Even if some second fields in the image to be processed drift, the category of each second field can be accurately identified. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the related technology;

[0067] Figure 2 A structural diagram illustrating the main framework of artificial intelligence;

[0068] Figure 3a A schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0069] Figure 3b This is another schematic diagram of the image processing system provided in the embodiments of this application;

[0070] Figure 3c A schematic diagram of an image processing device provided in an embodiment of this application;

[0071] Figure 4a A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0072] Figure 4b A schematic diagram of a figure provided for an embodiment of this application;

[0073] Figure 5 A flowchart illustrating a method for processing fields in an image provided in an embodiment of this application;

[0074] Figure 6A schematic diagram of a reference image and an image to be processed provided for embodiments of this application;

[0075] Figure 7 A schematic diagram of the field processing model provided in the embodiments of this application;

[0076] Figure 8a A schematic diagram illustrating the generation process of the first matrix provided in an embodiment of this application;

[0077] Figure 8b A schematic diagram illustrating the generation process of the second matrix provided in an embodiment of this application;

[0078] Figure 9 A schematic diagram of the matching results provided in the embodiments of this application;

[0079] Figure 10 A schematic diagram of the extraction results of key information provided in the embodiments of this application;

[0080] Figure 11 A schematic flowchart of the model training method provided in the embodiments of this application;

[0081] Figure 12 A schematic diagram of a device for processing fields in an image provided in an embodiment of this application;

[0082] Figure 13 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0083] Figure 14 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0084] Figure 15 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0085] Figure 16 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0086] This application provides a method and related device for processing fields in an image, which can accurately identify the category of all second fields in the image to be processed based on the matching results between multiple first fields in a reference image and multiple second fields in the image to be processed.

[0087] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0088] To improve office efficiency, automatic recognition technologies can be used to extract information from documents and receipts. For example, for a receipt image containing fields such as "Zhang XX" and "19XX year X month X day", the image can be processed to identify that "Zhang XX" belongs to the name category, "19XX year X month X day" belongs to the date category, and so on, thus completing the extraction of key information from the receipt.

[0089] Currently, when it is necessary to determine the category of multiple fields in an image to be processed (also known as a query image), a reference image (also known as a supporting image) can be obtained first. The reference image and the image to be processed are images of the same type of document or ticket, and both contain fields of the same category. For example, ... Figure 1 As shown ( Figure 1 (This is a schematic diagram of a related technology.) Both the reference image and the image to be processed are images of a certain type of invoice. The reference image presents multiple categories of fields, such as name, date, and amount. The image to be processed also presents multiple categories of fields, such as name, date, and amount. Therefore, the category of each field in the image to be processed can be determined based on the correspondence between the positions of each field in the reference image and the positions of each field in the image to be processed.

[0090] However, if there is an error in the printing of the absolute position of each field in the image to be processed, that is, if the fields in the image to be processed drift, such as... Figure 1 As shown, printing errors occur in multiple categories of fields in the image to be processed, such as the name category, date category, and amount category. This causes the absolute positions of all fields to shift upwards. Therefore, it is possible to misidentify the date category field in the image to be processed as the name category field in the reference image, and the amount category field in the image to be processed as the date category field in the reference image. In other words, the date category field in the image to be processed will be misidentified as the name category field, and the amount category field in the image to be processed will be misidentified as the date category field.

[0091] To address the aforementioned problems, this application provides a method and related equipment for processing fields in an image. This method can be implemented using artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Using artificial intelligence for image processing is a common application of artificial intelligence.

[0092] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 2 , Figure 2 This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0093] (1) Infrastructure

[0094] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0095] (2) Data

[0096] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0097] (3) Data processing

[0098] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0099] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0100] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0101] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0102] (4) General ability

[0103] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0104] (5) Smart Products and Industry Applications

[0105] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0106] The following sections will introduce several application scenarios for this application.

[0107] Figure 3a This is a schematic diagram of an image processing system provided in an embodiment of this application. The image processing system includes a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of image processing; as the initiator of image processing requests, requests are typically initiated by the user through the user device.

[0108] The aforementioned data processing equipment can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing equipment receives image processing requests from smart terminals through an interactive interface, and then performs image processing methods such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0109] exist Figure 3a In the image processing system shown, the user equipment can receive user instructions. For example, the user equipment can acquire an image input / selected by the user and then send a request to the data processing device, causing the data processing device to perform image processing applications (e.g., processing of fields in the image) on the image acquired by the user equipment, thereby obtaining the corresponding processing result for the image. For instance, the user equipment can acquire an image input by the user and then send a processing request to the data processing device for fields in the image, causing the data processing device to classify the fields in the image, thereby obtaining the category to which each field belongs. In this way, the extraction of key information from the image is completed.

[0110] exist Figure 3a In this embodiment, the data processing device can execute the processing method for fields in the image according to the present application.

[0111] Figure 3b This is another schematic diagram of the image processing system provided in the embodiments of this application. Figure 3b In this context, the user equipment (UE) directly functions as a data processing device. This UE can directly acquire input from the user and process it directly through its own hardware. The specific process is similar to... Figure 3a Similar to the description above, it will not be repeated here.

[0112] exist Figure 3b In the image processing system shown, the user equipment can receive instructions from the user. For example, the user equipment can acquire an image selected by the user and then perform image processing applications (such as processing fields in the image) on the image to obtain the corresponding processing result for the image.

[0113] exist Figure 3b In this embodiment, the user equipment itself can execute the image field processing method of this application.

[0114] Figure 3c This is a schematic diagram of an image processing device provided in an embodiment of this application.

[0115] The above Figure 3a and Figure 3b The user equipment in the context can specifically be Figure 3c Local device 301 or local device 302 in the system. Figure 3a The data processing equipment in the middle can specifically be Figure 3c The execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0116] Figure 3a and Figure 3b The processor in the image can be trained on data using neural network models or other models (e.g., support vector machine-based models) for machine learning / deep learning, and then use the trained or learned models to perform image processing applications on the image to obtain the corresponding processing results.

[0117] Figure 4a A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 4a In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include various scheduled tasks, callable resources, and other parameters.

[0118] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0119] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0120] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0121] exist Figure 4aIn the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0122] It is worth noting that, Figure 4a This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 4a In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 4a As shown, a neural network can be trained using training device 120.

[0123] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 4a The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 4a The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0124] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the main CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0125] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0126] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0127] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0128] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0129] The unified memory is used to store input data and output data.

[0130] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0131] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0132] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0133] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0134] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memories.

[0135] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0136] (1) Neural Network

[0137] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0138]

[0139] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0140] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0141] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0142] (2) Backpropagation algorithm

[0143] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0144] (3) Graph Theory

[0145] Graph theory (GT) is a branch of combinatorics closely related to other branches of mathematics, such as group theory, matrix theory, and topology. Graphs are the primary object of study in graph theory. Figure 4b As shown ( Figure 4b (A schematic diagram of a figure provided in an embodiment of this application) A figure is a graph composed of several given nodes and edges connecting two nodes. Such a graph is usually used to describe a specific relationship between certain things. Nodes (e.g., the first feature and the second feature in the embodiment of this application) are used to represent things, and the edges connecting two nodes (e.g., the first edge and the second edge in the embodiment of this application) are used to indicate that there is such a relationship between the two things.

[0146] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0147] The model training method provided in this application involves image processing and can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (such as multiple first fields of a reference image and multiple second fields of the image to be processed in the model training method provided in this application), ultimately obtaining a trained neural network (such as the field processing model in this application). Furthermore, the image field processing method provided in this application can utilize the trained neural network to input input data (the first field of a reference image and the second field of the image to be classified in the image field processing method provided in this application) into the trained neural network to obtain output data (such as the categories of multiple second fields in this application). It should be noted that the model training method and the image field processing method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of a whole process: such as the model training stage and the model application stage.

[0148] Figure 5This is a flowchart illustrating a method for processing fields in an image according to an embodiment of this application. The method utilizes a field processing model to process multiple first fields of a reference image and multiple second fields of the image to be processed, thereby determining the category of each second field in the image to be processed. For example... Figure 5 As shown, the method includes:

[0149] 501. Obtain multiple first fields from the reference image and multiple second fields from the image to be processed.

[0150] In this embodiment, when a user needs to extract key information from an image to be processed, i.e., needs to determine the category of fields in the image, the user can first obtain the image to be processed and a corresponding reference image. It should be noted that the image to be processed and the reference image are images of the same type of document or ticket. For example, both the image to be processed and the reference image could be images of an ID card, taxi receipts, or even high-speed train tickets, etc. As can be seen, the image to be processed and the reference image contain fields of the same category; therefore, the category of fields in the image to be processed can be determined based on the category of fields in the reference image.

[0151] For ease of explanation, the fields in the reference image will be referred to as the first field, and the fields in the image to be processed will be referred to as the second field. Specifically, the reference image presents multiple first fields and multiple categories, and the correspondence between these multiple first fields and multiple categories is determined, that is, the categories of the multiple first fields are determined. It should be noted that the content indicated by each first field is editable information (or non-fixed information) related to the user, while the content indicated by each category is non-editable information (or fixed information). Figure 6 As shown ( Figure 6 (This is a schematic diagram of a reference image and an image to be processed provided in an embodiment of this application). Let the reference image be an image of an ID card. The image presents categories such as name, gender, date of birth, address, and ID card number. It also presents fields such as Zhang XX, male, 196X year XX month XX day, XX city XX district, and 123XXXXXXX, which indicate editable information of the user (i.e., Zhang XX). The categories of these fields have been pre-annotated, that is, Zhang XX is a field of the name category, male is a field of the gender category, 19XX year XX month XX day is a field of the date of birth category, XX city XX district is a field of the address category, and 123XXXXXXX is a field of the ID card number category.

[0152] The image to be processed also presents multiple second fields and multiple categories, but the correspondence between these multiple second fields and multiple categories is undetermined; that is, the categories of the multiple second fields are undetermined. It should be noted that each second field indicates user-related editable information (or non-fixed information), while each category indicates non-editable information (or fixed information). It is understandable that the multiple categories presented in the reference image and the multiple categories presented in the image to be processed are usually the same, that is, they both contain fields of the same category, but the content indicated by the fields of the same category may differ. As before... Figure 6 In the example shown, suppose the image to be processed is also an image of an ID card. The image presents categories such as name, gender, birthday, address and ID card number, and also presents fields such as Wang X, female, XX month XX day of 195X, XX city XX street, 456XXXXXXX, (4) which indicate editable information of the user (i.e. Wang X), but the categories of these fields are undetermined.

[0153] After obtaining the reference image and the image to be processed, optical character recognition (OCR) technology can be used to process any second field of the image to be processed, thereby obtaining information about the second field, such as the content indicated by the second field and the region where the second field is located. In this way, information about multiple second fields of the image to be processed can be obtained. It should be noted that for any first field in the reference image, the content indicated by the first field and the region where the first field is located can be directly obtained (because the reference image can be understood as a template, so this information is pre-recorded, for example, this information can be recorded in the annotation file of the reference image, etc.), so information about multiple first fields of the reference image can be directly obtained.

[0154] After obtaining information on multiple first fields of the reference image and multiple second fields of the image to be processed, it is equivalent to confirming multiple first fields in the reference image and multiple second fields in the image to be processed.

[0155] It should be understood that for any first field in the reference image, the region where the first field is located refers to the area occupied by the first field in the reference image, such as a rectangular region, etc. Similarly, for any second field in the image to be processed, the region where the second field is located refers to the area occupied by the second field in the image to be processed, such as a rectangular region, etc.

[0156] It should also be understood that this embodiment is only used to illustrate the information of the first field, which includes the content indicated by the first field and the area where the first field is located, and does not limit the type of information contained in the information of the first field in this application.

[0157] 502. Obtain multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields through the field processing model.

[0158] After obtaining information from multiple first fields of the reference image and multiple second fields of the image to be processed, a field processing model can be obtained. The information from multiple first fields of the reference image and multiple second fields of the image to be processed can be processed through the field processing model to obtain multiple first features and multiple second features. The multiple first features correspond one-to-one with the multiple first fields, and the multiple second features correspond one-to-one with the multiple second fields.

[0159] The field processing model is a trained neural network model, such as... Figure 7 As shown ( Figure 7 This is a schematic diagram of the field processing model provided in an embodiment of this application. The model typically comprises two interconnected parts: a first part is a multilayer perceptron (MLP), and the second part is a graph solver built based on a graph matching algorithm, which can be any of the following algorithms: the Hungarian algorithm, the maximal flow algorithm, etc. The input of the MLP serves as the input of the entire field processing model, and the output of the MLP is connected to the input of the graph solver. The output of the graph solver serves as the output of the entire field processing model.

[0160] Specifically, after obtaining information from multiple first fields of the reference image and multiple second fields of the image to be processed, this information can be input into the multilayer perceptron of the field processing model. Then, for any one of the multiple first fields, the multilayer perceptron can process the information of that first field to obtain its first feature. For example, the multilayer perceptron can use the center point of the region containing the first field as its spatial feature, the size (or aspect ratio) of the region containing the first field as its shape feature, and perform feature extraction processing on the content indicated by the first field to obtain its text feature. Then, the multilayer perceptron can perform linear transformation processing on the spatial, shape, and text features of the first field to map them to a higher-dimensional space, obtaining new spatial, shape, and text features for the first field. These three features can then be used as the first feature of the first field. In this way, the multilayer perceptron can obtain multiple first features corresponding one-to-one with the multiple first fields.

[0161] Similarly, for any one of the multiple second fields, the multilayer perceptron can process the information of that second field to obtain its second features. For example, the multilayer perceptron can use the center point of the region containing the second field as its spatial feature, the size of the region as its shape feature, and perform feature extraction on the content indicated by the second field to obtain its text features. Then, the multilayer perceptron can perform linear transformations on the spatial, shape, and text features of the second field to map them to a higher-dimensional space, obtaining new spatial, shape, and text features. These three features can then serve as the second features of the second field. In this way, the multilayer perceptron can obtain multiple second features corresponding one-to-one with the multiple second fields.

[0162] It should be understood that this embodiment is only used to illustrate the inclusion of three features—spatial features, shape features, and text features—in the first feature of the first field, and does not limit the types of features included in the first feature of the first field in this application.

[0163] 503. The field processing model is used to process multiple first features and multiple second features to obtain matching results between multiple first fields and multiple second fields.

[0164] After obtaining multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields, the field processing model can process the multiple first features and multiple second features to obtain the matching results between the multiple first fields and multiple second fields. The matching results include the matching degree between any one of the multiple first fields and any one of the multiple second fields.

[0165] Furthermore, to improve the accuracy of matching results between multiple first fields and multiple second fields, the multilayer perceptron of the field processing model can calculate the pairwise distances between multiple first features (i.e., calculate the distance between any two first features), thereby obtaining multiple first edges. For example, for any two first fields, the multilayer perceptron can calculate the distance between the new spatial features of these two first fields (e.g., calculate the Euclidean distance between these two new spatial features, etc.) and calculate the distance between the new size features of these two first fields (e.g., concatenate these two new size features to obtain the distance between them, etc.). Then, the distance between the new spatial features and the distance between the new size features of these two first fields can be used as the distance between the first features of these two first fields. In this way, among multiple first fields, the multilayer perceptron can obtain the distance between the first features of any two first fields, i.e., multiple first edges.

[0166] Similarly, a multilayer perceptron can also calculate the pairwise distances between multiple second features (i.e., calculate the distance between any two second features), thus obtaining multiple second edges. For example, for any two second fields, the multilayer perceptron can calculate the distance between the new spatial features of these two second fields (e.g., calculate the Euclidean distance between these two new spatial features, etc.) and the distance between the new size features of these two second fields (e.g., concatenate these two new size features to obtain the distance between them, etc.). Then, the distance between the new spatial features and the distance between the new size features of these two second fields can be used as the distance between the second features of these two second fields. In this way, in multiple second fields, the multilayer perceptron can obtain the distance between the second features of any two second fields, i.e., multiple second edges.

[0167] Based on this, a multilayer perceptron can process multiple first features, multiple second features, multiple first edges, and multiple second edges, so that the graph solver can obtain matching results between multiple first fields and multiple second fields based on the processing results of the multilayer perceptron. Specifically, the multilayer perceptron and the graph solver can obtain matching results between multiple first fields and multiple second fields in the following ways:

[0168] (1) The multilayer perceptron processes multiple first features and multiple second features to obtain a first matrix (also called a node affinity matrix). The first matrix contains the similarity (also called affinity) between any first feature and any second feature.

[0169] To further understand the generation process of the first matrix, the following section combines... Figure 8a The process will be described in further detail. Figure 8a A schematic diagram illustrating the generation process of the first matrix provided in this application embodiment, as shown below. Figure 8a As shown, for a new spatial feature of a first field, the multilayer perceptron can calculate the similarity between this new spatial feature of the first field and the new spatial features of each second field, and use the similarity between the new spatial feature of the first field and the new spatial features of each second field as each element of a row in the first spatial matrix. Then, for new spatial features of other first fields, the multilayer perceptron can perform the same process, thus generating a complete first spatial matrix. In the first spatial matrix, each element of a row represents the similarity between a new spatial feature of a first field and the new spatial features of each second field, and each element of a column represents the similarity between a new spatial feature of a second field and the new spatial features of each first field.

[0170] Similarly, for a new size feature in a first field, the multilayer perceptron can calculate the similarity between this new size feature in the first field and the new size features in each second field, and use the similarity between these two new size features as each element of a row in the first size matrix. Then, for new size features in other first fields, the multilayer perceptron can perform the same process, thus generating a complete first size matrix. In this first size matrix, each element in a row represents the similarity between a new size feature in a first field and the new size features in each second field, and each element in a column represents the similarity between a new size feature in a second field and the new size features in each first field.

[0171] Similarly, for a new text feature in a first field, the multilayer perceptron can calculate the similarity between this new text feature in the first field and the new text features in each second field, and use the similarity between the new text feature in the first field and the new text features in each second field as each element in a row of the first text matrix. Then, for new text features in other first fields, the multilayer perceptron can perform the same process, thus generating a complete first text matrix. In the first text matrix, each element in a row represents the similarity between a new text feature in a first field and the new text features in each second field, and each element in a column represents the similarity between a new text feature in a second field and the new text features in each first field.

[0172] After obtaining the first spatial matrix, the first size matrix, and the first character matrix, the multilayer perceptron can perform a weighted summation (matrix addition) on these three matrices to obtain the first matrix. It should be noted that the weights corresponding to each matrix in the first spatial matrix, the first size matrix, and the first character matrix can be set according to actual needs; no limitation is made here.

[0173] (2) The multilayer perceptron processes multiple first edges and multiple second edges to obtain a second matrix (also called the edge affinity matrix). The second matrix contains the similarity (also called affinity) between any first edge and any second edge.

[0174] To further understand the generation process of the second matrix, the following section combines... Figure 8b The process will be described in further detail. Figure 8b A schematic diagram illustrating the generation process of the second matrix provided in this application embodiment, as shown below. Figure 8b As shown, for the distance between two new spatial features of the first field, the multilayer perceptron can calculate the similarity between the distance between these two new spatial features of the first field and the distance between any two new spatial features of the second field, and use the similarity between these two new spatial features of the first field and the distance between any two new spatial features of the second field as each element of a row in the second spatial matrix. Then, for the distance between the new spatial features of the remaining two first fields, the multilayer perceptron can perform the same process, thus generating a complete second spatial matrix.

[0175] For the distance between two new size features in the first field, the multilayer perceptron calculates the similarity between the distance between these two new size features in the first field and the distance between any two new size features in the second field. This similarity is then used as the element of each row in the second size matrix. Similarly, for the distances between the new size features in the remaining two first fields, the multilayer perceptron performs the same process, thus generating a complete second size matrix.

[0176] After obtaining the second spatial matrix and the second size matrix, the multilayer perceptron can perform a weighted summation of these two matrices to obtain the second matrix. It should be noted that the weights corresponding to each matrix in the second spatial matrix and the second size matrix can be set according to actual needs; no limitation is made here.

[0177] (3) After obtaining the first and second matrices, the multilayer perceptron can send them to the graph solver. The graph solver can process the first and second matrices based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields. These matching results include the matching degree between any first field and any second field. For example... Figure 9 As shown ( Figure 9 A schematic diagram of the matching results provided in an embodiment of this application. Figure 9 exist Figure 6 (Based on the above), the matching results show the matching degree between each field in the image to be processed and each field in the reference image. For example, the matching degree between Wang X and Zhang XX is 80%, the matching degree between Wang X and male is 10%, the matching degree between Wang X and 196X year XX month XX day is 10%, the matching degree between Wang X and XX city XX district is 10%, the matching degree between Wang X and 123XXXXXXX is 10%, and so on.

[0178] It should be understood that this embodiment only uses the distance between the first features of the two first fields (i.e., the first side) to illustrate the distance between the new spatial features of the two first fields and the new dimensional features of the two first fields, and does not limit the type of distance included in the distance between the first features of the two first fields in this embodiment. Similarly, the second side is also like this, and will not be described again here.

[0179] It should also be understood that this embodiment is only used to illustrate the generation of the first matrix based on the first spatial matrix, the first size matrix, and the first text matrix, and does not constitute a limitation on the type of matrix used to generate the first matrix in this application. Similarly, the same applies to the second matrix, which will not be described in detail here.

[0180] 504. Based on the matching results and the categories of multiple first fields, determine the categories of multiple second fields.

[0181] After obtaining the matching results output by the field processing model, since the categories of multiple first fields in the reference image are known, the categories of multiple second fields can be determined based on the matching results and the categories of multiple first fields.

[0182] Specifically, the categories among multiple second fields can be obtained in several ways, which will be described below:

[0183] In one possible implementation, among multiple second fields, for any given second field, based on the matching degree between that second field and each first field, the category of the first field with the highest matching degree is determined as the category of that second field. Then, for the other second fields, the same process can be performed, thus determining the category of each of the multiple second fields. As before... Figure 9 In the example shown, since Wang X and Zhang XX have the highest matching degree (80%), and Zhang XX is a field of the name category, Wang X can be identified as a field of the name category.

[0184] In another possible implementation, among multiple second fields, for any given second field, if the matching degree between the second field and each of the first fields is less than a preset threshold, then the second field is considered an abnormal field and removed from the multiple second fields. If the matching degree between the second field and any of the first fields is greater than or equal to the preset threshold, then the second field is considered a normal field and retained. The same process can then be performed on the other second fields. This completes the filtering of multiple second fields, resulting in the remaining second fields, which are all normal fields and no longer contain abnormal fields, thus improving field usability. Figure 9 In the example shown, since the matching degree of (4) with Zhang XX, male, 196X year XX month XX day, XX city XX district, and 123XXXXXXX is less than 20%, (4) is determined to be an abnormal field (for example, (4) is a mark manually marked on the ID card corresponding to the image to be processed, etc.), and (4) is removed from the fields presented in the image to be processed. Since the matching degree of Wang X with Zhang XX is 80%, the matching degree of female with male is 70%, the matching degree of 195X year XX month XX day with 196X year XX month XX day is 80%, the matching degree of XX city XX street with XX city XX district is 70%, and the matching degree of 123XXXXXXX with 456XXXXXXX is 70%, all are greater than 20%, so Wang X, female, 195X year XX month XX day, XX city XX street and 456XXXXXXX are all normal fields.

[0185] Then, among the remaining second fields, for any given second field, based on the matching degree between that second field and each first field, the category of the first field with the highest matching degree is determined as the category of that second field. The same process can then be performed for the other second fields, thus determining the category of each of the remaining second fields. Figure 9 In the example shown, among the remaining fields presented in the image to be processed, Wang X and Zhang XX have the highest matching degree (80%), female and male have the highest matching degree (70%), 195X year XX month XX day and 196X year XX month XX day have the highest matching degree (80%), XX city XX street and XX city XX district have the highest matching degree (70%), and 123XXXXXXX and 456XXXXXXX have the highest matching degree (70%). Therefore, Wang X can be identified as the field of name, female as the field of gender, 195X year XX month XX day as the field of birthday, XX city XX street as the field of address, and 456XXXXXXX as the field of ID card number.

[0186] After determining the categories of multiple second fields, the system can generate extraction results of key information in the image to be processed based on these multiple second fields and their categories, and return the extraction results to the user for use. For example... Figure 10 As shown ( Figure 10 A schematic diagram illustrating the extraction results of key information provided in the embodiments of this application. Figure 10 exist Figure 9 Based on the results of the extraction, it can be seen that the ID card belongs to a woman named Wang X, whose address is XX Street, XX City, whose birthday is XX Month XX Day, 195X, and whose ID card number is 456XXXXXXX.

[0187] Furthermore, the field processing model provided in this application embodiment can be compared with field processing models of related technologies. Specifically, the field processing model provided in this application embodiment and the field processing models of related technologies can be tested using datasets such as d0-d8, and the test results are shown in Table 1:

[0188] Table 1

[0189]

[0190] As can be seen from Table 1, the field processing model provided in this application embodiment can achieve the highest recognition accuracy after performing field recognition on most datasets.

[0191] In this embodiment, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the multiple first fields and multiple second fields are first processed using a field processing model to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the multiple first features and multiple second features are processed using the field processing model to obtain matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0192] Furthermore, the field processing model in this application embodiment introduces a graph matching algorithm. In the process of processing the second field of the image to be processed, the position of the second field is not sensitive. Even if some second fields in the image to be processed drift, the category of each second field can be accurately identified.

[0193] Furthermore, embodiments of this application can also detect abnormal fields in the image to be processed, thereby not assigning any category to the abnormal fields, but only assigning categories to the normal fields, which can further improve the accuracy of field category recognition.

[0194] The above is a detailed description of the image field processing method provided in the embodiments of this application. The model training method provided in the embodiments of this application will be introduced below. Figure 11 This is a schematic flowchart illustrating the model training method provided in an embodiment of this application. Figure 11 As shown, the method includes:

[0195] 1101. Obtain multiple first fields of the reference image and multiple second fields of the image to be processed.

[0196] When training the model to be trained, a batch of training samples can be obtained, namely, information from multiple first fields of the reference image and information from multiple second fields of the image to be processed. The model to be trained consists of two directly connected parts: a multilayer perceptron (MLP) and a graph solver. The input of the MLP serves as the input to the entire model to be trained, and the output of the MLP is connected to the input of the graph solver. The output of the graph solver serves as the output of the entire model to be trained. It is worth noting that the true class of each first field in the reference image is known, and the true class of each second field in the image to be processed is also known.

[0197] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0198] For information about the first and second fields, please refer to [link / reference]. Figure 5 The relevant descriptions of step 501 in the illustrated embodiment will not be repeated here.

[0199] 1102. Obtain multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields through the model to be trained.

[0200] After obtaining information from multiple first fields of the reference image and multiple second fields of the image to be processed, this information can be input into the model to be trained. The model then processes this information to obtain multiple first features and multiple second features, where each first feature corresponds one-to-one with a first field, and each second feature corresponds one-to-one with a second field. Specifically, the first and second features can be generated in various ways:

[0201] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0202] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0203] For the process of obtaining the first and second features, please refer to [reference needed]. Figure 5 The relevant descriptions of step 502 in the illustrated embodiment will not be repeated here.

[0204] 1103. The model to be trained processes multiple first features and multiple second features to obtain matching results between multiple first fields and multiple second fields.

[0205] After obtaining multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields, the model to be trained can process these first and second features to obtain matching results between the multiple first fields and multiple second fields. These matching results include the matching degree between any one of the first fields and any one of the second fields. Specifically, the matching results can be obtained in several ways:

[0206] In one possible implementation, the matching results between multiple first features and multiple second features are obtained by processing multiple first fields and multiple second fields through the model to be trained. This includes: processing multiple first features, multiple second features, multiple first edges, and multiple second edges through the model to be trained to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0207] In one possible implementation, the matching results between multiple first features, multiple second features, multiple first edges, and multiple second edges are processed by the model to be trained to obtain matching results between multiple first fields and multiple second fields. This includes: processing multiple first features and multiple second features by the model to be trained to obtain a first matrix, which contains the similarity between any one first feature and any one second feature; processing multiple first edges and multiple second edges by the model to be trained to obtain a second matrix, which contains the similarity between any one first edge and any one second edge; and processing the first matrix and the second matrix by the model to be trained based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields.

[0208] For the process of obtaining matching results, please refer to [link / reference]. Figure 5 The relevant descriptions of step 503 in the illustrated embodiment will not be repeated here.

[0209] 1104. Based on the matching results and the true categories of multiple first fields, determine the predicted categories of multiple second fields.

[0210] After obtaining the matching results output by the model to be trained, since the true categories of multiple first fields are known, the predicted categories of multiple second fields can be determined based on the matching results and the true categories of the multiple first fields. Specifically, the predicted categories of the second fields can be obtained in several ways:

[0211] In one possible implementation, the matching result includes the matching degree between any first field and any second field. Based on the matching result and the true categories of multiple first fields, determining the predicted categories of multiple second fields includes: for any second field, based on the matching degree between the second field and each first field, determining the true category of the first field with the highest matching degree as the predicted category of the second field.

[0212] In one possible implementation, before determining the true category of the first field with the highest matching degree as the predicted category of the second field based on the matching degree between the second field and each first field, for any second field, the method further includes: for any second field, if the matching degree between the second field and each first field is less than a preset threshold, then the second field is removed from the multiple second fields.

[0213] For the process of obtaining the predicted category of the second field, please refer to [link / reference]. Figure 5 The relevant descriptions of step 504 in the illustrated embodiment will not be repeated here.

[0214] 1105. Based on the true categories and predicted categories of multiple second fields, train the model to be trained to obtain the field processing model.

[0215] After obtaining the predicted categories of multiple second fields, since the true categories of multiple second fields are known, the target loss can be calculated based on the predicted categories and the true categories of multiple second fields. The target loss is used to indicate the difference between the predicted categories and the true categories of multiple second fields. That is, the target loss is used to indicate the difference between the predicted category and the true category of each second field.

[0216] After obtaining the target loss, the model parameters of the model to be trained can be updated based on the target loss, and the updated model can be trained using the next batch of training samples (i.e., re-execute steps 1102 to 1105) until the preset model training conditions are met (e.g., the target loss converges, etc.). Figure 5 The field processing model in the illustrated embodiment.

[0217] The field processing model trained in this embodiment has the ability to classify fields in the image to be processed. Specifically, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the field processing model is first used to process the multiple first fields and multiple second fields to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the field processing model is used to process the multiple first features and multiple second features to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0218] Furthermore, the field processing model trained in this embodiment introduces a graph matching algorithm. In the process of processing the second field of the image to be processed, it is not sensitive to the position of the second field. Even if some second fields in the image to be processed drift, the category of each second field can be accurately identified.

[0219] Furthermore, embodiments of this application can also detect abnormal fields in the image to be processed, thereby not assigning any category to the abnormal fields, but only assigning categories to the normal fields, which can further improve the accuracy of field category recognition.

[0220] The above is a detailed description of the model training method provided in the embodiments of this application. The following will introduce the image field processing device and the model training device provided in the embodiments of this application respectively. Figure 12 This is a schematic diagram of a processing apparatus for fields in an image provided in an embodiment of this application. For example... Figure 12 As shown, the device includes:

[0221] The acquisition module 1201 is used to acquire multiple first fields of the reference image and multiple second fields of the image to be processed;

[0222] The first processing module 1202 is used to obtain multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields through a field processing model.

[0223] The second processing module 1203 is used to process multiple first features and multiple second features through a field processing model to obtain matching results between multiple first fields and multiple second fields;

[0224] The determination module 1204 is used to determine the categories of multiple second fields based on the matching results and the categories of multiple first fields.

[0225] In this embodiment, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the multiple first fields and multiple second fields are first processed using a field processing model to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the multiple first features and multiple second features are processed using the field processing model to obtain matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0226] In one possible implementation, the second processing module 1203 is used to process multiple first features, multiple second features, multiple first edges, and multiple second edges through a field processing model to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0227] In one possible implementation, the second processing module 1203 is configured to: process multiple first features and multiple second features using a field processing model to obtain a first matrix, the first matrix containing the similarity between any first feature and any second feature; process multiple first edges and multiple second edges using a field processing model to obtain a second matrix, the second matrix containing the similarity between any first edge and any second edge; and process the first matrix and the second matrix using a graph matching algorithm based on the field processing model to obtain matching results between multiple first fields and multiple second fields.

[0228] In one possible implementation, the matching result includes the matching degree between any first field and any second field. The determining module 1204 is used to determine the category of the first field with the highest matching degree as the category of the second field for any second field, based on the matching degree between the second field and each first field.

[0229] In one possible implementation, the device further includes a rejection module, which rejects any second field from a plurality of second fields if the matching degree between the second field and each first field is less than a preset threshold.

[0230] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0231] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0232] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0233] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0234] Figure 13 This is a schematic diagram of the model training apparatus provided in an embodiment of this application. Figure 13 As shown, the device includes:

[0235] The acquisition module 1301 is used to acquire multiple first fields of the reference image and multiple second fields of the image to be processed;

[0236] The first processing module 1302 is used to obtain multiple first features corresponding to multiple first fields and multiple second features corresponding to multiple second fields through the model to be trained.

[0237] The second processing module 1303 is used to process multiple first features and multiple second features through the model to be trained, and obtain matching results between multiple first fields and multiple second fields;

[0238] The determination module 1304 is used to determine the predicted categories of multiple second fields based on the matching results and the true categories of multiple first fields;

[0239] Training module 1305 is used to train the model to be trained based on the true categories and predicted categories of multiple second fields to obtain the field processing model.

[0240] The field processing model trained in this embodiment has the ability to classify fields in the image to be processed. Specifically, after obtaining multiple first fields of the reference image and multiple second fields of the image to be processed, the field processing model is first used to process the multiple first fields and multiple second fields to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. Then, the field processing model is used to process the multiple first features and multiple second features to obtain the matching results between the multiple first fields and multiple second fields. Finally, based on the matching results and the categories of the multiple first fields, the categories of the multiple second fields are determined. In the aforementioned process, since the matching results include the matching degree between any first field and any second field, for any second field, the first field that best matches that second field can be determined, and the category of that first field can be used as the category of that second field. In this way, the categories of all second fields in the image to be processed can be accurately identified.

[0241] In one possible implementation, the second processing module 1303 is used to process multiple first features, multiple second features, multiple first edges, and multiple second edges through the model to be trained, to obtain matching results between multiple first fields and multiple second fields, where the first edge is the distance between two first features and the second edge is the distance between two second features.

[0242] In one possible implementation, the second processing module 1303 is configured to: process multiple first features and multiple second features using the model to be trained to obtain a first matrix, the first matrix containing the similarity between any first feature and any second feature; process multiple first edges and multiple second edges using the model to be trained to obtain a second matrix, the second matrix containing the similarity between any first edge and any second edge; and process the first matrix and the second matrix using the model to be trained based on a graph matching algorithm to obtain matching results between multiple first fields and multiple second fields.

[0243] In one possible implementation, the matching result includes the matching degree between any first field and any second field. The determining module 1304 is used to determine the true category of the first field with the highest matching degree as the predicted category of the second field for any second field, based on the matching degree between the second field and each first field.

[0244] In one possible implementation, the device further includes a rejection module, which rejects any second field from a plurality of second fields if the matching degree between the second field and each first field is less than a preset threshold.

[0245] In one possible implementation, the first feature is generated based on at least one of the content indicated by the first field, the center point location of the region where the first field is located, and the size of the region where the first field is located.

[0246] In one possible implementation, the second feature is generated based on at least one of the content indicated by the second field, the center point location of the region where the second field is located, and the size of the region where the second field is located.

[0247] In one possible implementation, the content indicated by the first field and the content indicated by the second field are editable information related to the user.

[0248] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0249] This application also relates to an execution device. Figure 14 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 14 As shown, the execution device 1400 can specifically manifest as a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. Among them, the execution device 1400 may be deployed with... Figure 12 The image field processing device described in the corresponding embodiment is used to implement Figure 5 This corresponds to the function of processing fields in the image in the embodiment. Specifically, the execution device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (wherein the execution device 1400 may have one or more processors 1403). Figure 14 (Taking a processor as an example), processor 1403 may include application processor 14031 and communication processor 14032. In some embodiments of this application, receiver 1401, transmitter 1402, processor 1403 and memory 1404 may be connected via a bus or other means.

[0250] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0251] Processor 1403 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0252] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1403 or by instructions in software form. The processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1404. Processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0253] Receiver 1401 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1402 can be used to output digital or character information through the first interface; transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1402 may also include a display device such as a display screen.

[0254] In one embodiment of this application, the processor 1403 is used to... Figure 5 The field processing model in the corresponding embodiment processes the fields in the image to determine the category of the fields in the image.

[0255] This application also relates to a training device. Figure 15 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 15 As shown, the training device 1500 is implemented by one or more servers. The training device 1500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1514 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1514 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the training device 1500.

[0256] The training device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558; or, one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0257] Specifically, the training equipment can perform Figure 11 The model training method in the corresponding embodiment.

[0258] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0259] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0260] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0261] For details, please refer to Figure 16 , Figure 16 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1600. The NPU 1600 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1603, which is controlled by the controller 1604 to retrieve matrix data from the memory and perform multiplication operations.

[0262] In some implementations, the arithmetic circuit 1603 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1603 is a two-dimensional pulsating array. The arithmetic circuit 1603 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1603 is a general-purpose matrix processor.

[0263] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1602 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1601 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1608.

[0264] Unified memory 1606 is used to store input and output data. Weight data is directly transferred to weight memory 1602 via Direct Memory Access Controller (DMAC) 1605. Input data is also transferred to unified memory 1606 via DMAC.

[0265] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1609.

[0266] The Bus Interface Unit (BIU) 1613 is used by the instruction fetch memory 1609 to fetch instructions from external memory, and also by the memory access controller 1605 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0267] The DMAC is mainly used to move input data from external memory DDR to unified memory 1606, or to weight data to weight memory 1602, or to input data to input memory 1601.

[0268] The vector computation unit 1607 includes multiple processing units that further process the output of the computation circuit 1603 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0269] In some implementations, the vector computation unit 1607 can store the processed output vector in the unified memory 1606. For example, the vector computation unit 1607 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1603, such as linearly interpolating the predicted label plane extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1607 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1603, for example, for use in subsequent layers in the neural network.

[0270] The instruction fetch buffer 1609 connected to the controller 1604 is used to store the instructions used by the controller 1604.

[0271] The unified memory 1606, input memory 1601, weight memory 1602, and instruction fetch memory 1609 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0272] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0273] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0274] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0275] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0276] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for processing fields in an image, characterized in that, The method includes: Obtain multiple first fields from the reference image and multiple second fields from the image to be processed; The field processing model is used to obtain multiple first features corresponding to the multiple first fields and multiple second features corresponding to the multiple second fields. The field processing model processes the plurality of first features and the plurality of second features based on a graph matching algorithm to obtain a matching result between the plurality of first fields and the plurality of second fields. The matching result includes the matching degree between any one of the plurality of first fields and any one of the plurality of second fields. Based on the matching results and the categories of the plurality of first fields, the categories of the plurality of second fields are determined.

2. The method according to claim 1, characterized in that, The first feature is generated based on at least one of the content indicated by the first field, the center point location of the area where the first field is located, and the size of the area where the first field is located.

3. The method according to claim 2, characterized in that, The second feature is generated based on at least one of the content indicated by the second field, the center point location of the area where the second field is located, and the size of the area where the second field is located.

4. The method according to any one of claims 1 to 3, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

5. The method according to any one of claims 1 to 3, characterized in that, The step of processing the plurality of first features and the plurality of second features using the field processing model based on a graph matching algorithm to obtain the matching results between the plurality of first fields and the plurality of second fields includes: The field processing model processes the plurality of first features, the plurality of second features, the plurality of first edges, and the plurality of second edges using a graph matching algorithm to obtain the matching results between the plurality of first fields and the plurality of second fields. The first edge is the distance between two first features, and the second edge is the distance between two second features.

6. The method according to claim 5, characterized in that, The process of processing the plurality of first features, the plurality of second features, the plurality of first edges, and the plurality of second edges using the field processing model based on a graph matching algorithm to obtain the matching results between the plurality of first fields and the plurality of second fields includes: The multiple first features and multiple second features are processed by the field processing model to obtain a first matrix, which contains the similarity between any first feature and any second feature; The field processing model is used to process multiple first edges and multiple second edges to obtain a second matrix, which contains the similarity between any first edge and any second edge. The field processing model processes the first matrix and the second matrix based on a graph matching algorithm to obtain matching results between the plurality of first fields and the plurality of second fields.

7. The method according to any one of claims 1 to 3, characterized in that, The matching result includes the matching degree between any first field and any second field. Determining the categories of the multiple second fields based on the matching result and the categories of the multiple first fields includes: For any second field, based on the matching degree between the second field and each of the first fields, the category of the first field with the highest matching degree is determined as the category of the second field.

8. The method according to claim 7, characterized in that, Before determining the category of the first field with the highest matching degree between any second field and each first field as the category of the second field, the method further includes: If the matching degree between any second field and each of the first fields is less than a preset threshold, then the second field is removed from the plurality of second fields.

9. A model training method, characterized in that, The method includes: Obtain multiple first fields from the reference image and multiple second fields from the image to be processed; The multiple first features corresponding to the multiple first fields and the multiple second features corresponding to the multiple second fields are obtained through the model to be trained. The training model processes the plurality of first features and the plurality of second features using a graph matching algorithm to obtain a matching result between the plurality of first fields and the plurality of second fields. The matching result includes the matching degree between any one of the plurality of first fields and any one of the plurality of second fields. Based on the matching results and the true categories of the multiple first fields, the predicted categories of the multiple second fields are determined; Based on the true categories and predicted categories of the multiple second fields, the model to be trained is trained to obtain a field processing model.

10. The method according to claim 9, characterized in that, The first feature is generated based on at least one of the content indicated by the first field, the center point location of the area where the first field is located, and the size of the area where the first field is located.

11. The method according to claim 10, characterized in that, The second feature is generated based on at least one of the content indicated by the second field, the center point location of the area where the second field is located, and the size of the area where the second field is located.

12. The method according to any one of claims 9 to 11, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

13. The method according to any one of claims 9 to 11, characterized in that, The step of processing the plurality of first features and the plurality of second features using the graph matching algorithm based on the model to be trained to obtain the matching results between the plurality of first fields and the plurality of second fields includes: The training model processes the plurality of first features, the plurality of second features, the plurality of first edges, and the plurality of second edges using a graph matching algorithm to obtain the matching results between the plurality of first fields and the plurality of second fields. The first edge is the distance between two first features, and the second edge is the distance between two second features.

14. The method according to claim 13, characterized in that, The step of processing the plurality of first features, the plurality of second features, the plurality of first edges, and the plurality of second edges using the graph matching algorithm based on the model to be trained to obtain the matching results between the plurality of first fields and the plurality of second fields includes: The multiple first features and multiple second features are processed by the model to be trained to obtain a first matrix, which contains the similarity between any first feature and any second feature. The training model processes multiple first edges and multiple second edges to obtain a second matrix, which contains the similarity between any first edge and any second edge. The training model processes the first matrix and the second matrix using a graph matching algorithm to obtain matching results between the plurality of first fields and the plurality of second fields.

15. A method for processing fields in an image, characterized in that, The method includes: Obtain target instructions, which are used to indicate multiple first fields in a reference image and multiple second fields in an image to be processed; The graph matching algorithm is used to process the plurality of first fields and the plurality of second fields indicated by the target instruction to obtain a recognition result, which is used to indicate the category of the plurality of second fields; The graph matching algorithm processes the plurality of first fields and the plurality of second fields indicated by the target instruction to obtain the recognition result, including: Obtain multiple first features corresponding to the multiple first fields, and multiple second features corresponding to the multiple second fields; The plurality of first features and the plurality of second features are processed to obtain a first matrix, wherein the first matrix contains the similarity between any first feature and any second feature; The first matrix is ​​processed using a graph matching algorithm to obtain the recognition result.

16. The method according to claim 15, characterized in that, The graph matching algorithm processes the first matrix to obtain the following recognition results: The first edge and the second edge are processed to obtain a second matrix. The second matrix contains the similarity between any first edge and any second edge. The first edge is the distance between two first features, and the second edge is the distance between two second features. The first and second matrices are processed using a graph matching algorithm to obtain the recognition result.

17. The method according to claim 15 or 16, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

18. An apparatus for processing fields in an image, characterized in that, The device includes: The acquisition module is used to acquire multiple first fields from the reference image and multiple second fields from the image to be processed; The first processing module is used to obtain, through a field processing model, a plurality of first features corresponding to the plurality of first fields and a plurality of second features corresponding to the plurality of second fields; The second processing module is used to process the plurality of first features and the plurality of second features based on the graph matching algorithm through the field processing model to obtain the matching result between the plurality of first fields and the plurality of second fields. The matching result includes the matching degree between any one of the plurality of first fields and any one of the plurality of second fields. A determination module is used to determine the category of the plurality of second fields based on the matching result and the categories of the plurality of first fields.

19. The apparatus according to claim 18, characterized in that, The first feature is generated based on at least one of the content indicated by the first field, the center point location of the area where the first field is located, and the size of the area where the first field is located.

20. The apparatus according to claim 19, characterized in that, The second feature is generated based on at least one of the content indicated by the second field, the center point location of the area where the second field is located, and the size of the area where the second field is located.

21. The apparatus according to any one of claims 18 to 20, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

22. The apparatus according to any one of claims 18 to 20, characterized in that, The second processing module is used to process the plurality of first features, the plurality of second features, the plurality of first edges and the plurality of second edges based on the graph matching algorithm through the field processing model to obtain the matching results between the plurality of first fields and the plurality of second fields, wherein the first edge is the distance between two first features and the second edge is the distance between two second features.

23. The apparatus according to claim 22, characterized in that, The second processing module is used for: The multiple first features and multiple second features are processed by the field processing model to obtain a first matrix, which contains the similarity between any first feature and any second feature; The field processing model is used to process multiple first edges and multiple second edges to obtain a second matrix, which contains the similarity between any first edge and any second edge. The field processing model processes the first matrix and the second matrix based on a graph matching algorithm to obtain matching results between the plurality of first fields and the plurality of second fields.

24. A model training device, characterized in that, The device includes: The acquisition module is used to acquire multiple first fields from the reference image and multiple second fields from the image to be processed; The first processing module is used to obtain, through the model to be trained, a plurality of first features corresponding to the plurality of first fields and a plurality of second features corresponding to the plurality of second fields. The second processing module is used to process the plurality of first features and the plurality of second features using the model to be trained based on a graph matching algorithm to obtain a matching result between the plurality of first fields and the plurality of second fields. The matching result includes the matching degree between any one of the plurality of first fields and any one of the plurality of second fields. The determination module is used to determine the predicted category of the plurality of second fields based on the matching results and the true categories of the plurality of first fields; The training module is used to train the model to be trained based on the true categories and predicted categories of the plurality of second fields to obtain a field processing model.

25. The apparatus according to claim 24, characterized in that, The first feature is generated based on at least one of the content indicated by the first field, the center point location of the area where the first field is located, and the size of the area where the first field is located.

26. The apparatus according to claim 25, characterized in that, The second feature is generated based on at least one of the content indicated by the second field, the center point location of the area where the second field is located, and the size of the area where the second field is located.

27. The apparatus according to any one of claims 24 to 26, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

28. The apparatus according to any one of claims 24 to 26, characterized in that, The second processing module is used to process the plurality of first features, the plurality of second features, the plurality of first edges and the plurality of second edges using the model to be trained based on a graph matching algorithm, to obtain the matching results between the plurality of first fields and the plurality of second fields, wherein the first edge is the distance between two first features and the second edge is the distance between two second features.

29. The apparatus according to claim 28, characterized in that, The second processing module is used for: The multiple first features and multiple second features are processed by the model to be trained to obtain a first matrix, which contains the similarity between any first feature and any second feature. The training model processes multiple first edges and multiple second edges to obtain a second matrix, which contains the similarity between any first edge and any second edge. The training model processes the first matrix and the second matrix using a graph matching algorithm to obtain matching results between the plurality of first fields and the plurality of second fields.

30. An apparatus for processing fields in an image, characterized in that, The device includes: The acquisition module is used to acquire target instructions, which are used to indicate multiple first fields in the reference image and multiple second fields in the image to be processed; The processing module is used to process the plurality of first fields and the plurality of second fields indicated by the target instruction based on a graph matching algorithm to obtain a recognition result, wherein the recognition result is used to indicate the category of the plurality of second fields; The processing module is used for: Obtain multiple first features corresponding to the multiple first fields, and multiple second features corresponding to the multiple second fields; The plurality of first features and the plurality of second features are processed to obtain a first matrix, wherein the first matrix contains the similarity between any first feature and any second feature; The first matrix is ​​processed using a graph matching algorithm to obtain the recognition result.

31. The apparatus according to claim 30, characterized in that, The processing module is used for: The first edge and the second edge are processed to obtain a second matrix. The second matrix contains the similarity between any first edge and any second edge. The first edge is the distance between two first features, and the second edge is the distance between two second features. The first and second matrices are processed using a graph matching algorithm to obtain the recognition result.

32. The apparatus according to claim 30 or 31, characterized in that, The content indicated by the first field and the content indicated by the second field are editable information related to the user.

33. An apparatus for processing fields in an image, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the processing device for fields in the image performs the method as described in any one of claims 1 to 17.

34. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 17.

35. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 17.

Images