Text classification method and text classification model training method and device
By combining text feature extraction network and graph convolution network, the problem of insufficient combination of image features and structural features in text classification in the prior art is solved, and higher detection accuracy and structured information extraction effect are achieved.
Patent Information
- Application Number
- CN202210154579.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-02-18
AI Technical Summary
The prior art is difficult to effectively combine image features and structural features in text classification, resulting in insufficient detection accuracy, especially when processing complex document image.
A text classification method is adopted to combine text feature extraction networks and graph convolution networks to extract image features and structural features, and input them into category prediction networks for training, comprehensively considering image information, semantic information and layout information to improve classification accuracy .
It significantly improves the detection accuracy of text classification, can more accurately identify and classify text fields in images, and provides a better foundation for structured information extraction and storage.
Smart Images

Figure CN114495113B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of text recognition, deep learning and image processing, and can be applied to scenarios such as extracting document information. More specifically, it relates to a text classification method and a training method, device, electronic device and storage medium for a text classification model. Background Art
[0002] With the development of computer technology and network technology, deep learning technology has been widely used in many fields. For example, deep learning technology can be used to extract documents included in images to realize electronic storage of documents. Summary of the invention
[0003] The present disclosure aims to provide a text classification method and a training method, device, electronic device and storage medium for improving detection accuracy.
[0004] According to one aspect of the present disclosure, a text classification method is provided, comprising: determining text features of each of a plurality of fields based on image features of an image to be processed and a plurality of fields included in the image to be processed; determining structural features of each field based on a plurality of text features of the plurality of fields and a plurality of position information of the plurality of fields in the image to be processed; and determining a category of each field based on the text features of each field and the structural features of each field.
[0005] According to another aspect of the present disclosure, a training method for a text classification model is provided, wherein the text classification model includes a text feature extraction network, a graph convolution network and a category prediction network; the training method includes: inputting image features of a sample image and multiple fields included in the sample image into the text feature extraction network to obtain text features of each of the multiple fields; the sample image also includes first information indicating the actual category of each field; according to multiple text features of the multiple fields and multiple position information of the multiple fields in the image to be processed, using a graph convolution network to determine graph features for the multiple fields, the graph features including structural features of each field; inputting multiple text features of the multiple fields and multiple structural features of the multiple fields into the category prediction network to obtain a predicted category of each field; and training the text classification model according to the predicted category and the actual category.
[0006] According to another aspect of the present disclosure, a text classification device is provided, including: a first text feature extraction module, used to determine the text features of each of a plurality of fields according to the image features of the image to be processed and the plurality of fields included in the image to be processed; a structural feature determination module, used to determine the structural features of each field according to the plurality of text features of the plurality of fields and the position information of the plurality of fields in the image to be processed; and a category determination module, used to determine the category of each field according to the text features of each field and the structural features of each field.
[0007] According to another aspect of the present disclosure, a training device for a text classification model is provided, wherein the text classification model includes a text feature extraction network, a graph convolution network and a category prediction network; the training device includes: a second text feature extraction module, used to input the image features of a sample image and multiple fields included in the sample image into the text feature extraction network to obtain text features of each of the multiple fields; the sample image also includes first information indicating the actual category of each field; a graph feature determination module, used to determine the graph features for the multiple fields using the graph convolution network according to multiple text features of the multiple fields and position information of the multiple fields in the image to be processed, the graph features including structural features of each field; a category prediction module, used to input multiple text features of the multiple fields and multiple structural features of the multiple fields into the category prediction network to obtain predicted categories of each field; and a first training module, used to train the text classification model according to the predicted categories and the actual categories.
[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text classification method and / or the text classification model training method provided by the present disclosure.
[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the text classification method and / or the text classification model training method provided by the present disclosure.
[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the text classification method and / or the text classification model training method provided by the present disclosure.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0013] Figure 1 It is a schematic diagram of an application scenario of a text classification method and / or a training method and device of a text classification model according to an embodiment of the present disclosure;
[0014] Figure 2 is a flowchart of a method for training a text classification model according to an embodiment of the present disclosure;
[0015] Figure 3 is a schematic diagram of the principle of determining text features of each of a plurality of fields according to an embodiment of the present disclosure;
[0016] Figure 4 is a schematic diagram of the principle of determining the structural features of each field according to an embodiment of the present disclosure;
[0017] Figure 5 is a flowchart of a method for training a text classification model according to an embodiment of the present disclosure;
[0018] Figure 6 is a schematic diagram of the principle of training a text classification model according to an embodiment of the present disclosure;
[0019] Figure 7 is a structural block diagram of a text classification device according to an embodiment of the present disclosure;
[0020] Figure 8 is a structural block diagram of a training device for a text classification model according to an embodiment of the present disclosure; and
[0021] Fig. 9 It is a block diagram of an electronic device used to implement the text classification method and / or the training method of the text classification model of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0023] The present disclosure provides a text classification method, which includes a text feature extraction stage, a structural feature extraction stage and a category determination stage. In the text feature extraction stage, the text feature of each of the multiple fields is determined according to the image feature of the image to be processed and the multiple fields included in the image to be processed. In the structural feature extraction stage, the structural feature of each field is determined according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the image to be processed. In the category determination stage, the category of each field is determined according to the text feature of each field and the structural feature of each field.
[0024] The following will be combined Figure 1 The application scenarios of the method and device provided by the present disclosure are described.
[0025] Figure 1 It is a schematic diagram of application scenarios of the text classification method and the training method and device of the text classification model according to the embodiments of the present disclosure.
[0026] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110, which may be various electronic devices with processing functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, servers, and the like.
[0027] The electronic device 110, for example, can perform text detection on the input image 120 to obtain multiple fields. Subsequently, the electronic device 110 can also classify the multiple fields and store the multiple fields according to the classification results to achieve electronic storage and structured storage of text in the image 120. Specifically, the electronic device 110 can use text recognition technology (such as optical character recognition technology OCR, etc.) to detect text to obtain multiple fields. The electronic device 110, for example, can classify the fields by performing semantic recognition on the multiple fields. Alternatively, the electronic device 110 can classify the fields by combining the image information of the image 120 and the semantic information of the multiple fields.
[0028] In one embodiment, the electronic device 110 may use the identified multiple fields and their respective categories as the detection result 130. Alternatively, the electronic device may pair the multiple fields according to their categories and use the key-value pairs formed by the pairing as the detection result 130. The electronic device 110 may also store the detection result.
[0029] In one embodiment, the electronic device 110 may, for example, use a text classification model 140 to classify multiple fields. For example, the text classification model 140 may be trained by a server 150. The electronic device 110 may be connected to the server 150 through a network to send a model acquisition request to the server 150. Accordingly, the server 150 may send the trained text classification model 140 to the electronic device 110 in response to the request.
[0030] In one embodiment, the electronic device 110 may also send the input image 120 to the server 150 , and the server 150 detects fields in the image 120 and classifies the detected fields based on the trained text classification model 140 .
[0031] In one embodiment, the text classification model can be constructed based on any one of the following detection methods: a method based on template matching, a method based on field search. Among them, the method based on template matching refers to presetting a layout image as a template based on experience, and setting the position where information needs to be extracted for each template, and then matching the image to be detected with each template, and detecting the text in the image to be detected based on the matching result to obtain the detection result. The method based on field search refers to enumerating the fields to be extracted based on experience, and setting the paragraph position where the field to be extracted may appear; then searching the field identified by the text recognition technology based on the enumerated fields and the set paragraph position.
[0032] In one embodiment, the text classification model can be constructed based on any one of the following models: a target detection model, a natural language processing (NLP) model, a graph convolutional network model, etc. Among them, the text classification model constructed based on the target detection model can detect text areas in the image to be processed, classify the text areas, and then use text recognition technology to recognize the text in each text area. The text classification model constructed based on the natural language processing model can convert the recognition result obtained by the text recognition technology to identify the image to be processed into a document, and then sequence the document. The text classification model constructed based on the graph convolutional network model can send the image to be processed and the recognition result obtained by the text recognition technology to identify the image to be processed into the graph convolutional network, and input the result output by the graph convolutional network into the recurrent network, etc., to sequence the text in the image to be processed.
[0033] It should be noted that the text classification method provided by the present disclosure can be executed by the electronic device 110 or by the server 150. Accordingly, the text classification device provided by the present disclosure can be set in the electronic device 110 or in the server 150. The training method of the text classification model provided by the present disclosure can be executed by the server 150. Accordingly, the training device of the text classification model provided by the present disclosure can be set in the server 150.
[0034] It should be understood that Figure 1 The number and type of the electronic devices 110 and the server 150 in the embodiment are only for illustration. According to the implementation requirements, there may be any number and type of electronic devices 110 and the server 150.
[0035] The following will be combined Figure 1 , through the following Figure 2 to Figure 4 The text classification method provided by the present disclosure is described in detail.
[0036] like Figure 2 As shown, the text classification method 200 of this embodiment may include operations S210 to S230.
[0037] In operation S210, a text feature of each of the plurality of fields is determined according to an image feature of the image to be processed and a plurality of fields included in the image to be processed.
[0038] According to an embodiment of the present disclosure, the image to be processed may be, for example, any image including text. The image may be obtained, for example, by screenshot, or by photographing a paper document. The paper document may be, for example, a bill, a receipt, a test paper, a book, etc., which is not limited by the present disclosure. In this embodiment, a text recognition technology may be used to perform text recognition on the image to be processed, thereby obtaining multiple fields included in the image to be processed and the position information of each of the multiple fields.
[0039] According to an embodiment of the present disclosure, the image to be processed may be input into an image feature extraction network to obtain image features of the image to be processed. Alternatively, the image features may be extracted using a Histogram of Oriented Gradients (HOG) algorithm, a Local Binary Pattern (LBP) operator, etc. The image feature extraction network may include, for example, a convolutional neural network, etc., which is not limited in the present disclosure.
[0040] This embodiment can fuse the image features and the semantic features of the field to obtain the text features of each field. For example, the image features and the semantic features of each field can be spliced, and the spliced features are used as the text features of each field. It can be understood that for each of the multiple fields, a text feature can be obtained, and for the multiple fields, a plurality of text features corresponding to the multiple fields can be obtained in total.
[0041] In operation S220, a structural feature of each field is determined according to a plurality of text features of the plurality of fields and a plurality of position information of the plurality of fields in the image to be processed.
[0042] According to an embodiment of the present disclosure, the position information can be represented by the position of the bounding box surrounding each field in the image to be processed. This embodiment can use the position information as reference information to extract the structural features of the field from multiple text features. The reason why structural features can be extracted from text features is that the text features are integrated with image features corresponding to the fields, and the image features can reflect the structure of the characters in the field to a certain extent. It can be understood that for each of the multiple fields, a piece of position information can be obtained through the aforementioned operation S210, and for multiple fields, a plurality of position information corresponding to the multiple fields can be obtained in total.
[0043] In one embodiment, a graph convolutional network may be used to extract the structural features of each field. Specifically, the embodiment may input the location information and text features into a graph convolutional network (GCN), and output the structural features of each field via the graph convolutional network.
[0044] In operation S230, a category of each field is determined according to a text feature of each field and a structural feature of each field.
[0045] According to an embodiment of the present disclosure, the text features and structural features can be spliced and input into a category prediction network composed of a recurrent neural network and a conditional random field network, and the category prediction network outputs the category of each field. Among them, the recurrent neural network can adopt a bidirectional long short-term memory network (Bi-directional Long-Short Term Memory, Bi-LSTM), etc., which is not limited by the present disclosure.
[0046] According to an embodiment of the present disclosure, the category of each field may be, for example, any one of a plurality of pre-set categories, and the plurality of pre-set categories may include, for example, time, place, person, etc. The plurality of categories may also, for example, reflect which type of data each field is in the form of a key-value pair. The categories of data in the form of a key-value pair include a key category (Key) and a value category (Value). It is understandable that the plurality of pre-set categories may be set according to the actual scenario, and the present disclosure does not limit this.
[0047] The text classification method of the disclosed embodiment comprehensively considers the text features and structural features of the field when classifying the field. Since the text features are obtained based on the field and image features, and the structural features are obtained based on the text features and the location information of the field, the classification result of the field is obtained by comprehensively considering the image information, semantic information and layout information, which can improve the accuracy of the classification result to a certain extent and improve the accuracy of identifying the field in the image to be processed. When the image to be processed is a photographed image of a receipt, etc., the text classification method of this embodiment can provide better basic information for the extraction and storage of structured information.
[0048] Figure 3 It is a schematic diagram of the principle of determining the text feature of each of multiple fields according to an embodiment of the present disclosure.
[0049] According to the embodiments of the present disclosure, Figure 3 As shown, when determining the text features of each field, the embodiment 300 can first determine the image sub-features 321 for each field in the image features 320 according to the position information 310 of each field in the image to be processed. Then, the image sub-features 321 are fused with the text embedding features 330 of each field to obtain the text features of each field.
[0050] Exemplarily, the image feature 320 may be obtained by performing a convolution operation on the image to be processed 301, and there is a mapping relationship between the obtained image feature 320 and the image to be processed 301. This embodiment may determine the image feature corresponding to the pixel at the position indicated by the position information in the image to be processed 301 based on the mapping relationship, and use the corresponding image feature as the image sub-feature 321 for the field. Alternatively, this embodiment may sample the image feature 320 according to the position information to obtain the image sub-feature 321 for the field. Specifically, based on the coordinates of the bounding box indicated by the position information, a regional pooling (ROI Pooling) method may be used to align the bounding box surrounding the field in the image to be processed 301 with the image feature 320, and use the partial image features aligned with the bounding box surrounding the field as the image sub-feature 321 for the field.
[0051] Exemplarily, semantic recognition can be performed on each field to obtain text embedding features 330. A semantic recognition model can be used to perform semantic recognition on the field. The semantic recognition model can include, for example, a convolutional recurrent neural network model (CRNN), etc., which is not limited in the present disclosure.
[0052] Exemplarily, the text embedding feature 330 of each field and the image sub-feature 321 for each field may be concatenated, and the concatenated feature may be used as the text feature of each field.
[0053] In one embodiment, the text feature 340 of each field can also be extracted from the spliced feature obtained by splicing the image sub-feature 321 and the text embedded feature 330. For example, in this embodiment, the spliced feature can be input into a feature extraction network, and the text feature 340 of each field can be extracted through the feature extraction network. In this way, the image sub-features and text embedded features in the spliced feature can be fully learned to improve the accuracy of the obtained text features.
[0054] For example, the feature extraction network can use a self-attention model to fully learn the important features in the spliced features and ignore the unimportant features, thereby improving the processing efficiency while improving the accuracy of the extracted text features. Figure 3 As shown, the self-attention model can adopt the Transformer model 302, etc., and the present disclosure does not limit this.
[0055] Figure 4 It is a schematic diagram of the principle of determining the structural characteristics of each field according to an embodiment of the present disclosure.
[0056] According to an embodiment of the present disclosure, when determining the structural features of each field, for example, the similarity between the text features of multiple fields and the association relationship between the location information of multiple fields can be considered. In this way, the association relationship between the fields can be fully learned when determining the structural features of the fields, so that the determined structural features can better reflect the characteristics of the fields. Among them, the similarity between the text features can be reflected by the differences between the text features, for example.
[0057] In one embodiment, when determining the structural features of each field, a difference feature representing the difference between the multiple text features can be determined based on the multiple text features of the multiple fields. At the same time, a relationship feature representing the relative relationship between the multiple position information can be determined based on the multiple position information of the multiple fields in the image to be processed. Finally, the structural features of each field are determined based on the difference feature and the relationship feature.
[0058] Exemplarily, the number of the plurality of text features is set to n. When determining the difference feature, the difference between the text feature and each of the n text features can be determined for each text feature, and n differences can be obtained. For the n text features, n*n differences can be obtained. This embodiment can determine the difference feature based on the n*n differences. This embodiment can also arrange the aforementioned n*n differences into a matrix form.
[0059] Exemplarily, the number of multiple text features is set to n. When determining the difference features, the n text features can also be first organized into a matrix form to obtain a feature matrix. Each element in the feature matrix represents a text feature of a field. For example, the n text features can be organized into a feature vector, and then the feature vector is copied into n copies, and the n feature vectors are arranged in sequence to form a feature matrix. Subsequently, a transposition operation can be performed on the feature matrix to obtain a transposed matrix of the feature matrix. Finally, the difference features are determined based on the difference between the feature matrix and the transposed matrix. Among them, the difference between the feature matrix and the transposed matrix is in matrix form, which is used to represent the difference between the n text features.
[0060] For example, the matrix representing the differences between the n text features may be normalized to obtain the difference features. Alternatively, the matrix representing the differences may be processed using a fully connected layer, and the processed features may be normalized to obtain the difference features.
[0061] Exemplarily, taking the case where the position information is represented by the position of the bounding box of the enclosing field in the image to be processed as an example, when determining the relational features characterizing the relative relationship between multiple positional information, at least one of the following parameters can be determined first: the lateral distance between multiple bounding boxes enclosing multiple fields, the longitudinal distance between multiple bounding boxes, the aspect ratio of each of the multiple bounding boxes, the area ratio between multiple bounding boxes, etc. Subsequently, these parameters can be characterized to obtain associated features. Among them, when these parameters are characterized, the features after the characterization process can be normalized, and the normalized features can be used as associated features. It can be understood that the above-mentioned parameters and the method for determining the relational features are only used as examples to facilitate the understanding of the present disclosure, and the present disclosure does not limit this.
[0062] It can be understood that when the number of the multiple text features is n, the number of the multiple bounding boxes is also n. For the bounding box A, n sets of parameters can be obtained through the method described above, and each set of parameters represents the association relationship between the bounding box A and a certain bounding box (for example, bounding box B) among the multiple bounding boxes. After characterizing each set of parameters, the obtained feature is the association feature between the bounding box A and the bounding box B.
[0063] According to the embodiments of the present disclosure, Figure 4 As shown, when determining the structural features, the embodiment 400 can input the n text features 401 into the difference extraction network 410, and obtain the difference features after being processed by the difference extraction network 410. At the same time, the n position information 402 can be input into the relationship extraction network 420, and obtain the relationship features after being processed by the relationship extraction network 420. It can be understood that the structures of the difference extraction network 410 and the relationship extraction network 420 can be set according to actual needs to respectively realize the functions of extracting difference features and association features. For example, the difference extraction network 410 may include a transposition layer, a difference calculation layer, a fully connected layer, and a normalization layer. The relationship extraction network 420 may include a parameter extraction layer, a parameter characterization layer, and a dimension mapping layer. Among them, the dimension mapping layer is used to map the features obtained by the characterization process to the same size as the difference features. In one implementation, when the difference extraction network 410 processes the n text features, the n text features can be first processed by a convolution layer and / or a fully connected layer, so that the difference extraction network 410 performs nonlinear learning on the n text features. Subsequently, the n features processed by the convolutional layer and / or the fully connected layer are organized into a matrix form, and the difference features are obtained by using the method for determining the difference features described above.
[0064] In one embodiment, after obtaining the difference feature and the relationship feature, the embodiment can use the difference feature as a weight to weight the relationship feature to obtain a weighted feature. If the n text features 401 described above are respectively used as n graph nodes X_0~X_n, and the weighted features are used to represent the connecting edges between the n graph nodes, a connected graph 403 can be constructed. This embodiment can use a graph convolutional network GCN 430 to iteratively update the connected graph 403 to obtain a connected graph 404 that meets the convergence condition. The features represented by each node in the connected graph 404 that meets the convergence condition are the structural features of each field.
[0065] In actual processing, the weighted features and multiple text features can be input into the graph convolutional network GCN 440, and after being processed by the graph convolutional network GCN 440, the structural features of each field in the multiple fields are obtained.
[0066] This embodiment obtains difference features based on text features that take image features into consideration, and weights relationship features based on the difference features, so that the weighted relationship features can more accurately express the layout relationship between multiple fields. In this way, the extracted structural features can be more accurate, and thus it is easy to improve the accuracy of the obtained field category, which is conducive to improving the accuracy of structured information extraction.
[0067] In order to facilitate the implementation of the text classification method provided by the present disclosure, the present disclosure also provides a training method for a text classification model. Figure 5 The training method is described in detail.
[0068] Figure 5 It is a flowchart of a method for training a text classification model according to an embodiment of the present disclosure.
[0069] like Figure 5 As shown, the text classification model training method 500 of this embodiment may include operations S510 to S540. The text classification model may include a text feature extraction network, a graph convolution network, and a category prediction network.
[0070] In operation S510, image features of a sample image and a plurality of fields included in the sample image are input into a text feature extraction network to obtain text features of each of the plurality of fields.
[0071] According to an embodiment of the present disclosure, the text feature extraction network may include a semantic extraction network (the semantic recognition model described above may be used). The semantic extraction network is used to extract semantic features of multiple fields as text embedding features. The implementation of operation S510 is similar to the implementation of operation S210 described above, and will not be repeated here.
[0072] In one embodiment, the text feature extraction network may include a feature segmentation network and a feature extraction network in addition to the semantic extraction network. The input of the feature segmentation network includes image features and position information of multiple fields in the sample image. The feature segmentation network is used to segment image sub-features for each of the multiple fields from the image features according to the position information. The input of the feature extraction network is the concatenated features obtained by concatenating the image sub-features for each field and the text embedding features of each field, and the output is the text features of each field.
[0073] In operation S520, a graph convolutional network is used to determine graph features for the multiple fields according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the image to be processed.
[0074] According to an embodiment of the present disclosure, the implementation of operation S520 is similar to the implementation of operation S220 described above, and will not be repeated here.
[0075] In one embodiment, the text classification model may also include the relationship extraction network described above and the difference extraction network described above. In operation S520, multiple text features may be first input into the difference extraction network to obtain difference features that characterize the differences between the multiple text features. At the same time, multiple position information of multiple fields in the sample image may be input into the relationship extraction network to obtain relationship features that characterize the relative relationship between the multiple position information. Finally, based on the difference features and the relationship features, a graph convolutional network is used to determine the graph features for the multiple fields. The graph features include the structural features of each field. It is understandable that the present disclosure does not limit the order in which the relationship features and the difference features are obtained. It is understandable that the graph features may be represented by the connected graph that satisfies the convergence conditions described above.
[0076] For example, this embodiment can weight the relationship features with the difference features as weights to obtain weighted features. Finally, the weighted features are input into the graph convolutional network to obtain graph features for multiple fields. The implementation principle of this processing method is the same as that of the previous Figure 4 The principles described are similar and will not be repeated here.
[0077] In operation S530, a plurality of text features of a plurality of fields and a plurality of structural features of a plurality of fields are input into a category prediction network to obtain a predicted category of each field.
[0078] According to an embodiment of the present disclosure, the category prediction network can be composed of the recurrent neural network and conditional random field network described above. The prediction category of each field can be any one of the multiple pre-set categories described above. The implementation method of operation S530 is similar to the implementation method of operation S230 described above, and will not be repeated here. It can be understood that the category prediction network can also be composed of a recurrent neural network and a softmax logistic regression layer, for example, and the present disclosure does not limit this.
[0079] In operation S540 , a text classification model is trained according to the predicted category and the actual category.
[0080] According to an embodiment of the present disclosure, the sample image may include first information indicating the actual category of each field. Operation S540 may determine the loss of the text classification model according to the difference between the predicted category and the actual category. A back propagation algorithm is used to adjust the network parameters in the text classification model to minimize the loss of the text classification model and complete the training of the text classification model.
[0081] For example, the loss of the text classification model can be determined by using a cross entropy loss function or a negative log-likelihood (NLL) loss function, and the present disclosure does not limit this. For example, crf_loss is used to represent the value of the negative log-likelihood loss, and the value can be calculated by the following formula (1):
[0082]
[0083] in, is the input sequence of the recurrent neural network. For the predicted category y, The score can be expressed as For a given input sequence The conditional probability output by the category prediction network for category y.
[0084] Figure 6 It is a schematic diagram of the principle of training a text classification model according to an embodiment of the present disclosure.
[0085] According to an embodiment of the present disclosure, when training a text classification model, a relationship prediction network can be added to the text classification model to predict the association relationship between fields based on the relationship features in the graph features output by the graph convolutional network, and the graph convolutional network can be trained using the actual association relationship as supervision information, thereby making the training of the graph convolutional network more targeted and tending to learn the structured information of the fields.
[0086] like Figure 6 As shown, in this embodiment 600, the text classification model may include a text feature extraction network, a difference extraction network 620, a relationship extraction network 630, a graph convolution network GCN 640, a category prediction network 650 and a relationship prediction network 660. Among them, the text feature extraction network may include a semantic extraction network 611, a feature segmentation network 612 and a feature extraction network 613.
[0087] When training a text classification model, text recognition technology can be used to perform text recognition on a sample image first, thereby obtaining multiple fields included in the sample image and location information of each field in the multiple fields. For example, it can be represented by a field layout diagram 601, where each bounding box in the layout diagram 601 surrounds a field.
[0088] After obtaining the layout diagram 601, the embodiment 600 can input multiple fields into a semantic extraction network 611 in a sequence form, thereby obtaining semantic features of multiple fields. At the same time, the image features of the sample image and multiple position information of multiple fields can be input into a feature segmentation network 612, thereby obtaining image sub-features for each field. The image sub-features and semantic features are spliced and input into a feature extraction network 613, and the text features of each field can be obtained, thereby obtaining a text feature sequence 602.
[0089] After obtaining the text feature sequence 602, the text feature sequence 602 can be input into the difference extraction network 620, and the difference extraction network 620 processes to obtain the difference features described above. Multiple position information is input into the relationship extraction network 630, and the relationship features described above can be obtained after being processed by the relationship extraction network 630. The relationship features are weighted with the difference features as weights to obtain weighted features. The text features in the text feature sequence 602 are respectively used as graph nodes X_0~X_n, and the weighted features are used as the connecting edges between n graph nodes, so that a connected graph 603 can be constructed. After the connected graph 603 is iteratively updated by GCN 640, a connected graph 604 that meets the convergence condition is obtained, and the connected graph 604 can be used as a graph feature for multiple fields. The multiple graph nodes 604_1 in the connected graph 604 can represent the structural features of multiple fields, and the edges 604_2 in the connected graph 604 can represent the relationship features between multiple fields.
[0090] After obtaining the connectivity graph 604, the features represented by the multiple graph nodes 604_1 can be used as the structural features of the multiple fields, thereby obtaining a structural feature sequence 605. The structural feature sequence 605 and the text feature sequence 602 are spliced and input into the category prediction network 650 to obtain the predicted category of each of the multiple fields. At the same time, the relationship features represented by the edges 604_2 can be input into the relationship prediction network 660 to obtain the predicted relationship information between the multiple fields.
[0091] In one embodiment, the sample image may further include second information indicating the relationship between the multiple fields. After obtaining the predicted category and the predicted relationship information, the embodiment 600 may calculate the loss value crf_loss of the category prediction loss according to the predicted category and the actual category included in the sample image using the formula (1) described above. At the same time, the loss value ce_loss of the relationship prediction loss may be determined according to the predicted relationship information and the second information. Subsequently, the text classification model may be trained according to the two-part loss.
[0092] Exemplarily, the loss value ce_loss of the relationship prediction loss can be calculated using a cross entropy loss function. In this embodiment, the weighted sum of the two parts of the loss can be used as the total loss of the text classification model, and the network parameters in the text classification model can be adjusted through the back propagation algorithm to minimize the total loss and complete the training of the text classification model. It can be understood that the weights used when calculating the weighted sum of the two parts of the loss can be set according to actual needs, and the present disclosure does not limit this.
[0093] Exemplarily, the relationship prediction network 660 may be composed of a fully connected layer and a softmax logistic regression layer, or may be composed of any network structure that can be used for classification, which is not limited in the present disclosure.
[0094] In one embodiment, the text classification model can also be trained in a self-supervisory manner according to the difference features, so that the difference extraction network can learn the differences between the text features more accurately. Specifically, the self-supervisory loss of the text classification model can be determined according to the difference features and multiple text features. Then, the text classification model is trained according to the self-supervisory loss.
[0095] For example, this embodiment can use the weighted sum of the self-supervised loss and the two parts of the loss described above as the total loss of the text classification model, and then train the text classification model according to the total loss. Alternatively, this embodiment can use the category prediction loss as the global loss to train the entire text classification model. At the same time, the relationship prediction loss is used as the local loss to train the GCN 640, the relationship extraction network 630 and the difference extraction network 620. The self-supervised loss is used as the local loss to train the difference extraction network 620, and the present disclosure does not limit this.
[0096] For example, the number of multiple fields is set to n, and the size of the difference feature is n×n. The value of the element in the i-th row and j-th column of the difference feature A is A. ij Representing the difference between the text features of the i-th field and the j-th field in the n fields, the value of the self-supervisory loss gl_loss can be expressed by the following formula (2):
[0097]
[0098] Among them, ||A|| F represents the F-norm of the difference feature A, v i 、v j They correspond to the text features of the i-th field and the text features of the j-th field, respectively. For example, they can be obtained by processing the text features of the i-th field and the text features of the j-th field through the convolutional layer and / or the fully connected layer described above. η and γ are weight parameters set according to actual needs.
[0099] It is understandable that the above calculation formula of the self-supervision loss is only used as an example to facilitate understanding of the present disclosure, and the present disclosure does not limit this. As long as the self-supervision loss can reflect the difference represented by the difference feature and the relationship between the differences calculated according to multiple text features.
[0100] Based on the text classification method provided by the present disclosure, the present disclosure also provides a text classification device. Figure 7 The device is described in detail.
[0101] Figure 7 It is a structural block diagram of a text classification device according to an embodiment of the present disclosure.
[0102] like Figure 7 As shown, the text classification device 700 of this embodiment may include a first text feature extraction module 710 , a structural feature determination module 720 and a category determination module 730 .
[0103] The first text feature extraction module 710 is used to determine the text feature of each of the multiple fields according to the image feature of the image to be processed and the multiple fields included in the image to be processed. In one embodiment, the first text feature extraction module 710 can be used to perform the operation S210 described above, which will not be repeated here.
[0104] The structural feature determination module 720 is used to determine the structural feature of each field according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the image to be processed. In one embodiment, the structural feature determination module 720 can be used to perform the operation S220 described above, which will not be repeated here.
[0105] The category determination module 730 is used to determine the category of each field according to the text features of each field and the structural features of each field. In one embodiment, the category determination module 730 can be used to perform the operation S230 described above, which will not be described in detail here.
[0106] According to an embodiment of the present disclosure, the structural feature determination module 720 may include a first difference feature determination submodule, a first relationship feature determination submodule and a structural feature determination submodule. The first difference feature determination submodule is used to determine a difference feature representing the difference between multiple text features based on multiple text features. The first relationship feature determination submodule is used to determine a relationship feature representing the relative relationship between multiple position information based on multiple position information of multiple fields in the image to be processed. The structural feature determination submodule is used to determine the structural features of each field based on the difference feature, the relationship feature and the multiple text features.
[0107] According to an embodiment of the present disclosure, the difference feature determination submodule may include a feature matrix determination unit and a difference feature determination unit. The feature matrix determination unit is used to determine a feature matrix composed of multiple text features. The difference feature determination unit is used to determine the difference feature according to the difference between the feature matrix and the transposed matrix of the feature matrix.
[0108] According to an embodiment of the present disclosure, the structural feature determination submodule may include a first weighting unit and a first feature determination unit. The weighting unit is used to weight the relationship feature with the difference feature as the weight to obtain the weighted feature. The feature determination unit is used to input the weighted feature and multiple text features into the graph convolutional network to obtain the structural feature of each field.
[0109] According to an embodiment of the present disclosure, the first text feature extraction module 710 may include a sub-feature determination sub-module, an embedded feature determination sub-module and a text feature determination sub-module. The sub-feature determination sub-module is used to determine the image sub-features for each field in the image features according to the position information of each field in the image to be processed. The embedded feature determination sub-module is used to determine the text embedded features of each field. The text feature determination sub-module is used to obtain the text features of each field according to the image sub-features and the text embedded features.
[0110] According to an embodiment of the present disclosure, the text feature determination submodule may include a splicing unit and an extraction unit. The splicing unit is used to splice the image subfeatures and the text embedding features to obtain splicing features. The extraction unit is used to extract the text features of each field from the splicing features.
[0111] Based on the training method of the text classification model provided by the present disclosure, the present disclosure also provides a training device for the text classification model. Figure 8 The device is described in detail.
[0112] Figure 8 It is a structural block diagram of a training device for a text classification model according to an embodiment of the present disclosure.
[0113] like Figure 8 As shown, the training device 800 of the text classification model of this embodiment may include a second text feature extraction module 810, a graph feature determination module 820, a category prediction module 830 and a first training module 840. The text classification model includes a text feature extraction network, a graph convolution network and a category prediction network.
[0114] The second text feature extraction module 810 is used to input the image features of the sample image and the multiple fields included in the sample image into the text feature extraction network to obtain the text features of each of the multiple fields. The sample image also includes first information indicating the actual category of each field. In one embodiment, the second text feature extraction module 810 can be used to perform the operation S510 described above, which will not be repeated here.
[0115] The graph feature determination module 820 is used to determine the graph features for the multiple fields using a graph convolutional network according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the image to be processed. The graph features include the structural features of each field. In one embodiment, the graph feature determination module 820 can be used to perform the operation S520 described above, which will not be repeated here.
[0116] The category prediction module 830 is used to input multiple text features of multiple fields and multiple structural features of multiple fields into the category prediction network to obtain the predicted category of each field. Among them, the graph features include the structural features of each field. In one embodiment, the category prediction module 830 can be used to perform the operation S530 described above, which will not be repeated here.
[0117] The first training module 840 is used to train the text classification model according to the predicted category and the actual category. In one embodiment, the first training module 840 can be used to perform the operation S540 described above, which will not be repeated here.
[0118] According to an embodiment of the present disclosure, the graph feature also includes a relationship feature between multiple fields, the sample image also includes second information indicating the relationship between multiple fields, and the text classification model also includes a relationship prediction network. The training device 800 of the above-mentioned text classification model may also include a relationship prediction module and a second training module. The relationship prediction module is used to input the relationship feature into the relationship prediction network to obtain the predicted relationship information between multiple fields. The second training module is used to train the text classification model based on the predicted relationship information and the second information.
[0119] According to an embodiment of the present disclosure, the text classification model also includes a relationship extraction network and a difference extraction network. The above-mentioned graph feature determination module 820 may include a second difference feature determination submodule, a second relationship feature determination submodule and a graph feature determination submodule. The second difference feature determination submodule is used to input multiple text features into the difference extraction network to obtain difference features that characterize the differences between multiple text features. The second relationship feature determination submodule is used to input multiple position information of multiple fields in the sample image into the relationship extraction network to obtain relationship features that characterize the relative relationship between multiple position information. The graph feature determination submodule is used to determine the graph features for multiple fields using a graph convolutional network based on the difference features, relationship features and multiple text features.
[0120] According to an embodiment of the present disclosure, the training device 800 of the text classification model may further include a loss determination module and a third training module. The loss determination module is used to determine the self-supervised loss of the text classification model based on the difference feature and the plurality of text features. The third training module is used to train the text classification model based on the self-supervised loss.
[0121] According to an embodiment of the present disclosure, the above-mentioned graph feature determination submodule may include a second weighting unit and a second feature determination unit. The second weighting unit is used to weight the relationship feature with the difference feature as the weight to obtain the weighted feature. The second feature determination unit is used to input the weighted feature into the graph convolution network to obtain the graph features for multiple fields.
[0122] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.
[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0124] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement the text classification method and / or text classification model training method of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0125] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0126] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0127] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a text classification method and / or a training method for a text classification model. For example, in some embodiments, the text classification method and / or the training method for a text classification model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the text classification method and / or the training method for a text classification model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the text classification method and / or the text classification model training method in any other appropriate manner (for example, by means of firmware).
[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0133] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with a blockchain.
[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A text classification method, comprising: Determine text features of each of the multiple fields according to image features of the image to be processed and multiple fields included in the image to be processed; Determine the structural feature of each field according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the image to be processed; as well as Determine the category of each field according to the text feature of each field and the structural feature of each field; Wherein, determining the structural characteristics of each field includes: Determining, based on the plurality of text features, difference features representing differences between the plurality of text features; Determining, according to the plurality of position information of the plurality of fields in the image to be processed, a relation feature representing a relative relationship between the plurality of position information; and Determining a structural feature of each field according to the difference feature, the relationship feature and the plurality of text features; Wherein, determining the structural feature of each field according to the difference feature, the relationship feature and the multiple text features includes: weighting the relationship feature with the difference feature as a weight to obtain a weighted feature; and The weighted features and the multiple text features are input into a graph convolutional network to obtain structural features of each field.
2. The method according to claim 1, wherein: Determining, based on the multiple text features, difference features representing differences between the multiple text features comprises: Determining a feature matrix composed of the plurality of text features; and The difference feature is determined according to a difference between the feature matrix and a transposed matrix of the feature matrix.
3. The method according to claim 1, wherein: Determining the text feature of each of the multiple fields includes: Determine, according to position information of each field in the image to be processed, an image sub-feature for each field in the image feature; Determining text embedding characteristics of each of the fields; and The text features of each field are obtained according to the image sub-features and the text embedding features.
4. The method according to claim 3, wherein: The obtaining of the text feature of each field according to the image sub-feature and the text embedding feature comprises: splicing the image sub-feature and the text embedding feature to obtain a spliced feature; and The text features of each field are extracted from the concatenated features.
5. A method for training a text classification model, wherein: The text classification model includes a text feature extraction network, a graph convolution network and a category prediction network; the method includes: Inputting image features of a sample image and a plurality of fields included in the sample image into the text feature extraction network to obtain text features of each of the plurality of fields; the sample image also includes first information indicating an actual category of each of the fields; Determine graph features for the multiple fields using the graph convolution network according to the multiple text features of the multiple fields and the multiple position information of the multiple fields in the sample image, wherein the graph features include structural features of each field; Inputting the plurality of text features of the plurality of fields and the plurality of structural features of the plurality of fields into the category prediction network to obtain a predicted category of each of the fields; and Training the text classification model according to the predicted category and the actual category; The text classification model further includes a relationship extraction network and a difference extraction network; and using the graph convolution network to determine graph features for the multiple fields includes: Inputting the plurality of text features into the difference extraction network to obtain difference features representing the differences between the plurality of text features; Inputting a plurality of position information of the plurality of fields in the sample image into the relationship extraction network to obtain a relationship feature representing a relative relationship between the plurality of position information; and Determining graph features for the multiple fields using the graph convolutional network according to the difference feature, the relationship feature, and the multiple text features; The step of determining the graph features for the multiple fields using the graph convolutional network according to the difference feature, the relationship feature, and the multiple text features includes: weighting the relationship feature with the difference feature as a weight to obtain a weighted feature; and The weighted features and the multiple text features are input into the graph convolutional network to obtain graph features for the multiple fields.
6. The method according to claim 5, wherein: The graph feature further includes a feature of the relationship between the multiple fields; the sample image further includes second information indicating the relationship between the multiple fields; The text classification model also includes a relationship prediction network; the method also includes: Inputting the relationship features into the relationship prediction network to obtain prediction relationship information between the multiple fields; as well as The text classification model is trained according to the predicted relationship information and the second information.
7. The method according to claim 5, further comprising: Determining a self-supervision loss of the text classification model according to the difference feature and the multiple text features; as well as The text classification model is trained according to the self-supervised loss.
8. A text classification device, comprising: A first text feature extraction module, used for determining a text feature of each of the multiple fields according to an image feature of the image to be processed and multiple fields included in the image to be processed; A structural feature determination module, used for determining the structural feature of each field according to a plurality of text features of the plurality of fields and a plurality of position information of the plurality of fields in the image to be processed; as well as A category determination module, used for determining the category of each field according to the text features of each field and the structural features of each field; Wherein, the structural feature determination module includes: A first difference feature determination submodule is used to determine, based on the multiple text features, a difference feature representing the difference between the multiple text features; A first relationship feature determination submodule is used to determine, based on the plurality of position information of the plurality of fields in the image to be processed, a relationship feature representing a relative relationship between the plurality of position information; and A structural feature determination submodule, used for determining the structural feature of each field according to the difference feature, the relationship feature and the plurality of text features; Wherein, the structural feature determination submodule includes: a first weighting unit, configured to weight the relationship feature by using the difference feature as a weight to obtain a weighted feature; and The first feature determination unit is used to input the weighted features and the multiple text features into a graph convolutional network to obtain the structural features of each field.
9. The device according to claim 8, wherein: The difference feature determination submodule includes: a feature matrix determining unit, configured to determine a feature matrix composed of the plurality of text features; and The difference feature determination unit is used to determine the difference feature according to the difference between the feature matrix and the transposed matrix of the feature matrix.
10. The device according to claim 8, wherein: The text feature extraction module comprises: A sub-feature determination submodule, used to determine the image sub-feature for each field in the image feature according to the position information of each field in the image to be processed; an embedding feature determination submodule, used to determine the text embedding feature of each field; and The text feature determination submodule is used to obtain the text feature of each field according to the image sub-feature and the text embedding feature.
11. The device according to claim 10, wherein: The text feature determination submodule includes: a splicing unit, configured to splice the image sub-feature and the text embedding feature to obtain a splicing feature; and The extraction unit is used to extract the text features of each field from the concatenated features.
12. A training device for a text classification model, wherein: The text classification model includes a text feature extraction network, a graph convolution network and a category prediction network; the device includes: A second text feature extraction module is used to input the image features of the sample image and the multiple fields included in the sample image into the text feature extraction network to obtain the text features of each of the multiple fields; the sample image also includes first information indicating the actual category of each field; A graph feature determination module, configured to determine graph features for the multiple fields using the graph convolution network according to multiple text features of the multiple fields and position information of the multiple fields in the sample image, wherein the graph features include structural features of each field; a category prediction module, configured to input the plurality of text features of the plurality of fields and the plurality of structural features of the plurality of fields into the category prediction network to obtain a predicted category of each field; and A first training module, used for training the text classification model according to the predicted category and the actual category; Wherein, the text classification model further includes a relationship extraction network and a difference extraction network; the graph feature determination module includes: A second difference feature determination submodule is used to input the plurality of text features into the difference extraction network to obtain a difference feature representing the difference between the plurality of text features; A second relationship feature determination submodule is used to input multiple position information of the multiple fields in the sample image into the relationship extraction network to obtain relationship features representing the relative relationship between the multiple position information; and A graph feature determination submodule, configured to determine graph features for the multiple fields using the graph convolutional network according to the difference feature, the relationship feature, and the multiple text features; Wherein, the graph feature determination submodule includes: a second weighting unit, configured to weight the relationship feature by using the difference feature as a weight to obtain a weighted feature; and The second feature determination unit is used to input the weighted features into the graph convolutional network to obtain graph features for the multiple fields.
13. The device according to claim 12, wherein: The graph feature further includes a feature of the relationship between the multiple fields; the sample image further includes second information indicating the relationship between the multiple fields; The text classification model also includes a relationship prediction network; the device also includes: A relationship prediction module, used for inputting the relationship features into the relationship prediction network to obtain predicted relationship information between the multiple fields; as well as The second training module is used to train the text classification model according to the predicted relationship information and the second information.
14. The apparatus according to claim 12, further comprising: A loss determination module, used to determine the self-supervision loss of the text classification model according to the difference feature and the multiple text features; as well as The third training module is used to train the text classification model according to the self-supervision loss.
15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 7.
16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.
17. A computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.
