Document image feature extraction method, device, electronic device and readable medium
By extracting text features and sequence labeling information in document images, splicing and feature encoding processing, multimodal feature sequences are generated, which solves the problem of difficult model training in the prior art, and improves the training effect and output accuracy of the model with few sample data.
Patent Information
- Application Number
- CN202411773097.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-04-08
AI Technical Summary
When extracting information from document images, the prior art is limited by the limitation of sample data, making model training difficult.
By extracting text features in document images, a text sequence is obtained; word elements are extracted from text sequences and marked to obtain sequence labeling information; then text sequence and sequence labeling information are spliced together to obtain target feature information. This feature information is generated by feature encoding processing to generate a multimodal feature sequence as input to the model.
By emphasizing the correlation between the original text sequence and the sequence labeling information that has been labeled, the model's processing ability of document images is improved, and the training effect and the accuracy of model output when there is less sample data are enhanced.
Smart Images
Figure CN119785370A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning technology, and in particular to a document image feature extraction method, a document image feature extraction device, an electronic device and a computer-readable medium. Background Art
[0002] In the prior art, when it is necessary to extract information from document images, it is usually necessary to obtain a large amount of sample data to train the model in a strongly supervised scenario to achieve information extraction from document images. However, in actual situations, the sample data that can be obtained is usually limited, resulting in the inability to complete the training of the model well. Summary of the invention
[0003] The embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for extracting features of a document image to solve the problem of difficulty in training a document image processing model.
[0004] The embodiment of the present invention discloses a method for extracting features of a document image, comprising:
[0005] Extract text features from document images to obtain text sequences;
[0006] Extracting word units from the text sequence and marking the word units to obtain sequence marking information;
[0007] The text sequence and the sequence annotation information are concatenated to obtain target feature information.
[0008] Optionally, the step of extracting text features from the document image to obtain a text sequence includes:
[0009] A preset text extraction model is used to extract text features in the document image to obtain a text sequence.
[0010] Optionally, the step of extracting word units from the text sequence and marking the word units to obtain sequence marking information includes:
[0011] Words are extracted from the text sequence, and sequence annotation information is generated based on the word-grams and the relationship between the entities to which the word-grams belong in the text sequence.
[0012] Optionally, the step of concatenating the text sequence and the sequence annotation information to obtain target feature information includes:
[0013] splicing the text sequence and the sequence annotation information to obtain a spliced sequence;
[0014] The concatenated sequence is subjected to feature encoding processing to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
[0015] Optionally, the method further comprises:
[0016] Adding mask information to the target feature information to obtain masked target feature information;
[0017] The target feature information that has been masked is input into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
[0018] Optionally, the method further comprises:
[0019] The hidden vector information is input into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
[0020] Optionally, the method further comprises:
[0021] If it is determined that the type corresponding to the word element in the hidden vector information is a text sequence type, the word element is used as a candidate word constraint;
[0022] If it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type, the word is used as a candidate type and / or a candidate label.
[0023] The present invention also provides a document image feature extraction device, comprising:
[0024] A feature extraction module is used to extract text features from document images to obtain text sequences;
[0025] A tagging module, used for extracting word units from the text sequence and tagging the word units to obtain sequence tagging information;
[0026] The splicing module is used to splice the text sequence and the sequence annotation information to obtain target feature information.
[0027] Optionally, the feature extraction module includes:
[0028] The feature extraction submodule is used to extract text features in the document image using a preset text extraction model to obtain a text sequence.
[0029] Optionally, the marking module includes:
[0030] The annotation submodule extracts word units from the text sequence and generates sequence annotation information based on the relationship between the word units and the entities to which the word units belong in the text sequence.
[0031] Optionally, the splicing module includes:
[0032] A sequence splicing submodule, used for splicing the text sequence and the sequence annotation information to obtain a spliced sequence;
[0033] The multimodal conversion submodule is used to perform feature encoding processing on the spliced sequence to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
[0034] Optionally, the device further comprises:
[0035] A mask processing module, used for adding mask information to the target feature information to obtain the target feature information after mask processing;
[0036] A vector output module is used to input the masked target feature information into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
[0037] Optionally, the device further comprises:
[0038] The vector classification module is used to input the hidden vector information into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
[0039] Optionally, the device further comprises:
[0040] A first classification module, configured to use the word-unit as a candidate word constraint if it is determined that the type corresponding to the word-unit in the hidden vector information is a text sequence type;
[0041] The second classification module is used to use the word as a candidate type and / or candidate label if it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type.
[0042] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0043] The memory is used to store computer programs;
[0044] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.
[0045] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.
[0046] The embodiments of the present invention include the following advantages:
[0047] By using the feature extraction method for document images provided by the embodiment of the present invention, text features in the document image are extracted to obtain a text sequence; words are extracted from the text sequence, and the words are annotated to obtain sequence annotation information; the text sequence and the sequence annotation information are concatenated to obtain target feature information. Thus, the target feature information obtained can emphasize the correlation between the original text sequence and the annotated sequence annotation information, so that in the subsequent model processing of the document image, the correlation between the original text sequence and the annotated sequence annotation information can be focused on, thereby improving the accuracy of the model output, and obtaining a better training effect when the sample data is small. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flowchart of the steps of a feature extraction method for a document image provided in an embodiment of the present invention;
[0049] Figure 2 is a schematic diagram of a process of feature extraction of a document image provided in an embodiment of the present invention;
[0050] Figure 3 is a structural block diagram of a document image feature extraction device provided in an embodiment of the present invention;
[0051] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present invention;
[0052] Figure 5 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] Reference Figure 1 , shows a flowchart of a method for extracting features of a document image provided in an embodiment of the present invention, which may specifically include the following steps:
[0055] Step 101, extracting text features from a document image to obtain a text sequence;
[0056] In the embodiment of the present invention, for a document image, text features in the document image may be first extracted. The text features may be expressed in text form, such as characters, numbers, tag symbols, spaces, etc. corresponding to different languages. The text features may be used as a text sequence, and the characters in the text sequence may be used as word units of the text sequence.
[0057] Step 102, extracting word units from the text sequence, and marking the word units to obtain sequence marking information;
[0058] Specifically, the information recorded in the text sequence may include at least one entity word, and the entity word may belong to at least one type of entity. The word unit may be labeled to determine the entity type corresponding to the word unit in the text sequence.
[0059] In a specific implementation, a word unit can be extracted from a text sequence, the entity type corresponding to the word unit can be identified, and sequence annotation information can be generated, wherein the sequence annotation information can include the word units belonging to the entity words in the text sequence, and the entity types corresponding to the word units. For example, for the text sequence "Zhaoqing East Station", which can include the entity word "Zhaoqing East", the entity type of which can be the starting station (start station), then the sequence annotation information can record the word units "Zhao", "Qing", "East" in the entity word and the entity type "start station".
[0060] Step 103: concatenate the text sequence and the sequence annotation information to obtain target feature information.
[0061] Afterwards, the text sequence and the sequence information can be spliced to obtain the target feature information. Thus, the target feature information emphasizing the correlation between the original text sequence and the sequence annotation information after the annotation processing can be obtained, so that in the process of subsequent model processing of document images, the correlation between the original text sequence and the sequence annotation information after the annotation processing can be focused on, thereby improving the accuracy of the model output. The subsequent model processing of document images may include document translation, document classification, etc., and the present invention does not limit this.
[0062] By using the feature extraction method for document images provided by the embodiment of the present invention, text features in the document image are extracted to obtain a text sequence; words are extracted from the text sequence, and the words are annotated to obtain sequence annotation information; the text sequence and the sequence annotation information are concatenated to obtain target feature information. Thus, the target feature information obtained can emphasize the correlation between the original text sequence and the annotated sequence annotation information, so that in the subsequent model processing of the document image, the correlation between the original text sequence and the annotated sequence annotation information can be focused on, thereby improving the accuracy of the model output, and obtaining a better training effect when the sample data is small.
[0063] In one embodiment of the present invention, the step of extracting text features from the document image to obtain a text sequence includes:
[0064] S11, using a preset text extraction model to extract text features in the document image to obtain a text sequence.
[0065] In a specific implementation, a text extraction model for identifying text features in a document image may be pre-trained, and the text extraction model may be used to extract text features in the document image to obtain a text sequence.
[0066] In the specific implementation, a multimodal pre-trained model such as LayoutLMv2 can be used as a text extraction model. It can comprehensively extract text, layout and image information from document images, and can better understand and extract structured information in documents.
[0067] In one embodiment of the present invention, the step of extracting word units from the text sequence and marking the word units to obtain sequence marking information includes:
[0068] S21, extracting word-grams from the text sequence, and generating sequence annotation information based on the word-grams and the relationship between the entities to which the word-grams belong in the text sequence.
[0069] In the embodiment of the present invention, characters in a text sequence may be used as word-grams. The text sequence may include at least one entity word, and an entity word may include at least one word-gram, so that a word-gram may belong to an entity.
[0070] In order to strengthen the relationship between a word and its corresponding entity during feature extraction, sequence annotation information may be generated based on the relationship between the word and the entity to which the word belongs in the text sequence.
[0071] As a specific example of the present invention, for the text sequence "Zhaoqing East Station", it may include the entity word "Zhaoqing East", and its entity type may be the starting station (start station). Then the sequence annotation information may record the word elements "Zhao", "Qing", "East" in the entity word and the entity type "start station".
[0072] Sequence annotation information can also include functional tags such as 、 <e> 、 <t>,in Indicates the beginning of an entity word. <e>Indicates the end of an entity word. <t>Indicates the type of entity word, then the sequence annotation information can be expressed as: SOS Zhaoqing East <e>start station <t>SOS stands for EOS. Among them, SOS indicates the start of the sequence and EOS indicates the end of the sequence.
[0073] In one embodiment of the present invention, the step of concatenating the text sequence and the sequence annotation information to obtain target feature information includes:
[0074] S31, concatenating the text sequence and the sequence annotation information to obtain a concatenated sequence;
[0075] Specifically, the original text sequence and sequence annotation information may be first concatenated to obtain a concatenated sequence, which may be in a text form.
[0076] S32, performing feature encoding processing on the spliced sequence to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
[0077] Subsequently, in order to further improve the accuracy of feature extraction, the complementary advantages between different modal information can be used to improve the robustness of the model. Thus, the concatenated sequence can be subjected to feature encoding processing to obtain a feature sequence of at least one modality.
[0078] Specifically, the original concatenated sequence may be represented in text form, and the concatenated sequence in text form may be subjected to feature encoding processing to convert it into at least one form of vision, space, and position.
[0079] For visual information, images corresponding to different words in the text sequence can be extracted from the document image to obtain a visual feature sequence. For entity types, images corresponding to entity words can be used in the visual feature sequence.
[0080] Spatial information may refer to features related to spatial position, distance, direction, shape, etc. in a document image, which means mapping spatial features (such as spatial position, distance, direction, shape, etc.) to a low-dimensional vector space to obtain a spatial feature sequence.
[0081] The position information may refer to the position information of different word units in the sequence. Different positions in the sequence may be numbered in order to obtain a sequence having the same length as the concatenated sequence, thereby obtaining a position feature sequence.
[0082] In one embodiment of the present invention, the method further comprises:
[0083] S41, adding mask information to the target feature information to obtain masked target feature information;
[0084] S42, inputting the masked target feature information into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
[0085] In a specific implementation, in order to further improve the accuracy of the hidden vector contained in the model in the extracted feature information, mask information can be further added to the target feature information, and the masked target feature information is input into a preset encoder model.
[0086] The mask model can control the encoder model to focus on any word in the text sequence part in the target feature information when processing the text sequence part, so as to fully understand the contextual relationship of the text in the text sequence.
[0087] When processing any word in the sequence annotation information part of the target feature information, the focus is on the feature information located before the word, so that when the model processes the sequence annotation information part of the target feature information, it can only focus on the previous information to infer the subsequent information, avoiding the leakage of future information, that is, the subsequent information, so that the model can have better output accuracy in the task of continuing the previous context based on the previous context reasoning.
[0088] In one embodiment of the present invention, the method further comprises:
[0089] S51, inputting the hidden vector information into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
[0090] In a specific implementation, in order to make the model pay more attention to the information it wants during the feature processing, the output process of the model can be constrained. Therefore, a classification module can be set up, which can be used to respectively identify the types corresponding to the word units in the hidden vector information, so that the model can fully consider the types corresponding to different word units during the output process, thereby improving the accuracy of the model output.
[0091] Specifically, the vector corresponding to the sequence annotation information in the hidden vector information can be mainly processed, and input into the classification module to identify the type corresponding to the word unit.
[0092] In one embodiment of the present invention, the method further comprises:
[0093] S61, if it is determined that the type corresponding to the word unit in the hidden vector information is a text sequence type, the word unit is used as a candidate word constraint;
[0094] S62: If it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type, the word is used as a candidate type and / or a candidate label.
[0095] In a specific implementation, the classification module can be used to determine whether the type corresponding to the word in the hidden vector information belongs to the text sequence type. If the word belongs to the text sequence type, the word can be used as a candidate word constraint, and the word in the candidate word constraint can be considered as a reference for the output text during the model output process. If the type corresponding to the word is not a text sequence type, the word is not suitable as a reference for the output text, but can only be used as a candidate type corresponding to the entity word and / or a candidate label in the sequence, such as 、 <e> 、 <t>wait.
[0096] Therefore, during the model output process, candidate word constraints can be added to the model to further improve the accuracy of the model output.
[0097] As a specific example of the present invention, Figure 2 1 is a flow chart of feature extraction of a document image provided in an embodiment of the present invention. The method comprises:
[0098] First pre-train a text extraction model (such as LayoutLMv2), then obtain the pre-trained weights, and then fine-tune it on the downstream task. The fine-tuning process includes:
[0099] A text extraction model is used to extract text features in the document image to obtain a text sequence (Source sequence).
[0100] A token sequence annotation design is introduced to extract tokens from the text sequence, and sequence annotation information (Target sequence) is generated based on the tokens and the relationship between the entities to which the tokens belong in the text sequence.
[0101] Characters in a text sequence may be used as word-grams. The text sequence may include at least one entity word, and an entity word may include at least one word-gram, so that a word-gram may belong to an entity.
[0102] In order to strengthen the relationship between a word and its corresponding entity during feature extraction, sequence annotation information may be generated based on the relationship between the word and the entity to which the word belongs in the text sequence.
[0103] As a specific example of the present invention, for the text sequence "Zhaoqing East Station", it may include the entity word "Zhaoqing East", and its entity type may be the starting station (start station). Then the sequence annotation information may record the word elements "Zhao", "Qing", "East" in the entity word and the entity type "start station".
[0104] Sequence annotation information can also include functional tags such as 、 <e> 、 <t>,in Indicates the beginning of an entity word. <e>Indicates the end of an entity word. <t>Indicates the type of entity word, then the sequence annotation information can be expressed as: SOS Zhaoqing East <e>start station <t>SOS stands for EOS. Among them, SOS indicates the start of the sequence and EOS indicates the end of the sequence.
[0105] The text sequence (Source sequence) and the sequence annotation information (Target sequence) are connected to obtain a spliced sequence.
[0106] The concatenated sequence is subjected to feature encoding processing to obtain a feature sequence of at least one modality as target feature information; the modalities include text features, visual features, spatial features, and position features.
[0107] The original splicing sequence can be represented in text form, and the splicing sequence in text form can be feature encoded to convert it into at least one form of vision, space, and position.
[0108] For visual information, images corresponding to different words in the text sequence can be extracted from the document image to obtain a visual feature sequence. For entity types, images corresponding to entity words can be used in the visual feature sequence.
[0109] Spatial information may refer to features related to spatial position, distance, direction, shape, etc. in a document image, which means mapping spatial features (such as spatial position, distance, direction, shape, etc.) to a low-dimensional vector space to obtain a spatial feature sequence.
[0110] The position information may refer to the position information of different word units in the sequence. Different positions in the sequence may be numbered in order to obtain a sequence having the same length as the concatenated sequence, thereby obtaining a position feature sequence.
[0111] The target feature information is then sent to the backbone part of the downstream task model (transformer encoder) for training.
[0112] Create a mask unit for the target feature information. The mask unit is similar to the triangle mask module. Through the attention mechanism, the text sequences (Source sequence) pay attention to each other during the encoding process, while the sequence annotation information (Target sequence) can only use the previous information.
[0113] The mask model can control the encoder model to focus on any word in the text sequence part in the target feature information when processing the text sequence part, so as to fully understand the contextual relationship of the text in the text sequence.
[0114] When processing any word in the sequence annotation information part of the target feature information, the focus is on the feature information located before the word, so that when the model processes the sequence annotation information part of the target feature information, it can only focus on the previous information to infer the subsequent information, avoiding the leakage of future information, that is, the subsequent information, so that the model can have better output accuracy in the task of continuing the previous context based on the previous context reasoning.
[0115] After the hidden vector information (target hidden states) is extracted by the backbone part of the model (transformer encoder), the vector corresponding to the sequence annotation information (Target sequence) is extracted. Through the goal-directed classification module (Goal-directed Classification), the token in the hidden vector information is classified into two categories to determine whether it belongs to the candidate word (Source). If it belongs to the candidate word (Source), the corresponding sequence is used as a candidate word constraint to generate the corresponding token through multi-classification processing. If it does not belong to the Source, it can only be the category of the entity. 、 <e> 、 <t>For the three functional labels, multi-classification is also performed to generate labels.
[0116] In a specific implementation, the classification module can be used to determine whether the type corresponding to the word in the hidden vector information belongs to the text sequence type. If the word belongs to the text sequence type, the word can be used as a candidate word constraint, and the word in the candidate word constraint can be considered as a reference for the output text during the model output process. If the type corresponding to the word is not a text sequence type, the word is not suitable as a reference for the output text, but can only be used as a candidate type corresponding to the entity word and / or a candidate label in the sequence, such as 、 <e> 、 <t>wait.
[0117] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0118] Reference Figure 3 , shows a structural block diagram of a document image feature extraction device provided in an embodiment of the present invention, which may specifically include the following modules:
[0119] A feature extraction module 301 is used to extract text features in a document image to obtain a text sequence;
[0120] A labeling module 302 is used to extract word units from the text sequence and label the word units to obtain sequence labeling information;
[0121] The splicing module 303 is used to splice the text sequence and the sequence annotation information to obtain target feature information.
[0122] Optionally, the feature extraction module includes:
[0123] The feature extraction submodule is used to extract text features in the document image using a preset text extraction model to obtain a text sequence.
[0124] Optionally, the marking module includes:
[0125] The annotation submodule extracts word units from the text sequence and generates sequence annotation information based on the relationship between the word units and the entities to which the word units belong in the text sequence.
[0126] Optionally, the splicing module includes:
[0127] A sequence splicing submodule, used for splicing the text sequence and the sequence annotation information to obtain a spliced sequence;
[0128] The multimodal conversion submodule is used to perform feature encoding processing on the spliced sequence to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
[0129] Optionally, the device further comprises:
[0130] A mask processing module, used for adding mask information to the target feature information to obtain the target feature information after mask processing;
[0131] A vector output module is used to input the masked target feature information into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
[0132] Optionally, the device further comprises:
[0133] The vector classification module is used to input the hidden vector information into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
[0134] Optionally, the device further comprises:
[0135] A first classification module, configured to use the word-unit as a candidate word constraint if it is determined that the type corresponding to the word-unit in the hidden vector information is a text sequence type;
[0136] The second classification module is used to use the word as a candidate type and / or candidate label if it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type.
[0137] By using the feature extraction device for document images provided by the embodiment of the present invention, text features in the document image are extracted to obtain a text sequence; words are extracted from the text sequence, and the words are annotated to obtain sequence annotation information; the text sequence and the sequence annotation information are concatenated to obtain target feature information. Thus, the target feature information obtained can emphasize the correlation between the original text sequence and the annotated sequence annotation information, so that in the subsequent model processing of the document image, the correlation between the original text sequence and the annotated sequence annotation information can be focused on, thereby improving the accuracy of the model output, and obtaining a better training effect when the sample data is small.
[0138] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0139] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.
[0140] Memory 403, used for storing computer programs;
[0141] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:
[0142] Extract text features from document images to obtain text sequences;
[0143] Extracting word units from the text sequence and marking the word units to obtain sequence marking information;
[0144] The text sequence and the sequence annotation information are concatenated to obtain target feature information.
[0145] Optionally, the step of extracting text features from the document image to obtain a text sequence includes:
[0146] A preset text extraction model is used to extract text features in the document image to obtain a text sequence.
[0147] Optionally, the step of extracting word units from the text sequence and marking the word units to obtain sequence marking information includes:
[0148] Words are extracted from the text sequence, and sequence annotation information is generated based on the word-grams and the relationship between the entities to which the word-grams belong in the text sequence.
[0149] Optionally, the step of concatenating the text sequence and the sequence annotation information to obtain target feature information includes:
[0150] splicing the text sequence and the sequence annotation information to obtain a spliced sequence;
[0151] The concatenated sequence is subjected to feature encoding processing to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
[0152] Optionally, the method further comprises:
[0153] Adding mask information to the target feature information to obtain masked target feature information;
[0154] The target feature information that has been masked is input into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
[0155] Optionally, the method further comprises:
[0156] The hidden vector information is input into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
[0157] Optionally, the method further comprises:
[0158] If it is determined that the type corresponding to the word element in the hidden vector information is a text sequence type, the word element is used as a candidate word constraint;
[0159] If it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type, the word is used as a candidate type and / or a candidate label.
[0160] Through the electronic device provided by the embodiment of the present invention, the text features in the document image are extracted to obtain a text sequence; words are extracted from the text sequence, and the words are annotated to obtain sequence annotation information; the text sequence and the sequence annotation information are spliced to obtain target feature information. Thus, the target feature information obtained can emphasize the correlation between the original text sequence and the annotated sequence annotation information, so that in the subsequent model processing of the document image, the correlation between the original text sequence and the annotated sequence annotation information can be focused on, thereby improving the accuracy of the model output, and even in the case of less sample data, a better training effect can be obtained.
[0161] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0162] The communication interface is used for communication between the above terminal and other devices.
[0163] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0164] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0165] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the feature extraction method of the document image described in the above embodiment.
[0166] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the feature extraction method of the document image described in the above embodiment.
[0167] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0168] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0169] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.< / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e> < / t> < / e>
Claims
1. A feature extraction method for a document image, characterized in that: include: Extract text features from document images to obtain text sequences; Extracting word units from the text sequence and marking the word units to obtain sequence marking information; The text sequence and the sequence annotation information are concatenated to obtain target feature information.
2. The method according to claim 1, characterized in that: The step of extracting text features from the document image to obtain a text sequence includes: A preset text extraction model is used to extract text features in the document image to obtain a text sequence.
3. The method according to claim 1, characterized in that The step of extracting word units from the text sequence and marking the word units to obtain sequence marking information includes: Words are extracted from the text sequence, and sequence annotation information is generated based on the word-grams and the relationship between the entities to which the word-grams belong in the text sequence.
4. The method according to claim 1, characterized in that: The step of concatenating the text sequence and the sequence annotation information to obtain target feature information includes: splicing the text sequence and the sequence annotation information to obtain a spliced sequence; The concatenated sequence is subjected to feature encoding processing to obtain a feature sequence of at least one modality as the target feature information; the modality includes at least one of text, vision, space, and position.
5. The method according to claim 1, further comprising: Adding mask information to the target feature information to obtain masked target feature information; The target feature information that has been masked is input into a preset encoder model to obtain hidden vector information output by the encoder model; wherein the mask information is used to control the encoder model to focus on the text sequence part when processing any word in the text sequence part of the target feature information, and to focus on the feature information located before the word when processing any word in the sequence annotation information part of the target feature information.
6. The method according to claim 5, characterized in that The method further comprises: The hidden vector information is input into a preset classification module; the classification module is used to respectively identify the types corresponding to the word units in the hidden vector information.
7. The method according to claim 6, characterized in that The method further comprises: If it is determined that the type corresponding to the word element in the hidden vector information is a text sequence type, the word element is used as a candidate word constraint; If it is determined that the type corresponding to the word in the hidden vector information is not a text sequence type, the word is used as a candidate type and / or a candidate label.
8. A feature extraction device for a document image, characterized in that: include: A feature extraction module is used to extract text features from document images to obtain text sequences; A labeling module, used for extracting word units from the text sequence and labeling the word units to obtain sequence labeling information; The splicing module is used to splice the text sequence and the sequence annotation information to obtain target feature information.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.
10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.