A method and device for information extraction of a shipping order combining reading order and multimodal

Abstract

The application discloses a kind of fusion reading order and multimodal sea transport single information extraction method and device, the method includes: the sea transport single document is handled by OCR, and text sequence is sorted according to boundary frame coordinates;Utilize LayoutLM model to extract the word-level multimodal representation of fusion text, position and visual feature;Enhance the word interaction in frame by graph attention network;Local and global semantic features are modeled by two layers BiLSTM respectively;Finally, the structured key-value pair information is extracted by predicting sequence label through CRF layer.The application effectively solves the problem of sea transport single information extraction in complex layout, and significantly improves the extraction accuracy.

Description

Technical Field

[0001] This invention relates to the field of multimodal information extraction technology, and in particular to a method and apparatus for extracting ocean bill of lading information by integrating reading order and multimodality. Background Technology

[0002] Information extraction is a crucial task in the field of Natural Language Processing (NLP), aiming to identify and extract key information needed by users from structured or semi-structured document images to achieve automatic understanding and utilization of document content. Traditional information extraction methods are mostly based on sequence labeling, such as BIO (Begin Inside Outside) labeling or direct classification strategies. These methods often follow the Named Entity Recognition (NER) technology in Natural Language Processing, that is, first extracting the text content and its corresponding spatial location information from the image using OCR (Optical Character Recognition) tools, and then predicting the entity type. Although existing OCR technology can recognize text content relatively accurately, it has significant shortcomings in modeling the text arrangement order. In practical applications, the reading order of the document has a significant impact on the information extraction results. Reading order usually follows the natural reading logic from left to right and from top to bottom; if the model ignores this order, it is easy to cause the extraction results to be biased or even wrong.

[0003] To address this, some studies have attempted to incorporate spatial location information of text and page layout of images into models through encoding, aiming to enhance their adaptability to layout changes in visual rich documents. However, most of these methods still employ direct classification strategies, failing to fully utilize the structural information implicit in document layout and neglecting the impact of reading order on extraction performance. Furthermore, pure classification methods perform poorly when dealing with numerical entities or semantically ambiguous entities. For example, different categories may correspond to the same person's name, or the same text may belong to different categories depending on its location; these situations easily lead to misclassification. Summary of the Invention

[0004] The purpose of this invention is to provide a method for extracting shipping manifest information that integrates reading order and multimodality. This method can improve the extraction of visual rich document information and takes into account the impact of OCR text recognition order on model performance. It is very effective for extracting hierarchical nested structure information and for extracting different document formats.

[0005] To achieve the above objectives, the present invention provides a method for extracting ocean bill of lading information that integrates reading order and multimodal processing, comprising:

[0006] Step 1: Perform optical character recognition (OCR) processing on the shipping manifest document image to obtain each text segment and its corresponding bounding box coordinates; based on the bounding box coordinates, sort all text segments according to a preset reading order to obtain a sorted text sequence.

[0007] Step 2: Use the LayoutLM model to extract multimodal features from the text sequence output in Step 1 to obtain the word-level feature representations in each text segment. The multimodal features include text semantic features, two-dimensional position embedding features, and aligned visual features.

[0008] Step 3: For all words inside the bounding box of each text segment, construct a fully connected graph structure and introduce a graph attention network (GAT) to promote information interaction and feature enhancement between words inside the bounding box.

[0009] Step 4: A two-layer bidirectional long short-term memory network (BiLSTM) is used for multi-level sequence modeling: the first layer of the BiLSTM network is used to capture the sequence dependencies between internal words within the same bounding box; the second layer of the BiLSTM network is used to capture the sequence dependencies between different bounding boxes to model the global reading order.

[0010] Step 5: Input the feature sequence output from the second layer of BiLSTM in step 4 into the Conditional Random Field (CRF) layer to perform sequence prediction of named entity tags, obtain the BIO tag corresponding to each word, and then extract the structured key-value pair information.

[0011] Furthermore, the methods for obtaining the two-dimensional location embedding features in step 2 include:

[0012] Based on the text bounding box coordinates, normalize them into virtual coordinates;

[0013] Calculate the embedding representation of the virtual coordinates in the four dimensions of x-axis starting position, y-axis starting position, width w, and height h respectively;

[0014] The four-dimensional embedding representations are summed to obtain the final two-dimensional position embedding feature.

[0015] Furthermore, the methods for extracting visual features in step 2 include:

[0016] The document images are used to extract features using a ResNet residual network pre-trained on the large-scale visual recognition dataset ImageNet to obtain a global visual feature map.

[0017] Based on the bounding box coordinates of each text segment, the local visual features of the corresponding region are extracted from the global visual feature map through the Region of Interest Align (ROIAlign) operation.

[0018] The local visual features are aligned and fused with the text and position features extracted by the LayoutLM model to obtain the aligned visual features.

[0019] Furthermore, the graph attention network in step 3 employs a multi-head self-attention mechanism to calculate the attention weights of each lexical node within the bounding box and all its neighboring nodes, thereby aggregating neighborhood information to update node features.

[0020] Furthermore, in step 5, the CRF layer automatically learns the constraint rules between BIO labels during training. The constraint rules include: the sentence beginning label is set to "B-" or "O"; in "I-label2" following "B-label1", label2 is set to be the same as label1; "O" can be followed by labels other than "I-".

[0021] Furthermore, step 5 includes a key-value association step: consecutive terms predicted to be the same entity name are merged into entity values, and the entity values ​​are associated with the corresponding key categories by calculating semantic similarity according to a predefined list of entity categories, forming complete key-value pairs.

[0022] Furthermore, the predefined entity categories include shipper, consignee, port of destination, and port of discharge, where each key category corresponds to a value category, and the value category is identified by the suffix "-value".

[0023] Furthermore, the specific implementation of the preset reading order sorting in step 1 is as follows: first compare the ordinates of the upper left corner of the text segment bounding box, and rank the one with the smaller ordinate first; if the ordinates are the same, compare the abscissas, and rank the one with the smaller abscissa first.

[0024] Furthermore, the LayoutLM model used in step 2 integrates the masking language model task and the sentence order prediction task during the training phase to learn the semantic and layout information of the document text.

[0025] The present invention also provides a device for extracting ocean bill of lading information that integrates reading order and multimodality, comprising:

[0026] The document processing module is used to perform optical character recognition (OCR) processing on the shipping manifest document image to obtain each text segment and its corresponding bounding box coordinates; based on the bounding box coordinates, all text segments are sorted according to a preset reading order to obtain a sorted text sequence.

[0027] The multimodal feature extraction module is used to extract multimodal features from the text sequence output by the document processing module using the LayoutLM model, and obtain the feature representation at the word level in each text segment. The multimodal features include text semantic features, two-dimensional position embedding features, and aligned visual features.

[0028] The graph structure modeling module is used to construct a fully connected graph structure for all words inside the bounding box of each text segment, and introduces the graph attention network GAT to promote information interaction and feature enhancement between words inside the bounding box.

[0029] The sequence modeling module is used for multi-level sequence modeling using a two-layer bidirectional long short-term memory network (BiLSTM): the first layer of the BiLSTM network is used to capture the sequence dependencies between internal words within the same bounding box; the second layer of the BiLSTM network is used to capture the sequence dependencies between different bounding boxes to model the global reading order.

[0030] The information decoding module is used to input the feature sequence output by the second layer BiLSTM into the conditional random field (CRF) layer to perform sequence prediction of named entity tags, obtain the BIO tag corresponding to each word, and then extract the structured key-value pair information.

[0031] The present invention has the following advantages due to the adoption of the above technical solutions:

[0032] 1. This invention proposes a feature representation based on multimodal fusion, which effectively extracts the nested key-value relationships in sub-tables.

[0033] 2. The LayoutLM model used in this invention fuses text and location information, and visual information is further fused based on the text and location information fused by LayoutLM. A two-layer BiLSTM network and a CRF model are used to address the impact of text order on the model's recognition performance in OCR recognition.

[0034] 3. This invention has a good extraction effect on some semi-structured documents with missing key-value relationships, such as documents with only keys or only values. Attached Figure Description

[0035] Figure 1 This is a framework diagram of the method for extracting ocean bill of lading information by integrating reading order and multimodality according to an embodiment of the present invention;

[0036] Figure 2 This is a flowchart of the method for extracting ocean bill of lading information by integrating reading order and multimodality according to an embodiment of the present invention;

[0037] Figure 3This is a key-value entity relationship diagram defined in the embodiments of the present invention;

[0038] Figure 4 This is a key-value entity relationship diagram of nested sub-tables in an embodiment of the present invention. Detailed Implementation

[0039] In the accompanying drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0040] In the description of this invention, the terms "center," "longitudinal," "lateral," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this invention.

[0041] like Figure 1 and Figure 2 As shown, this embodiment uses a typical international bill of lading as an example to explain in detail the implementation process of the method of the present invention. This bill of lading contains key fields such as shipper, consignee, port of destination, and cargo description, and its layout is complex, including tables and free text areas.

[0042] Step 1, OCR Recognition and Reading Order Sorting: Use the open-source OCR engine PaddleOCR to sort the image of the shipping manifest document ( Figure 1 The semi-structured visual rich document mentioned in the text is processed using optical character recognition (OCR) to generate a corresponding bounding box for each text region. The coordinates of each text segment and its corresponding bounding box are obtained, with each bounding box represented by the coordinates of its upper left and lower right corners. Based on the bounding box coordinates, all text segments are sorted according to a preset reading order to obtain a sorted text sequence.

[0043] The preset reading order simulates the human reading order, which is generally from left to right and from top to bottom, but other orders are not excluded. Sorting methods may include, for example:

[0044] Step 11: Establish a Cartesian coordinate system with the top left corner of the shipping manifest document image as the origin, calculate the coordinate information of each bounding box in this coordinate system, and record the coordinate information (x, y, z) of each bounding box. min , y min , x max , y max ).

[0045] Step 12, the primary sort key is the y-axis. min According to y min The values ​​are sorted from smallest to largest (i.e., from top to bottom).

[0046] Step 13, the secondary sort key is the x-coordinate. min For y min Text segments with the same or very close values ​​(set a threshold, such as 5 pixels) are ranked according to x. min The values ​​are sorted from smallest to largest (i.e., from left to right).

[0047] This sorting process organizes the originally scattered OCR recognition results into a text sequence that conforms to human reading habits, laying the foundation for subsequent processing.

[0048] Step 2, Multimodal Feature Extraction: The LayoutLM model is used to extract multimodal features from the text sequence output in Step 1, obtaining token-level feature representations for each text segment. These multimodal features include textual semantic features, 2D position embedding features, and aligned visual features.

[0049] In the above embodiments, text and position feature extraction refers to: arranging the sorted text sequence, such as ['SHIPPER', 'ABC', 'CO.,', 'LTD.'], into the input format required by the LayoutLM pre-trained model, and then inputting it into the LayoutLM model. The LayoutLM pre-trained model is based on the BERT architecture, obtains the contextual semantic embedding of each word, and while capturing text semantic features, fuses bounding box position information to generate a word-level feature representation that integrates text semantics and layout information.

[0050] In the above embodiments, the two-dimensional location embedding specifically includes: the LayoutLM pre-trained model processes the bounding box coordinates corresponding to each word. Specifically, firstly, the coordinates are normalized to a virtual coordinate space of [0, 1000]. Then, the x-coordinates of the normalized coordinates are calculated respectively. min , y min , x max , y max The four values ​​are represented as vectors in their respective embedding tables, and these four vectors are added together to obtain the final two-dimensional positional embedding. This embedding is added to the textual semantic embedding, thus incorporating spatial layout information.

[0051] In the above embodiments, visual feature extraction and alignment specifically include:

[0052] Step 21: Use ResNet (Residual Network), which is pre-trained on the large-scale visual recognition dataset ImageNet, to perform convolution operations on the original shipping manifest image to obtain a global visual feature map.

[0053] Step 22: For each text segment bounding box recognized by OCR, the Region of Interest (ROI) Alignment technique is used to precisely locate and extract the local visual features of that region from the global features extracted by ResNet, achieving alignment between the text position and visual information, and outputting the aligned visual features. Specifically, the bounding box of the text segment is mapped onto the global feature map, and the region is uniformly divided into k×k (e.g., 7×7) bins. Bilinear interpolation is performed on the four feature points within each bin to calculate their precise feature values. Then, max pooling is performed on the values ​​within each bin, finally outputting a fixed-size (e.g., 7×7×C) visual feature tensor. This process avoids quantization errors and ensures precise alignment between features and text positions.

[0054] Step 23: Concatenate the aligned visual feature vector of each word with the text-position feature vector output by the LayoutLM pre-trained model to form the final word-level multimodal feature representation, which is used for subsequent document graph construction and sequence modeling.

[0055] Step 3: For all words within the bounding box of each text segment, construct a fully connected graph structure and introduce a Graph Attention Network (GAT) to promote information interaction and feature enhancement between words within the bounding box. To capture complex dependencies between words within the same bounding box, such as "ABC", "CO.", and "LTD." belonging to the "SHIPPER" field, construct a fully connected graph structure for words within the bounding box of each text segment.

[0056] The graph structure specifically includes: using the multimodal features of each word as the initial features of the graph nodes. The words output from step 2, which integrate text, layout, and image features, are used as graph nodes, where words within each bounding box are considered local nodes, and edges between nodes represent the relationships between words or bounding boxes, thereby constructing a local document graph based on the Graph Attention Network (GAT).

[0057] A two-layer Graph Attention Network (GAT) is introduced. GAT aggregates information about nodes and their neighbors, extracting high-dimensional semantic features through node and edge representation learning. It employs a multi-head self-attention (MHSA) mechanism (e.g., four heads) to calculate the attention weights between each word node within the bounding box and all other nodes. Nodes with higher weights exhibit stronger information transmission. Through message passing and aggregation in GAT, the attention weights of all nodes are combined with the edge weights of the adjacency matrix to update the node representations. This ensures that the final word representation incorporates both local structural information and captures global semantic dependencies. In this way, the features of each word node are enhanced, incorporating contextual information from other words in the same group.

[0058] Step 4, Sequence Modeling with a Multi-Layer Bidirectional Long Short-Term Memory Network (BiLSTM): Using the high-dimensional semantic features at the word level output in Step 3 as input, the features of all words enhanced by GAT are arranged according to the preset reading order in Step 1 and input into a two-layer Bidirectional Long Short-Term Memory Network (BiLSTM) for sequence modeling to generate a multimodal representation of the document, specifically including:

[0059] The first layer is a bidirectional long short-term memory network (BiLSTM) with a hidden layer dimension of 256. It processes word sequences within each bounding box to extract local multimodal features, primarily used to capture local sequence dependencies such as semantic, layout, and visual contextual relationships within the bounding box words. For example, it is better able to learn continuous company name sequences like "ABC -> CO., -> LTD."

[0060] The second layer, a Bidirectional Long Short-Term Memory (BiLSTM) network, also has 256 hidden layers. This layer takes the output of the first-layer BiLSTM network as input and processes the local features of the first-layer BiLSTM output between bounding boxes. It primarily captures global sequence dependencies and reading order across bounding boxes, extracts global multimodal features, and fuses global contextual information extracted from the second-layer BiLSTM network to simulate the document reading order and overall structure. For example, it can learn the contextual relationship between the "SHIPPER" field box and the "CONSIGNEE" field box below it, understand the global semantic flow of the entire document, and ultimately generate a multimodal representation of the document that can be used for more accurate key information extraction and classification tasks.

[0061] Step 5, CRF Decoding and Key-Value Association: Using the multimodal representation of the document generated in Step 4 as input, the prediction results of the named entity dimension are extracted. A Conditional Random Field (CRF) structure is used to constrain the prediction results with context, obtaining the BIO tag corresponding to each token. This outputs the structured information (key-value pairs) and category label of each entity in the document. B, I, and O tags are concepts in named entity recognition. B represents the start of a named entity, I represents the middle part of a named entity, and O represents an entity unrelated to the named entity.

[0062] In one embodiment, step 5 specifically includes:

[0063] Step 51: Input the feature sequence output from the second-layer BiLSTM into the Conditional Random Field (CRF) layer for decoding. The CRF layer is a discriminative model that predicts the conditional probability distribution of another set of random condition variables given a set of input random variables. The CRF layer defines label transition constraints, such as automatically learning during training that "B-PER" can only be followed by "I-PER" or "O", but not "B-LOC"; "O" cannot be followed by "I-xxx", etc. These constraints effectively prevent the occurrence of illegal label sequences (such as [I-PER, I-PER, O]), ensuring the legality of the output labels. The CRF outputs the BIO label for each word, such as [B-SHIPPER, I-SHIPPER, I-SHIPPER, O, B-CONSIGNEE, ...], thereby enabling the prediction and classification of named entity attributes at the word level after segmentation. This step uses the CRF layer to capture the dependencies between labels and automatically learns the constraint rules during training, thus ensuring the consistency and accuracy of the prediction results.

[0064] Step 52: Merge consecutive tokens with the same entity label into a single entity value. For example, merge the tokens ['ABC', 'CO.', 'LTD.'] corresponding to [B-SHIPPER, I-SHIPPER, I-SHIPPER] into the value "ABCCO. LTD.". BertTokenizer is used as the tokenizer. BertTokenizer is chosen because its segmentation performance is better than other segmentation tools in this task.

[0065] Based on a predefined list of entity categories, including [SHIPPER, CONSIGNEE, NOTIFY_PARTY, PORT_OF_LOADING, PORT_OF_DISCHARGE] and their corresponding *-values, key-value pairings are performed. The specific pairing strategy is as follows: traverse all identified entities. If an entity's label is a key such as SHIPPER, find the category with the highest semantic similarity in the subsequent text, using it as the classification result for the nearest neighbor with the corresponding value, such as SHIPPER-value, and pair them to form key-value pairs. The final output is a structured classification result.

[0066] This invention also provides a device for extracting ocean bill of lading information by integrating reading order and multimodality, which includes a document processing module, a multimodal feature extraction module, a graph structure modeling module, a sequence modeling module, and an information decoding module, wherein:

[0067] The document processing module is used to perform optical character recognition (OCR) processing on the shipping manifest document image to obtain each text segment and its corresponding bounding box coordinates; based on the bounding box coordinates, all text segments are sorted according to a preset reading order to obtain a sorted text sequence.

[0068] The multimodal feature extraction module is used to extract multimodal features from the text sequence output by the document processing module using the LayoutLM model, and obtain the word-level feature representation in each text segment. The multimodal features include text semantic features, two-dimensional position embedding features, and aligned visual features.

[0069] The graph structure modeling module is used to construct a fully connected graph structure for all words inside the bounding box of each text segment, and introduces the graph attention network GAT to promote information interaction and feature enhancement between words inside the bounding box.

[0070] The sequence modeling module is used to perform multi-level sequence modeling using a two-layer bidirectional long short-term memory network (BiLSTM): the first layer of the BiLSTM network is used to capture the sequence dependencies between internal words within the same bounding box; the second layer of the BiLSTM network is used to capture the sequence dependencies between different bounding boxes to model the global reading order.

[0071] The information decoding module is used to input the feature sequence output by the second layer BiLSTM into the conditional random field (CRF) layer to perform sequence prediction of named entity tags, obtain the BIO tag corresponding to each word, and then extract the structured key-value pair information.

[0072] Compared to typical rich documents, ocean waybill documents exhibit more complex and diverse document structures and layouts. To address the challenges of semantic similarity or content ambiguity in rich document key information extraction, the significant impact of the complex layout of ocean waybills on visual information extraction accuracy, and the influence of text order on OCR recognition results, this paper proposes a method for ocean waybill information extraction that integrates reading order and multimodal approaches. In visual rich document images, text is typically presented in key-value pairs, where the "key" semantically determines the category of the "value." For example, a document may contain multiple structurally similar but semantically different fields, such as "shipper information," "consignee information," "shipping location," and "consignee location." These categories often appear as phrases rather than complete sentences, lacking rich contextual semantic information, making it difficult for traditional classification methods based on sequence labeling or named entity recognition to effectively distinguish them. In contrast, the key-value pair-based extraction method aligns better with human reading comprehension logic. By identifying the semantic content of the "key" and combining it with the spatial layout and contextual relationships of its corresponding "value," category determination and information classification can be achieved more accurately.

[0073] like Figures 3-4 The image shows a sample shipping manifest. Rectangles represent text information and location information, with the location determined by the top-left and bottom-right points of the rectangles. Labels indicate the corresponding predicted categories, displayed above the corresponding rectangles. As you can see in this image, the top represents the key, and the bottom represents the value. However, a large amount of data consists of left-key and right-value pairs, which is common in semi-structured or multi-structured documents. We achieve the final relationship extraction by classifying and predicting each part.

[0074] This invention comprehensively considers text, waybill layout structure information, and visual features for the identification and extraction of key-value pairs in ocean waybills, achieving better recognition and extraction results. From the OCR recognition results, the dataset is preprocessed. Since the OCR results lack label information, 23 predefined labels need to be manually added, including consignee, shipper, port of destination, port of discharge, consignee-value, shipper-value, port of destination-value, and port of discharge-value, corresponding to the key label and its corresponding value label, respectively. Content not included in the predefined labels is defined as an O label. To fuse textual, location, and image information, this invention employs the LayoutLM model. LayoutLM uses the standard BERT network structure and is pre-trained on a large-scale document dataset. The resulting LayoutLM model has achieved impressive results in various downstream document analysis tasks. LayoutLM introduces 2D positional embedding, which differs from the standard BERT model's positional embedding in that it uses four embedding layers corresponding to the four positions of the text block (top left (x0, y0) and bottom right (x1, y1)) and two embedding index tables corresponding to the x and y directions. After obtaining the LayoutLM embedding, it is then fused with the text-aligned image to better utilize visual information. However, the classification performance of LayoutLM depends on the order of text recognition by OCR. Therefore, to address this issue, we use two layers of BiLSTM to construct the reading order: one BiLSTM captures multimodal information within the bounding boxes, and the other captures multimodal information between the bounding boxes. Finally, a CRF layer is used for class prediction.

[0075] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting ocean bill of lading information that integrates reading order and multimodal processing, characterized in that, include: Step 1: Perform optical character recognition (OCR) processing on the shipping manifest document image to obtain each text segment and its corresponding bounding box coordinates; based on the bounding box coordinates, sort all text segments according to a preset reading order to obtain a sorted text sequence. Step 2: Use the LayoutLM model to extract multimodal features from the text sequence output in Step 1 to obtain the word-level feature representations in each text segment. The multimodal features include text semantic features, two-dimensional position embedding features, and aligned visual features. Step 3: For all words inside the bounding box of each text segment, construct a fully connected graph structure and introduce a graph attention network (GAT) to promote information interaction and feature enhancement between words inside the bounding box. Step 4: A two-layer bidirectional long short-term memory network (BiLSTM) is used for multi-level sequence modeling: the first layer of the BiLSTM network is used to capture the sequence dependencies between internal words within the same bounding box; the second layer of the BiLSTM network is used to capture the sequence dependencies between different bounding boxes to model the global reading order. Step 5: Input the feature sequence output from the second layer of BiLSTM in step 4 into the Conditional Random Field (CRF) layer to perform sequence prediction of named entity tags, obtain the BIO tag corresponding to each word, and then extract the structured key-value pair information.

2. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, The methods for obtaining the two-dimensional location embedding features in step 2 include: Based on the text bounding box coordinates, normalize them into virtual coordinates; Calculate the embedding representation of the virtual coordinates in the four dimensions of x-axis starting position, y-axis starting position, width w, and height h respectively; The four-dimensional embedding representations are summed to obtain the final two-dimensional position embedding feature.

3. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1 or 2, characterized in that, The methods for extracting visual features in step 2 include: The document images are used to extract features using a ResNet residual network pre-trained on the large-scale visual recognition dataset ImageNet to obtain a global visual feature map. Based on the bounding box coordinates of each text segment, the local visual features of the corresponding region are extracted from the global visual feature map through the Region of Interest Align (ROIAlign) operation. The local visual features are aligned and fused with the text and position features extracted by the LayoutLM model to obtain the aligned visual features.

4. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, The graph attention network in step 3 employs a multi-head self-attention mechanism to calculate the attention weights of each lexical node within the bounding box and all its neighboring nodes, thereby aggregating neighborhood information to update node features.

5. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, In step 5, the CRF layer automatically learns the constraint rules between BIO labels during training. The constraint rules include: the sentence beginning label is set to "B-" or "O"; the label2 in "I-label2" following "B-label1" is set to be the same as label1; "O" can be followed by labels other than "I-".

6. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, Step 5 is followed by a key-value association step: consecutive terms predicted to be the same entity name are merged into entity values, and the entity values ​​are associated with the corresponding key categories by calculating semantic similarity according to a predefined list of entity categories, forming complete key-value pairs.

7. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 6, characterized in that, The predefined entity categories include shipper, consignee, port of destination, and port of discharge. Each key category corresponds to a value category, which is identified by the suffix "-value".

8. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, The specific implementation of the preset reading order sorting in step 1 is as follows: first compare the ordinates of the top left corner of the text segment bounding box, and rank the one with the smaller ordinate first; if the ordinates are the same, compare the abscissas, and rank the one with the smaller abscissa first.

9. The method for extracting ocean bill of lading information by integrating reading order and multimodality according to claim 1, characterized in that, The LayoutLM model used in step 2 integrates the masking language model task and the sentence order prediction task during the training phase to learn the semantic and layout information of the document text.

10. A device for extracting ocean bill of lading information that integrates reading order and multimodal processing, characterized in that, include: The document processing module is used to perform optical character recognition (OCR) processing on the shipping manifest document image to obtain each text segment and its corresponding bounding box coordinates; based on the bounding box coordinates, all text segments are sorted according to a preset reading order to obtain a sorted text sequence. The multimodal feature extraction module is used to extract multimodal features from the text sequence output by the document processing module using the LayoutLM model, and obtain the feature representation at the word level in each text segment. The multimodal features include text semantic features, two-dimensional position embedding features, and aligned visual features. The graph structure modeling module is used to construct a fully connected graph structure for all words inside the bounding box of each text segment, and introduces the graph attention network GAT to promote information interaction and feature enhancement between words inside the bounding box. The sequence modeling module is used for multi-level sequence modeling using a two-layer bidirectional long short-term memory network (BiLSTM): the first layer of the BiLSTM network is used to capture the sequence dependencies between internal words within the same bounding box; the second layer of the BiLSTM network is used to capture the sequence dependencies between different bounding boxes to model the global reading order. The information decoding module is used to input the feature sequence output by the second layer BiLSTM into the conditional random field (CRF) layer to perform sequence prediction of named entity tags, obtain the BIO tag corresponding to each word, and then extract the structured key-value pair information.

Images