Complex instruction multi-modal understanding method and system based on two-stage segmentation of YOLO

Abstract

The application discloses a complex specification multi-modal understanding method and system based on YOLO two-stage segmentation, comprising the following steps: performing pre-processing on an input specification image; inputting the pre-processed image into a coarse-grained detection stage of an improved YOLOv8 network to generate a coarse-grained candidate region set; inputting the coarse-grained candidate region set into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure; constructing a layout hierarchical relationship structure based on an accurate layout segmentation result; according to the layout hierarchical relationship structure, performing multi-modal feature extraction on the accurate layout segmentation result, and calculating a matching weight; inputting a graphic-text semantic alignment result and the layout hierarchical relationship structure into a logical structure modeling module together to construct a layout element graph; and generating a knowledge graph semantic output result according to a structured logical reasoning result. The application realizes automatic structured understanding of a complex specification by adopting YOLO two-stage segmentation and multi-modal semantic alignment.

Description

Technical Field

[0001] This invention relates to the field of document image processing and multimodal semantic understanding technology, and in particular to a method and system for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation. Background Technology

[0002] In the field of document image analysis, instruction manuals present significant challenges to automated processing due to their complex layout and dense combination of text and images. Existing technologies mostly employ single-stage detection models based on convolutional neural networks to identify text blocks, image blocks, and table regions. However, these models are prone to false positives and false negatives when dealing with small text sizes, cross-region layouts, and irregular chart structures in instruction manuals. This results in insufficient accuracy in page segmentation, making it difficult to meet the requirements of subsequent content analysis.

[0003] In terms of semantic understanding, traditional methods typically rely on encoding OCR text to perform semantic reasoning. However, there is a lack of effective correlation between visual and textual information, making it difficult to establish a matching relationship between image content and corresponding textual descriptions. For common illustrations, steps, and explanations in instruction manuals, existing technologies often cannot achieve accurate alignment through single-modal modeling. Furthermore, existing hierarchical judgment methods based on spatial layout rely on simple rules and cannot combine the size proportions and semantic similarity of page elements for comprehensive judgment, making it difficult to construct a stable logical hierarchical structure between titles, body text, and charts.

[0004] In terms of structured representation, traditional document parsing systems largely rely on rule-driven content extraction methods, which are less adaptable to manuals with different formats and layout styles, and cannot handle cross-level references or multi-paragraph related content. Without a unified structural modeling mechanism, the system struggles to output knowledge representations that can be directly used for structured analysis.

[0005] Therefore, how to provide a multimodal understanding method and system for complex instruction manuals based on YOLO two-stage segmentation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a multimodal understanding method and system for complex instruction manuals based on YOLO two-stage segmentation. This invention utilizes an improved coarse-grained detection and fine-grained segmentation structure of YOLOv8 to achieve high-precision recognition of instruction manual layout elements. It combines a visual feature extraction model and a text semantic encoding model to achieve multimodal alignment of text and images, and uses a graph-structured reasoning network and knowledge graph construction mechanism to realize the structured expression of the instruction manual content. This invention can simultaneously handle complex layout structures, cross-modal semantic associations, and multi-level logical relationships, possessing advantages such as high parsing accuracy, strong structuring capabilities, and adaptability to different instruction manual formats.

[0007] The multimodal understanding method and system for complex instruction manuals based on YOLO two-stage segmentation according to embodiments of the present invention includes the following steps:

[0008] The input instruction manual image is subjected to image enhancement, adaptive resolution adjustment, and noise reduction to obtain a preprocessed image;

[0009] The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, and a coarse-grained candidate region set is generated using a dynamic anchor box matching detection head.

[0010] The coarse-grained candidate region set is input into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, and end-to-end optimization is performed to obtain accurate page segmentation results.

[0011] A hierarchical structure of page layout is constructed based on the spatial position, size ratio, and textual semantic similarity of each page element in the accurate page layout segmentation results.

[0012] Based on the hierarchical structure of the page layout, multimodal features are extracted from the precise page segmentation results, matching weights are calculated, and the semantic alignment results of the text and images are output.

[0013] The semantic alignment results of the text and images, along with the hierarchical relationship structure of the layout, are input into the logical structure modeling module to construct the layout element diagram and generate structured logical reasoning results.

[0014] Generate a semantic output of a knowledge graph containing entities, relationships, and attributes based on the results of structured logical reasoning.

[0015] Optionally, the step of inputting the preprocessed image into the coarse-grained detection stage of the improved YOLOv8 network, and generating a coarse-grained candidate region set using a dynamic anchor box matching detection head, specifically includes:

[0016] The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, which includes a backbone feature extraction layer, a bidirectional feature fusion layer, and a dynamic anchor box matching detection head.

[0017] In the backbone feature extraction layer, the C2f-CBAM module is used to perform multi-scale semantic feature extraction on the preprocessed image. Through the C2f structure and CBAM attention unit, multi-scale feature maps are obtained.

[0018] In the bidirectional feature fusion layer, the Bi-FPN structure is used to fuse the multi-scale feature maps output by the backbone feature extraction layer to obtain a fused feature set;

[0019] In the dynamic anchor box matching detection head, the prediction and matching of candidate regions are performed using the fused feature set to obtain the candidate region set and the corresponding category probability vector and confidence score;

[0020] The candidate region set is filtered based on the confidence threshold and the non-maximum suppression threshold, and non-maximum suppression is performed to obtain the retained candidate regions;

[0021] The retained candidate regions are divided into text block regions, table regions, title regions, or image regions according to the category probability vector, forming a coarse-grained candidate region set.

[0022] Optionally, the step of inputting the coarse-grained candidate region set into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, performing end-to-end optimization, and obtaining accurate page segmentation results specifically includes:

[0023] The coarse-grained candidate region set is input as the segmentation target into the fine-grained segmentation network based on the improved YOLOv8 feature extraction structure. At the same time, the multi-scale feature map of the backbone feature extraction layer generated by the improved YOLOv8 network in the coarse-grained detection stage and the fused feature set output by the bidirectional feature fusion layer are also input into the fine-grained segmentation network to serve as the source of feature cropping for candidate regions and input information for cross-stage feature sharing.

[0024] For each candidate region in the coarse-grained candidate region set, local feature regions corresponding to its spatial range are extracted on the multi-scale feature map of the fused feature set according to the center position, width, and height of the candidate region. Specifically, at each feature scale, the geometric position of the candidate region in the original image is mapped to the feature map coordinate system of the corresponding scale, and a rectangular feature block matching the candidate region is cropped according to the proportion of the candidate region. To ensure that the feature blocks of different scales can maintain shape consistency, a region alignment operation using bilinear interpolation is performed on the cropped feature blocks to ensure that they maintain a strict correspondence with the candidate region in terms of spatial resolution and sampling position.

[0025] After the above cropping and alignment processes, a set of regional features is obtained, which consists of regional features corresponding to multiple scales. Each candidate region corresponds to a set of regional features obtained by alignment and cropping on feature maps at different scales.

[0026] The set of regional features obtained after region cropping and alignment is input into the encoding end of the fine-grained segmentation network layer by layer according to the order of the encoding layers. In each encoding layer, deformable convolution processing, including offset sampling and convolution calculation, is performed on the input regional features according to the structural configuration of the encoding layer.

[0027] Specifically, when entering the current encoding layer, the sampling position of the input region features is first adjusted according to the learnable offset parameter corresponding to the encoding layer, so that the sampling points can adapt to the geometric changes inside the candidate region. Then, convolution calculation is performed at the adjusted sampling position to generate the corresponding encoded features.

[0028] The above process is carried out sequentially in each layer of the encoding end. Each layer continues to perform offset sampling and convolution processing based on the output of the previous layer. Through layer-by-layer processing, the set of encoded features corresponding to all encoding layers is finally obtained.

[0029] The encoded feature set is input into the attention-enhanced U-Net decoder, which performs upsampling and layer-by-layer feature fusion. The multi-scale feature map of the coarse-grained detection stage is input to the corresponding decoding layer through a skip connection through a cross-stage feature sharing mechanism, so that the decoder obtains a decoded feature set composed of multi-scale encoded features and cross-stage shared features.

[0030] After completing feature upsampling and layer-by-layer feature fusion at the decoding end, the resulting set of decoded features is sequentially input into the pixel-level classification unit of the fine-grained segmentation network to determine the category of each candidate region's corresponding decoded features pixel by pixel.

[0031] Specifically, for each candidate region, based on its spatial range in the input image, the decoded features are mapped back to a feature space with the same size as the candidate region, and the probability value of each pixel position belonging to different layout categories is calculated in the feature space.

[0032] Through the above pixel-level classification operation, a segmentation mask image with the same size as the candidate region is generated for each candidate region. Each pixel in the segmentation mask image gives the probability result of its category in text block, table, title area or image area. The segmentation mask images corresponding to all candidate regions are combined to form a segmentation mask set.

[0033] After generating the segmentation mask, the fine-grained segmentation network is trained end-to-end and its parameters are updated using a joint loss mechanism that includes detection box loss and segmentation mask loss.

[0034] The joint loss mechanism includes calculating the detection box loss for each candidate region during training based on the center position, width, and height of its detection box, which is used to measure the difference between the geometric parameters of the predicted candidate region and the true labeled region.

[0035] For each candidate region, the segmentation mask image is used to calculate the segmentation mask loss based on the correspondence between the predicted class probability of each pixel in the mask and the true class, which is used to measure the accuracy of pixel-level classification prediction.

[0036] The detection box loss and the segmentation mask loss are weighted and summed according to preset loss weights to form a joint loss value for optimizing the entire fine-grained segmentation network. The joint loss value is used as a training feedback signal to update all learnable parameters of the network through gradient descent, thereby improving both detection accuracy and segmentation accuracy. After end-to-end training is completed, the segmentation mask set output by the fine-grained segmentation network is regarded as the accurate layout segmentation result.

[0037] Optionally, the step of inputting the encoded feature set into the attention-enhanced U-Net decoder, performing upsampling and layer-by-layer feature fusion to obtain the decoded feature set specifically includes:

[0038] The encoded feature set is input into the attention-enhanced U-Net decoder in the order of the encoding layers, so that the encoded features output by each encoding layer enter the corresponding decoding layer as the initial input features of the decoding layer.

[0039] In each decoding layer, an upsampling operation is performed on the input encoded features to gradually enlarge the spatial size of the encoded features according to the decoding layer. Convolution and channel attention processing are then performed sequentially on the upsampled encoded features. The channel attention processing generates intermediate decoded features for the current decoding layer by weighting the response of each channel.

[0040] After completing the channel attention processing, spatial attention processing is performed on the intermediate decoding features in the current decoding layer. By weighting the spatial position response of the intermediate decoding features, attention-enhanced decoding features of the current decoding layer are obtained.

[0041] While performing upsampling, convolution processing, and channel attention and spatial attention processing, cross-stage shared features corresponding to the current decoding layer are selected from the multi-scale feature map generated in the coarse-grained detection stage. The cross-stage shared features include the multi-scale feature map output by the backbone feature extraction layer and the mid-level features in the fusion feature set output by the bidirectional feature fusion layer.

[0042] Based on the spatial size of the current decoding layer, the selected cross-stage shared features are adjusted by interpolation to maintain the same resolution as the attention enhancement feature map of the current decoding layer, so that they can be concatenated with the features of the current decoding layer at the same spatial scale. After the size adjustment is completed, the obtained cross-stage shared features are retained for the skip connection operation in the next step.

[0043] The attention-enhanced decoding features are concatenated with the corresponding cross-stage shared features using a skip connection method to form a fused feature in the channel dimension. The fused feature is then subjected to convolution and feature refinement to generate the output feature of the decoding layer.

[0044] By using the output features as input to the next decoding layer, the attention-enhanced U-Net decoder sequentially outputs features from multiple decoding layers, ultimately forming a set of decoded features composed of output features from different decoding layers.

[0045] Optionally, the construction of the page layout hierarchy based on the spatial position, size ratio, and textual semantic similarity of each page element in the accurate page layout segmentation results specifically includes:

[0046] For each page element in the accurate page segmentation result, read the segmentation mask corresponding to the page element, and determine the bounding rectangle region of the page element in the original instruction manual image based on the distribution range of all foreground pixels in the segmentation mask.

[0047] Specifically, the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate of all foreground pixels in the page element segmentation mask are calculated respectively. The average of the minimum horizontal coordinate and the maximum horizontal coordinate is used as the horizontal center position of the page element, and the average of the minimum vertical coordinate and the maximum vertical coordinate is used as the vertical center position of the page element.

[0048] The width of the page element is obtained by subtracting the minimum horizontal coordinate from the maximum horizontal coordinate, and the height of the page element is obtained by subtracting the minimum vertical coordinate from the maximum vertical coordinate. Thus, the horizontal center position and the vertical center position are defined together as the center position parameter of the page element, and the width and height are defined together as the size parameter of the page element. The spatial position parameter and the size parameter of the page element are combined to form a spatial feature vector.

[0049] Text encoding processing is performed on the text content corresponding to each page element. Specifically, this includes: first, extracting the text paragraphs corresponding to the spatial position of the page element as the input text sequence; then, semantic encoding is performed on the text sequence to generate a vector representation for the meaning of the text, which is defined as the text encoding vector of the page element.

[0050] For any two different page elements, obtain their corresponding text encoding vectors and use the cosine similarity formula to calculate their semantic similarity. The semantic similarity is used to quantify the semantic closeness of the text content between the two page elements.

[0051] Based on spatial feature vectors and semantic similarity, pairwise relationships are determined for all page elements. According to the comprehensive relationship of spatial position, size ratio and text semantic similarity, a preliminary relationship set consisting of parent-child relationship, sibling relationship and reference relationship between page elements is constructed.

[0052] Specifically, for every two page elements, firstly, the horizontal center position, vertical center position, width, and height represented by the spatial feature vectors are compared to determine their relative positional relationship and size ratio on the page; then, the semantic similarity between the two is combined to determine the semantic relevance of the text content; based on the combined results of spatial position, size ratio, and semantic similarity, each pair of page elements is labeled with its possible relationship type, including possible parent-child relationship, sibling relationship, or reference relationship, and the above judgment results are recorded as the preliminary relationship of the pair of page elements. By evaluating all page elements pairwise, a preliminary relationship set consisting of multiple preliminary relationships is formed.

[0053] For any two page elements in the preliminary relationship set, based on the hierarchical mapping determination criteria, it is comprehensively determined whether the two page elements have a hierarchical inclusion or subordinate relationship. The hierarchical mapping determination criteria specifically include:

[0054] If two page elements are arranged vertically, and the vertical distance between the bottom coordinates of the upper element and the top coordinates of the lower element is less than the first spatial threshold, then a vertical relationship is determined to exist.

[0055] If the horizontal positions of two page elements are basically the same, and the height of the upper element is less than that of the lower element, and the semantic similarity between the two texts is higher than the first semantic threshold, then the upper element is determined to be the title area and the lower element is the body text area.

[0056] If there is a spatial containment relationship between two page elements, and the area difference between the containing area and the contained area is less than a set ratio threshold, and the semantic similarity of their texts is within the second semantic threshold, then the containing area is determined to be the parent element, and the other area is its child content.

[0057] Based on the above hierarchical mapping determination conditions, the page elements in the preliminary relationship set are analyzed one by one to generate a complete hierarchical mapping relationship set.

[0058] In the set of hierarchical mapping relationships, the relationships that satisfy the condition of correspondence between title and body text are marked as title-body text relationships, the relationships that satisfy the condition of correspondence between chart area and explanatory text are marked as chart-explanatory relationships, and the relationships that satisfy the condition of correspondence between step text and step description are marked as step-description relationships.

[0059] Summarize the hierarchical mapping relationships between all the marked and filtered page elements to construct a complete page hierarchy structure.

[0060] Optionally, the step of extracting multimodal features from the precise page segmentation results based on the page layout hierarchy and calculating matching weights to output the image-text semantic alignment results specifically includes:

[0061] Based on the hierarchical structure of the page layout, determine the image regions and text content pairs that need to be semantically aligned;

[0062] The image region is preprocessed, including size normalization and channel normalization. The processed image region is then input into the Vision Transformer network to extract the visual semantic feature vector of the image region.

[0063] After the text content is segmented, interference symbols are removed, and the encoding format is unified, it is input into the BERT model to perform deep semantic representation of the text and obtain the contextual semantic feature vector of the text content.

[0064] The visual semantic feature vector and the text semantic feature vector are input together into the joint attention mechanism module to obtain the matching weights;

[0065] Based on the matching weights, the semantic coupling relationship between the image region and the text content is quantified and output to generate the image-text semantic alignment result.

[0066] Optionally, the step of inputting the image-text semantic alignment result and the page layout hierarchy structure into the logical structure modeling module to construct the page element diagram and generate structured logical reasoning results specifically includes:

[0067] Input the semantic alignment results of the text and images along with the hierarchical relationship structure of the page layout into the logical structure modeling module to initialize the page element diagram;

[0068] Based on the semantic alignment results of the text and images and the hierarchical structure of the layout, the edge weights between nodes are calculated, a weighted layout element graph is constructed, and a set of nodes and a set of edges are formed.

[0069] Perform contextual semantic aggregation on each node in the weighted layout element graph to output a structure-aware semantic representation;

[0070] The structure-aware semantic representation sequence is input into the Transformer model, and the long-range dependencies between nodes are modeled through the self-attention mechanism. Content sequence reasoning is then performed to generate a structured text sequence.

[0071] Based on the content structure identified in the structured text sequence, each page element is mapped to a predefined logical structure template, and the structured logical reasoning result is output.

[0072] Optionally, generating the knowledge graph semantic output result containing entities, relations, and attributes based on the structured logic reasoning result specifically includes:

[0073] Each page element node identified in the structured logic reasoning results is used as a candidate entity set. Semantic keywords are extracted from each page element node to construct an initial entity set.

[0074] Based on the semantic keywords of each page element node in the initial entity set, and combined with the node connection edge information in the structured logic reasoning results, the semantic similarity between any two entities is calculated. When the semantic similarity exceeds a set threshold, it is determined that there is a semantic relationship and the entity is added to the relationship set.

[0075] For each pair of entities in the relation set, extract relation description words, map the entity pairs to semantic relation types, and generate structured relation data in the form of triples;

[0076] Extract attribute information related to each entity from the semantic alignment results of the graph and text, and attach each attribute information to the corresponding entity node in the form of key-value pairs to construct the graph node structure.

[0077] By integrating the graph node structure with entity relation triples, a knowledge graph semantic output result containing entities, attributes, and semantic relations is constructed.

[0078] The complex instruction manual multimodal understanding system based on YOLO two-stage segmentation according to an embodiment of the present invention includes the following modules:

[0079] The document preprocessing module is used to perform image enhancement, resolution adaptive adjustment, and noise reduction on the input instruction manual image to obtain a preprocessed image;

[0080] The YOLO two-stage layout segmentation module is used to perform coarse-grained detection and fine-grained segmentation on the preprocessed image. It adopts an improved YOLOv8 feature extraction structure, combined with backbone network sharing and attention-enhanced U-Net decoder, to identify various layout element regions in the instruction manual and generate accurate segmentation results.

[0081] The layout structure modeling and hierarchy recognition module is used to construct the hierarchical relationship structure of the layout based on the position, size and semantic information of each layout element in the accurate segmentation results.

[0082] The multimodal feature extraction and alignment module is used to extract visual and linguistic features from the image regions and OCR text content in the accurate layout segmentation results, and to perform image-text semantic matching and alignment through a joint attention mechanism to generate image-text aligned output results.

[0083] The structured logic modeling and reasoning module is used to model the semantic alignment results of text and graphics together with the hierarchical relationship structure of the layout into a layout element diagram, and generate structured logic reasoning results.

[0084] The knowledge graph construction module is used to generate semantic output results of knowledge graphs containing entities, relationships, and attributes based on the results of structured logical reasoning.

[0085] The beneficial effects of this invention are:

[0086] This invention introduces an improved YOLO two-stage page segmentation mechanism, achieving refined identification of various page elements in instruction manuals. This allows for accurate segmentation of small text areas, dense paragraph areas, and complex charts within the same detection framework, thereby improving the input quality for subsequent semantic processing steps. Based on the accurate page segmentation results, this invention further constructs a hierarchical relationship combining spatial location, size proportion, and semantic similarity. This enables the systematic identification of the subordinate structure between titles, body text, step descriptions, and chart illustrations, avoiding the instability in hierarchical judgment caused by relying on a single rule.

[0087] In terms of multimodal semantic processing, this invention integrates visual semantic features extracted by Vision Transformer with textual semantic features encoded by the BERT model, and utilizes a joint attention mechanism to calculate image-text matching weights. This enables the semantic correspondence between image regions and text content to be established in a unified representation space, thereby effectively enhancing the reliability of image-text association analysis. Based on the aforementioned image-text semantic alignment and hierarchical structure, this invention constructs a knowledge graph centered on entities, attributes, and semantic relationships, allowing complex instruction manual content to be presented in a structured manner, facilitating subsequent automated retrieval and content understanding.

[0088] This invention improves the overall accuracy of layout parsing, the quality of text and image alignment, and the ability to express structure in complex instruction manuals through multi-stage detection, cross-modal semantic modeling, and logical structure reasoning. It can adapt to various typesetting styles and document structures, thereby enhancing the usability and consistency of instruction manual content in automated processing systems. Attached Figure Description

[0089] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0090] Figure 1 This is a flowchart of the method and system for understanding complex instruction manuals based on YOLO two-stage segmentation proposed in this invention.

[0091] Figure 2 This is a schematic diagram of the structure of the two-stage layout segmentation module in the complex instruction manual multimodal understanding method based on YOLO two-stage segmentation proposed in this invention.

[0092] Figure 3 This is a schematic diagram of the system structure of the complex specification multimodal understanding method and system based on YOLO two-stage segmentation proposed in this invention. Detailed Implementation

[0093] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0094] refer to Figure 1 and Figure 2 A multimodal understanding method and system for complex instruction manuals based on YOLO two-stage segmentation, including the following steps:

[0095] The input instruction manual image is subjected to image enhancement, adaptive resolution adjustment, and noise reduction to obtain a preprocessed image;

[0096] The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, and a coarse-grained candidate region set is generated using a dynamic anchor box matching detection head.

[0097] The coarse-grained candidate region set is input into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, and end-to-end optimization is performed to obtain accurate page segmentation results.

[0098] A hierarchical structure of page layout is constructed based on the spatial position, size ratio, and textual semantic similarity of each page element in the accurate page layout segmentation results.

[0099] Based on the hierarchical structure of the page layout, multimodal features are extracted from the precise page segmentation results, matching weights are calculated, and the semantic alignment results of the text and images are output.

[0100] The semantic alignment results of the text and images, along with the hierarchical relationship structure of the layout, are input into the logical structure modeling module to construct the layout element diagram and generate structured logical reasoning results.

[0101] Generate a semantic output of a knowledge graph containing entities, relationships, and attributes based on the results of structured logical reasoning.

[0102] In this embodiment, the step of inputting the preprocessed image into the coarse-grained detection stage of the improved YOLOv8 network, and generating a coarse-grained candidate region set using a dynamic anchor box matching detection head, specifically includes:

[0103] The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, which includes a backbone feature extraction layer, a bidirectional feature fusion layer, and a dynamic anchor box matching detection head.

[0104] In the backbone feature extraction layer, the C2f-CBAM module is used to perform multi-scale semantic feature extraction on the preprocessed image. Through the C2f structure and CBAM attention unit, multi-scale feature maps are obtained.

[0105] The C2f-CBAM module receives an initial feature map generated from a preprocessed image and performs three convolution operations and one residual connection operation sequentially through a C2f structure. The first convolutional layer is used to extract low-level spatial features, the second convolutional layer is used to extract mid-level semantic features, and the third convolutional layer is used to generate high-level feature representations. The residual connection is used to maintain gradient propagation and feature continuity between input and output.

[0106] The feature map processed by the C2f structure is input into the CBAM attention unit. In the channel attention stage, the response weights of each channel feature are calculated, and the semantically significant channels are enhanced. In the spatial attention stage, the spatial dimension of the same feature map is weighted to suppress the background area and highlight the text, table, title area and image area.

[0107] The feature map output after channel and spatial attention calculation contains semantic and spatial information at different levels and maintains feature scale consistency, serving as the multi-scale feature map for the subsequent bidirectional feature fusion layer in the improved YOLOv8 network.

[0108] In the bidirectional feature fusion layer, the Bi-FPN structure is used to fuse the multi-scale feature maps output by the backbone feature extraction layer to obtain a fused feature set;

[0109] The bidirectional feature fusion layer receives multi-scale feature maps from different depths of the backbone feature extraction layer, which correspond to the high-resolution features of the shallow layer, the semantic structure features of the middle layer, and the abstract semantic features of the deep layer, respectively.

[0110] The Bi-FPN structure establishes feature propagation paths in both top-down and bottom-up directions. First, deep features are upsampled and then fused in a weighted manner to supplement the shallow details with deep semantic information. Then, shallow features are downsampled and then passed to adjacent deep feature maps to feed back the spatial details of the shallow layers into the deep semantic representation.

[0111] In each feature fusion process, the Bi-FPN structure assigns learnable weight coefficients to the input multi-scale feature maps, with higher weights indicating a greater contribution to the fusion.

[0112] All input multi-scale feature maps were normalized before fusion to ensure the consistency of numerical distribution of features at different levels.

[0113] The fused multi-scale feature map is output after one activation operation and convolution refinement operation, resulting in a fused feature set with fully interactive multi-level feature information. The fused feature set retains the edge and structural details in the shallow features, as well as the semantic and contextual information in the deep features.

[0114] In the dynamic anchor box matching detection head, the prediction and matching of candidate regions are performed using a fused feature set. Specifically, the dynamic anchor box matching detection head receives multi-scale feature maps from the backbone feature extraction layer, establishes a set of candidate anchor points for the spatial location of each feature layer, and assigns different scales and aspect ratios to each anchor point according to the resolution of the multi-scale feature map. The class probability distribution and bounding box parameters corresponding to each anchor point are calculated through convolutional classification branch and regression branch respectively to obtain preliminary candidate region results.

[0115] To improve detection accuracy, the detection head adopts a dynamic anchor box matching strategy, which dynamically adjusts the matching relationship based on the degree of overlap between the candidate box and the real target area and the classification confidence.

[0116] When the intersection-union ratio (IU) of a candidate box with the target region is higher than a set threshold, the candidate box is judged as a positive sample. When the IU is lower than the threshold, the candidate box is regarded as a negative sample and is suppressed during training. After dynamic matching and parameter regression optimization, the dynamic anchor box matching detection head outputs a set of candidate regions. Each candidate region contains center coordinates, width and height, and outputs the corresponding class probability vector and confidence score, generating a candidate region set composed of multiple candidate regions.

[0117] In the candidate region selection stage, the set of candidate regions output by the dynamic anchor frame matching detection head is subjected to confidence judgment and overlap suppression processing to generate the final coarse-grained candidate region set.

[0118] Specifically, firstly, the confidence score corresponding to each candidate region is read. When the confidence score of a candidate region is lower than the preset threshold, the candidate region is considered an invalid detection and is removed. Candidate regions with confidence scores higher than the threshold are retained to enter the next step of screening.

[0119] Subsequently, the spatial overlap ratio between candidate regions of the same category is calculated. When the overlap ratio of two candidate regions exceeds the set non-maximum suppression threshold, only the region with higher confidence is retained, and the region with lower confidence is removed from the results to avoid the same object being detected repeatedly.

[0120] After confidence screening and non-maximum suppression, the remaining candidate region set is considered as a valid detection result. Based on the category probability vector of each candidate region, the candidate regions are divided into four types of layout elements: text block regions, table regions, title regions, and image regions.

[0121] All the candidate regions that are retained and have been classified and labeled together constitute a coarse-grained candidate region set, which accurately describes the main layout of the instruction manual pages.

[0122] In this embodiment, the step of inputting the coarse-grained candidate region set into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, performing end-to-end optimization, and obtaining accurate page segmentation results specifically includes:

[0123] The coarse-grained candidate region set is input as the segmentation target into the fine-grained segmentation network based on the improved YOLOv8 feature extraction structure. At the same time, the multi-scale feature map of the backbone feature extraction layer generated by the improved YOLOv8 network in the coarse-grained detection stage and the fused feature set output by the bidirectional feature fusion layer are also input into the fine-grained segmentation network to serve as the source of feature cropping for candidate regions and input information for cross-stage feature sharing.

[0124] For each candidate region in the coarse-grained candidate region set, local feature regions corresponding to its spatial range are extracted on the multi-scale feature map of the fused feature set according to the center position, width, and height of the candidate region. Specifically, at each feature scale, the geometric position of the candidate region in the original image is mapped to the feature map coordinate system of the corresponding scale, and a rectangular feature block matching the candidate region is cropped according to the proportion of the candidate region. To ensure that the feature blocks of different scales can maintain shape consistency, a region alignment operation using bilinear interpolation is performed on the cropped feature blocks to ensure that they maintain a strict correspondence with the candidate region in terms of spatial resolution and sampling position.

[0125] After the above cropping and alignment processes, a set of regional features is obtained, which consists of regional features corresponding to multiple scales. Each candidate region corresponds to a set of regional features obtained by alignment and cropping on feature maps at different scales.

[0126] The set of regional features obtained after region cropping and alignment is input into the encoding end of the fine-grained segmentation network layer by layer according to the order of the encoding layers. In each encoding layer, deformable convolution processing, including offset sampling and convolution calculation, is performed on the input regional features according to the structural configuration of the encoding layer.

[0127] Specifically, when entering the current encoding layer, the sampling position of the input region features is first adjusted according to the learnable offset parameter corresponding to the encoding layer, so that the sampling points can adapt to the geometric changes inside the candidate region. Then, convolution calculation is performed at the adjusted sampling position to generate the corresponding encoded features.

[0128] The above process is carried out sequentially in each layer of the encoding end. Each layer continues to perform offset sampling and convolution processing based on the output of the previous layer. Through layer-by-layer processing, the set of encoded features corresponding to all encoding layers is finally obtained.

[0129] The encoded feature set is input into the attention-enhanced U-Net decoder, which performs upsampling and layer-by-layer feature fusion. The multi-scale feature map of the coarse-grained detection stage is input to the corresponding decoding layer through a skip connection through a cross-stage feature sharing mechanism, so that the decoder obtains a decoded feature set composed of multi-scale encoded features and cross-stage shared features.

[0130] After completing feature upsampling and layer-by-layer feature fusion at the decoding end, the resulting set of decoded features is sequentially input into the pixel-level classification unit of the fine-grained segmentation network to determine the category of each candidate region's corresponding decoded features pixel by pixel.

[0131] Specifically, for each candidate region, based on its spatial range in the input image, the decoded features are mapped back to a feature space with the same size as the candidate region, and the probability value of each pixel position belonging to different layout categories is calculated in the feature space.

[0132] Through the above pixel-level classification operation, a segmentation mask image with the same size as the candidate region is generated for each candidate region. Each pixel in the segmentation mask image gives the probability result of its category in text block, table, title area or image area. The segmentation mask images corresponding to all candidate regions are combined to form a segmentation mask set.

[0133] After generating the segmentation mask, the fine-grained segmentation network is trained end-to-end and its parameters are updated using a joint loss mechanism that includes detection box loss and segmentation mask loss.

[0134] The joint loss mechanism includes calculating the detection box loss for each candidate region during training based on the center position, width, and height of its detection box, which is used to measure the difference between the geometric parameters of the predicted candidate region and the true labeled region.

[0135] For each candidate region, the segmentation mask image is used to calculate the segmentation mask loss based on the correspondence between the predicted class probability of each pixel in the mask and the true class, which is used to measure the accuracy of pixel-level classification prediction.

[0136] The detection box loss and the segmentation mask loss are weighted and summed according to preset loss weights to form a joint loss value for optimizing the entire fine-grained segmentation network. The joint loss value is used as a training feedback signal to update all learnable parameters of the network through gradient descent, thereby improving both detection accuracy and segmentation accuracy. After end-to-end training is completed, the segmentation mask set output by the fine-grained segmentation network is regarded as the accurate layout segmentation result.

[0137] In this embodiment, the step of inputting the encoded feature set into the attention-enhanced U-Net decoder and performing upsampling and layer-by-layer feature fusion to obtain the decoded feature set specifically includes:

[0138] The encoded feature set is input into the attention-enhanced U-Net decoder in the order of the encoding layers, so that the encoded features output by each encoding layer enter the corresponding decoding layer as the initial input features of the decoding layer.

[0139] In each decoding layer, an upsampling operation is performed on the input encoded features to gradually enlarge the spatial size of the encoded features according to the decoding layer. Convolution and channel attention processing are then performed sequentially on the upsampled encoded features. The channel attention processing generates intermediate decoded features for the current decoding layer by weighting the response of each channel.

[0140] After completing the channel attention processing, spatial attention processing is performed on the intermediate decoding features in the current decoding layer. By weighting the spatial position response of the intermediate decoding features, attention-enhanced decoding features of the current decoding layer are obtained.

[0141] While performing upsampling, convolution processing, and channel attention and spatial attention processing, cross-stage shared features corresponding to the current decoding layer are selected from the multi-scale feature map generated in the coarse-grained detection stage. The cross-stage shared features include the multi-scale feature map output by the backbone feature extraction layer and the mid-level features in the fusion feature set output by the bidirectional feature fusion layer.

[0142] Based on the spatial size of the current decoding layer, the selected cross-stage shared features are adjusted by interpolation to maintain the same resolution as the attention enhancement feature map of the current decoding layer, so that they can be concatenated with the features of the current decoding layer at the same spatial scale. After the size adjustment is completed, the obtained cross-stage shared features are retained for the skip connection operation in the next step.

[0143] The attention-enhanced decoding features are concatenated with the corresponding cross-stage shared features using a skip connection method to form a fused feature in the channel dimension. The fused feature is then subjected to convolution and feature refinement to generate the output feature of the decoding layer.

[0144] By using the output features as input to the next decoding layer, the attention-enhanced U-Net decoder sequentially outputs features from multiple decoding layers, ultimately forming a set of decoded features composed of output features from different decoding layers.

[0145] In this embodiment, the construction of the page layout hierarchy structure based on the spatial position, size ratio, and text semantic similarity of each page element in the accurate page layout segmentation result specifically includes:

[0146] For each page element in the accurate page segmentation result, read the segmentation mask corresponding to the page element, and determine the bounding rectangle region of the page element in the original instruction manual image based on the distribution range of all foreground pixels in the segmentation mask.

[0147] Specifically, the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate of all foreground pixels in the page element segmentation mask are calculated respectively. The average of the minimum horizontal coordinate and the maximum horizontal coordinate is used as the horizontal center position of the page element, and the average of the minimum vertical coordinate and the maximum vertical coordinate is used as the vertical center position of the page element.

[0148] The width of the page element is obtained by subtracting the minimum horizontal coordinate from the maximum horizontal coordinate, and the height of the page element is obtained by subtracting the minimum vertical coordinate from the maximum vertical coordinate. Thus, the horizontal center position and the vertical center position are defined together as the center position parameter of the page element, and the width and height are defined together as the size parameter of the page element. The spatial position parameter and the size parameter of the page element are combined to form a spatial feature vector.

[0149] Text encoding processing is performed on the text content corresponding to each page element. Specifically, this includes: first, extracting the text paragraphs corresponding to the spatial position of the page element as the input text sequence; then, semantic encoding is performed on the text sequence to generate a vector representation for the meaning of the text, which is defined as the text encoding vector of the page element.

[0150] For any two different page elements, obtain their corresponding text encoding vectors and use the cosine similarity formula to calculate their semantic similarity. The semantic similarity is used to quantify the semantic closeness of the text content between the two page elements.

[0151] Based on spatial feature vectors and semantic similarity, pairwise relationships are determined for all page elements. According to the comprehensive relationship of spatial position, size ratio and text semantic similarity, a preliminary relationship set consisting of parent-child relationship, sibling relationship and reference relationship between page elements is constructed.

[0152] Specifically, for every two page elements, firstly, the horizontal center position, vertical center position, width, and height represented by the spatial feature vectors are compared to determine their relative positional relationship and size ratio on the page; then, the semantic similarity between the two is combined to determine the semantic relevance of the text content; based on the combined results of spatial position, size ratio, and semantic similarity, each pair of page elements is labeled with its possible relationship type, including possible parent-child relationship, sibling relationship, or reference relationship, and the above judgment results are recorded as the preliminary relationship of the pair of page elements. By evaluating all page elements pairwise, a preliminary relationship set consisting of multiple preliminary relationships is formed.

[0153] For any two page elements in the preliminary relationship set, based on the hierarchical mapping determination criteria, it is comprehensively determined whether the two page elements have a hierarchical inclusion or subordinate relationship. The hierarchical mapping determination criteria specifically include:

[0154] If two page elements are arranged vertically, and the vertical distance between the bottom coordinates of the upper element and the top coordinates of the lower element is less than the first spatial threshold, then a vertical relationship is determined to exist.

[0155] If the horizontal positions of two page elements are basically the same, and the height of the upper element is less than that of the lower element, and the semantic similarity between the two texts is higher than the first semantic threshold, then the upper element is determined to be the title area and the lower element is the body text area.

[0156] If there is a spatial containment relationship between two page elements, and the area difference between the containing area and the contained area is less than a set ratio threshold, and the semantic similarity of their texts is within the second semantic threshold, then the containing area is determined to be the parent element, and the other area is its child content.

[0157] Based on the above hierarchical mapping determination conditions, the page elements in the preliminary relationship set are analyzed one by one to generate a complete hierarchical mapping relationship set;

[0158] In the set of hierarchical mapping relationships, the relationships that satisfy the condition of correspondence between title and body text are marked as title-body text relationships, the relationships that satisfy the condition of correspondence between chart area and explanatory text are marked as chart-explanatory relationships, and the relationships that satisfy the condition of correspondence between step text and step description are marked as step-description relationships.

[0159] Summarize the hierarchical mapping relationships between all the page elements that have been marked and filtered to construct a complete page hierarchy structure.

[0160] In this embodiment, the step of extracting multimodal features from the precise page segmentation results based on the page layout hierarchy and calculating matching weights to output the image-text semantic alignment results specifically includes:

[0161] Based on the hierarchical structure of the page layout, the image regions and text content pairs that need to be aligned with the text semantics are determined; wherein, the image regions are image blocks from the precise page segmentation results, and the text content is the OCR-recognized text fragments associated with the corresponding image blocks in the same hierarchical structure.

[0162] The image region is preprocessed, including size normalization and channel normalization. The processed image region is then input into the Vision Transformer network to extract the visual semantic feature vector of the image region. The visual semantic feature vector is used to describe the global content semantics of the image.

[0163] After the text content is segmented, interference symbols are removed, and the encoding format is unified, it is input into the BERT model to perform deep semantic representation of the text and obtain the context semantic feature vector of the text content. The context semantic feature vector is used to reflect the context dependency structure of the text semantics.

[0164] The visual semantic feature vector and the text semantic feature vector are input together into the joint attention mechanism module. First, a linear transformation is performed on the visual semantic feature to generate a query vector, and a linear transformation is performed on the text semantic feature to generate a key vector and a value vector. Based on this, the matching weight used to measure the semantic correspondence between the visual and text is obtained by calculating the dot product similarity between the query vector and the key vector.

[0165] Based on the matching weights, the semantic coupling relationship between the image region and the text content is quantified and output to generate the image-text semantic alignment result.

[0166] The semantic coupling relationship refers to the degree of fit between the visual information contained in the image region and the semantic information expressed by the text content in terms of logical meaning, descriptive object, or contextual content. It is used to measure whether the image and text jointly describe the same content or belong to the same information unit.

[0167] In this embodiment, the step of inputting the semantic alignment result of the text and image together with the hierarchical relationship structure of the page layout into the logical structure modeling module to construct the page element diagram and generate the structured logical reasoning result specifically includes:

[0168] The image-text semantic alignment result and the page layout hierarchy structure are input into the logical structure modeling module to initialize the page element graph structure. Each node in the page element graph structure represents a page element. The node attributes include spatial location information, the level to which it belongs, the image-text matching result and the semantic feature vector. Each edge represents the structural relationship between any two nodes.

[0169] Based on the matching weights of image regions and text paragraphs in the image-text semantic alignment results, as well as the spatial and semantic subordinate relationships in the page layout hierarchy, the edge weights between nodes are calculated, a weighted page element graph structure is constructed, and a node set and an edge set are formed to represent all page elements and their relationships, respectively.

[0170] The weighted layout element graph structure is input into the graph neural network model. The context semantic aggregation operation is performed on each node, and the structure-aware semantic representation is output to capture the multi-level semantic dependencies and structural relationships between local layout elements.

[0171] The structure-aware semantic representation sequence is input into the Transformer model, and the long-range dependencies between nodes are modeled through the self-attention mechanism. Content sequence reasoning is performed to generate a structured text sequence, which includes the logical position, content affiliation, and sequential number of page elements.

[0172] Based on the content structure identified in the structured text sequence, each page element is mapped to a predefined logical structure template, and the structured logical reasoning result is output. The structured logical reasoning result includes the hierarchical and content reference relationships between the main text module, title module, image and text module, table module and step module.

[0173] In this embodiment, generating a knowledge graph semantic output containing entities, relationships, and attributes based on the structured logical reasoning results specifically includes:

[0174] Each page element node identified in the structured logic reasoning results is used as a candidate entity set. The core nouns, terms, symbols, and title words in each page element node are extracted as semantic keywords to construct an initial entity set.

[0175] Based on the semantic keywords of each page element node in the initial entity set, and combined with the node connection edge information in the structured logic reasoning results, the semantic similarity between any two entities is calculated. When the semantic similarity exceeds a set threshold, it is determined that there is a semantic relationship and the entity is added to the relationship set.

[0176] For each pair of entities in the relation set, analyze its contextual information in the structured logical reasoning results, connection patterns and semantic content in the graph-text semantic alignment results, extract relation description words to represent the semantic relation, map the entity pair to the semantic relation type, and generate structured relation data in the form of triples, wherein the triples are (entity head node, relation type, entity tail node).

[0177] Based on the construction of entity relationship triples, attribute information related to each entity is extracted from the graph-text semantic alignment results. The attribute information includes unit, quantity, type, function, location and status. Each attribute information is appended to the corresponding entity node in the form of key-value pairs to construct a graph node structure containing entity ontology, attribute set and semantic relationship.

[0178] By integrating the knowledge graph node structure with entity relationship triples, a knowledge graph semantic output result containing entities, attributes, and semantic relationships is constructed to achieve automatic understanding and structured semantic representation of complex instruction manual content.

[0179] refer to Figure 3 A multimodal understanding system for complex instruction manuals based on YOLO two-stage segmentation includes the following modules:

[0180] The document preprocessing module is used to perform image enhancement, resolution adaptive adjustment, and noise reduction on the input instruction manual image to obtain a preprocessed image;

[0181] The YOLO two-stage layout segmentation module is used to perform coarse-grained detection and fine-grained segmentation on the preprocessed image. It adopts an improved YOLOv8 feature extraction structure, combined with backbone network sharing and attention-enhanced U-Net decoder, to identify various layout element regions in the instruction manual and generate accurate segmentation results.

[0182] The layout structure modeling and hierarchy recognition module is used to construct the hierarchical relationship structure of the layout based on the position, size and semantic information of each layout element in the accurate segmentation results.

[0183] The multimodal feature extraction and alignment module is used to extract visual and linguistic features from the image regions and OCR text content in the accurate layout segmentation results, and to perform image-text semantic matching and alignment through a joint attention mechanism to generate image-text aligned output results.

[0184] The structured logic modeling and reasoning module is used to model the semantic alignment results of text and graphics together with the hierarchical relationship structure of the layout into a layout element diagram, and generate structured logic reasoning results.

[0185] The knowledge graph construction module is used to generate semantic output results of knowledge graphs containing entities, relationships, and attributes based on the results of structured logical reasoning.

[0186] Example 1:

[0187] To verify the feasibility of this invention in practice, it was applied to a scenario involving the parsing of an industrial equipment instruction manual. This instruction manual consists of multiple functional chapters, including structural diagrams, workflow diagrams, tabular data, hierarchical headings, and lengthy text descriptions. The overall layout is complex, with numerous issues related to cross-image / text correspondence and description jumps. Traditional layout detection methods in this scenario have exhibited low detection rates for small text blocks, inconsistencies in image / text correspondence, and inaccurate identification of heading and body text hierarchies, failing to guarantee the continuity of the structured expression of subsequent content. The multimodal fusion and hierarchical reasoning mechanism of this invention aims to solve these problems.

[0188] In the actual parsing process, the documentation is input into the system as a scanned image. The system first performs page enhancement and noise suppression through an image preprocessing module, ensuring the input image achieves stable and clear visual quality. Subsequently, a two-stage page segmentation module built with improved YOLOv8 is used to perform coarse-grained detection and fine-grained segmentation of text blocks, image blocks, and table regions on the page. Because the documentation contains densely packed small-font text and a small area of ​​operation prompts, traditional methods have significantly insufficient recall in these areas. This invention utilizes deformable convolution and cross-stage feature sharing mechanisms to achieve higher-resolution representation of these areas in the feature map, thereby improving the stability of fine-grained segmentation.

[0189] After obtaining accurate page segmentation results, the system constructs page hierarchy relationships based on spatial location, size ratio, and semantic similarity. For example, the functional headings at the top of the page in the instruction document have a high semantic match with the following body paragraphs, and are identified as heading-body pairs in the hierarchical reasoning module; the workflow diagram and the step explanation text to its right are identified as image-instruction pairs due to the strong semantic correlation between the image and text. The recognition of these hierarchical structures enables the effective reconstruction of the logical relationships between multiple paragraphs within the instruction document.

[0190] In the image-text semantic feature modeling stage, the system extracts visual semantic features from image regions, generates text semantic vectors for the corresponding OCR text, and aligns the two types of features through a joint attention mechanism, achieving high matching scores in multiple flowchart regions of the instruction document. In the subsequent logical structure modeling, the graph neural network and Transformer inference module further integrate image-text information, outputting a structured knowledge representation containing entities, attributes, and relationships, mapping the component names, parameter definitions, work steps, and graphical explanations in the instruction document into a queryable knowledge graph structure.

[0191] To verify the effectiveness of this invention, the performance of traditional detection methods and this invention was compared in the aforementioned documentation. The system recorded the detection accuracy, text-image alignment consistency, and hierarchical structure recognition accuracy. Experimental data show that this invention significantly improves performance in complex layout scenarios, especially in small-sized text recognition and text-image relationship localization. Table 1 shows the relevant experimental data.

[0192] Table 1 Comparison of Complex Document Parsing Performance Data

[0193]

[0194] As can be seen from the table above, this invention achieves significant improvements over traditional methods in several key performance indicators for parsing complex instruction manuals. Regarding text block detection rate, this invention, through a combination of two-stage page segmentation and an improved YOLOv8 feature extraction structure, achieves an overall text region detection rate of 95.7%, a 13.3 percentage point improvement over the traditional method's 82.4%. Particularly noteworthy is the recall rate for small-sized text. This invention utilizes deformable convolution and a cross-stage feature sharing mechanism to enhance the expression of local detail features, increasing the recall rate from 67.1% in the traditional method to 92.3%, a 25.2 percentage point improvement. This demonstrates that this invention can effectively handle tightly packed small-font text regions in instruction manuals.

[0195] In image region recognition, the detection accuracy of this invention reaches 97.2%, an improvement of 8.7 percentage points compared to the traditional method's 88.5%, indicating that the two-stage structure has stronger stability when processing heterogeneous visual layouts. For the segmentation consistency index of table regions, the intersection-union ratio of this invention reaches 0.86, an improvement of 0.15 compared to the traditional method's 0.71, demonstrating that this invention has a greater advantage in maintaining the integrity of the table structure and boundary accuracy. In cross-modal semantic processing, the image-text semantic alignment consistency rate of this invention reaches 93.6%, an improvement of 18.8 percentage points compared to the traditional method's 74.8%. This is attributed to the synergistic effect of Vision Transformer, BERT text encoding, and the joint attention mechanism, resulting in a more accurate semantic correspondence between image content and OCR text.

[0196] In terms of recognizing page layout hierarchy, this invention achieves accuracy rates of 91.1% and 94.8% for the two core indicators of title-text recognition and figure-description recognition, respectively, representing improvements of 21.7 and 22.8 percentage points compared to traditional methods. This indicates that the cross-judgment mechanism based on spatial location, size ratio, and semantic similarity can effectively recover the hierarchical structure of the explanatory document. Furthermore, this invention achieves a structured knowledge output completeness of 89.4%, a 25.9 percentage point improvement over the traditional method's 63.5%, demonstrating its advantages in automatically generating entity, relationship, and attribute structures.

[0197] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal understanding method for complex instruction manuals based on YOLO two-stage segmentation, characterized in that, Includes the following steps: The input instruction manual image is subjected to image enhancement, adaptive resolution adjustment, and noise reduction to obtain a preprocessed image; The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, and a coarse-grained candidate region set is generated using a dynamic anchor box matching detection head. The coarse-grained candidate region set is input into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, and end-to-end optimization is performed to obtain accurate page segmentation results. A hierarchical structure of page layout is constructed based on the spatial position, size ratio, and textual semantic similarity of each page element in the accurate page layout segmentation results. Based on the hierarchical structure of the page layout, multimodal features are extracted from the precise page segmentation results, matching weights are calculated, and the semantic alignment results of the text and images are output. The semantic alignment results of the text and images, along with the hierarchical relationship structure of the layout, are input into the logical structure modeling module to construct the layout element diagram and generate structured logical reasoning results. Generate a knowledge graph semantic output containing entities, relationships, and attributes based on the results of structured logical reasoning; The step of inputting the coarse-grained candidate region set into a fine-grained segmentation network based on an improved YOLOv8 feature extraction structure, performing end-to-end optimization, and obtaining accurate page segmentation results specifically includes: The coarse-grained candidate region set is input as the segmentation target into the fine-grained segmentation network based on the improved YOLOv8 feature extraction structure, and the multi-scale feature map and the fused feature set are input into the fine-grained segmentation network. For each candidate region in the coarse-grained candidate region set, the corresponding region features are cropped from the fused feature set using the region alignment operator to generate a region feature set; The set of regional features is input layer by layer into the encoding end of the fine-grained segmentation network. In each encoding layer, the deformable convolution operator is used to perform sampling offset and convolution operations on the input features to obtain the encoded feature set. The encoded feature set is input into the attention-enhanced U-Net decoder, and upsampling and layer-by-layer feature fusion are performed to obtain the decoded feature set; The decoded feature set is sequentially input into the pixel-level classification unit of the fine-grained segmentation network. The class of the decoded feature corresponding to each candidate region is determined pixel by pixel to generate a segmentation mask set. After generating the segmentation mask, the fine-grained segmentation network is optimized end-to-end using a joint loss mechanism that includes detection box loss and segmentation mask loss. After optimization, the accurate layout segmentation result is output. The step of inputting the encoded feature set into the attention-enhanced U-Net decoder and performing upsampling and layer-by-layer feature fusion to obtain the decoded feature set specifically includes: The encoded feature set is input into the attention-enhanced U-Net decoder in the order of the encoding layers, so that the encoded features output by each encoding layer enter the corresponding decoding layer as the initial input features of the decoding layer. In each decoding layer, an upsampling operation is performed on the input encoded features, followed by convolution and channel attention processing to generate the intermediate decoding features of the current decoding layer. Spatial attention processing is performed on the intermediate decoding features in the current decoding layer to obtain the attention-enhanced decoding features of the current decoding layer. Select cross-stage shared features corresponding to the current decoding layer from the multi-scale feature map generated in the coarse-grained detection stage, and adjust the size by interpolation; Attention-enhanced decoding features are concatenated with corresponding cross-stage shared features using a skip connection method to form fused features in the channel dimension. Convolution and feature refinement are then performed on the fused features to generate the output features of the decoding layer. The output features are used as input to the next decoding layer to form a set of decoding features composed of output features from different decoding layers.

2. The method for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation according to claim 1, characterized in that, The coarse-grained detection stage, which inputs the preprocessed image into the improved YOLOv8 network, specifically includes generating a coarse-grained candidate region set using a dynamic anchor box matching detection head: The preprocessed image is input into the coarse-grained detection stage of the improved YOLOv8 network, which includes a backbone feature extraction layer, a bidirectional feature fusion layer, and a dynamic anchor box matching detection head. In the backbone feature extraction layer, the C2f-CBAM module is used to perform multi-scale semantic feature extraction on the preprocessed image. Through the C2f structure and CBAM attention unit, multi-scale feature maps are obtained. In the bidirectional feature fusion layer, the Bi-FPN structure is used to fuse the multi-scale feature maps output by the backbone feature extraction layer to obtain a fused feature set; In the dynamic anchor box matching detection head, the prediction and matching of candidate regions are performed using the fused feature set to obtain the candidate region set and the corresponding category probability vector and confidence score; The candidate region set is filtered based on the confidence threshold and the non-maximum suppression threshold, and non-maximum suppression is performed to obtain the retained candidate regions; The retained candidate regions are divided into text block regions, table regions, title regions, or image regions according to the category probability vector, forming a coarse-grained candidate region set.

3. The method for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation according to claim 1, characterized in that, The construction of the page layout hierarchy based on the spatial position, size ratio, and textual semantic similarity of each page element in the accurate page layout segmentation results specifically includes: For each page element in the accurate page segmentation result, the bounding rectangle region of the corresponding page element is calculated based on the spatial range of pixels in the segmentation mask to obtain the spatial position parameters and size parameters of the page element. The spatial position parameters and size parameters of the page element are then combined to form a spatial feature vector. Perform text encoding processing on the text content corresponding to each page element to obtain a text encoding vector; For any two different page elements, obtain their corresponding text encoding vectors and calculate the semantic similarity between the text encoding vectors; Based on spatial feature vectors and semantic similarity, pairwise relationships are determined for all page elements. A preliminary relationship set is constructed according to the comprehensive relationship of spatial location, size ratio and text semantic similarity. Based on the preliminary set of relationships, the hierarchical relationships between page elements are summarized and organized to form a set of hierarchical mapping relationships; In the set of hierarchical mapping relationships, the relationships that satisfy the condition of correspondence between title and body text are marked as title-body text relationships, the relationships that satisfy the condition of correspondence between chart area and explanatory text are marked as chart-explanatory relationships, and the relationships that satisfy the condition of correspondence between step text and step description are marked as step-description relationships. Summarize the hierarchical mapping relationships between all marked and filtered page elements to construct the page hierarchy structure.

4. The method for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation according to claim 1, characterized in that, Based on the hierarchical structure of the page layout, multimodal feature extraction is performed on the precise page segmentation results, and matching weights are calculated to output the image-text semantic alignment results. Specifically, this includes: Based on the hierarchical structure of the page layout, determine the image regions and text content pairs that need to be semantically aligned; The image region is preprocessed, including size normalization and channel normalization. The processed image region is then input into the Vision Transformer network to extract the visual semantic feature vector of the image region. After the text content is segmented, interference symbols are removed, and the encoding format is unified, it is input into the BERT model to perform deep semantic representation of the text and obtain the contextual semantic feature vector of the text content. The visual semantic feature vector and the text semantic feature vector are input together into the joint attention mechanism module to obtain the matching weights; Based on the matching weights, the semantic coupling relationship between the image region and the text content is quantified and output to generate the image-text semantic alignment result.

5. The method for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation according to claim 1, characterized in that, The step of inputting the semantic alignment results of the text and images along with the hierarchical relationship structure of the page layout into the logical structure modeling module to construct the page element diagram and generate structured logical reasoning results specifically includes: Input the semantic alignment results of the text and images along with the hierarchical relationship structure of the page layout into the logical structure modeling module to initialize the page element diagram; Based on the semantic alignment results of the text and images and the hierarchical structure of the layout, the edge weights between nodes are calculated, a weighted layout element graph is constructed, and a set of nodes and a set of edges are formed. Perform contextual semantic aggregation on each node in the weighted layout element graph to output a structure-aware semantic representation; The structure-aware semantic representation sequence is input into the Transformer model, and the long-range dependencies between nodes are modeled through the self-attention mechanism. Content sequence reasoning is then performed to generate a structured text sequence. Based on the content structure identified in the structured text sequence, each page element is mapped to a predefined logical structure template, and the structured logical reasoning result is output.

6. The method for multimodal understanding of complex instruction manuals based on YOLO two-stage segmentation according to claim 1, characterized in that, The specific steps of generating a knowledge graph semantic output containing entities, relationships, and attributes based on structured logical reasoning results include: Each page element node identified in the structured logic reasoning results is used as a candidate entity set. Semantic keywords are extracted from each page element node to construct an initial entity set. Based on the semantic keywords of each page element node in the initial entity set, and combined with the node connection edge information in the structured logic reasoning results, the semantic similarity between any two entities is calculated. When the semantic similarity exceeds a set threshold, it is determined that there is a semantic relationship and the entity is added to the relationship set. For each pair of entities in the relation set, extract relation description words, map the entity pairs to semantic relation types, and generate structured relation data in the form of triples; Extract attribute information related to each entity from the semantic alignment results of the graph and text, and attach each attribute information to the corresponding entity node in the form of key-value pairs to construct the graph node structure. By integrating the graph node structure with entity relation triples, a knowledge graph semantic output result containing entities, attributes, and semantic relations is constructed.

7. A multimodal understanding system for complex instruction manuals based on YOLO two-stage segmentation, comprising executing the multimodal understanding method for complex instruction manuals based on YOLO two-stage segmentation as described in any one of claims 1 to 6, characterized in that, Includes the following modules: The document preprocessing module is used to perform image enhancement, resolution adaptive adjustment, and noise reduction on the input instruction manual image to obtain a preprocessed image; The YOLO two-stage layout segmentation module is used to perform coarse-grained detection and fine-grained segmentation on the preprocessed image. It adopts an improved YOLOv8 feature extraction structure to generate accurate segmentation results. The layout structure modeling and hierarchy recognition module is used to construct the hierarchical relationship structure of the layout based on the position, size and semantic information of each layout element in the accurate segmentation results. The multimodal feature extraction and alignment module is used to extract visual and linguistic features from the accurate layout segmentation results and generate image-text aligned output results through a joint attention mechanism. The structured logic modeling and reasoning module is used to model the semantic alignment results of text and graphics together with the hierarchical relationship structure of the layout into a layout element diagram, and generate structured logic reasoning results. The knowledge graph construction module is used to generate semantic output results of knowledge graphs based on the results of structured logical reasoning.

Images