A textbook schematic diagram structured information parsing method and system and a storage medium

Abstract

A textbook schematic diagram structured information analysis method, system and storage medium, the method comprises generating a synthetic schematic diagram according to the relationship between different visual objects in the textbook schematic diagram according to the predefined relationship type, and the predefined relationship type includes parallel relationship, containing relationship, connection relationship and similar relationship. The pre-established relationship predictor is trained by using the synthetic schematic diagram and the position of the visual object; the textbook schematic diagram visual object detection and relationship prediction are carried out by using the trained relationship predictor; the relationship score between the textbook schematic diagram visual objects is calculated according to the textbook schematic diagram visual object detection and relationship prediction result, the relationship content is generated according to the relationship score, and the textbook schematic diagram structured information analysis is realized. According to the internal law of the schematic diagram, the schematic diagram structured information analysis based on the synthetic data is realized, the understanding of complex semantic information is enhanced under the condition that the sample amount is small and the relationship between elements is less labeled information.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a method, system, and storage medium for parsing structured information from textbook diagrams. Background Technology

[0002] In textbook question-answering tasks, the parsing of diagrams is crucial. In existing technologies, analysis by Kembhavi et al. shows that the vast majority (>90%) of diagram-related questions in textbook question-answering tasks require diagram parsing, and most of these questions require understanding the rich structural information within the textbook diagrams. The parsing of structured information from diagrams is essential for textbook question-answering models, forming the foundation for understanding the complex semantic information of diagrams.

[0003] The task of interpreting textbook illustrations is similar to that of visual question answering. Existing research mainly falls into three categories: methods based on joint embedding, methods based on attention mechanisms, and methods based on pre-trained models. Methods based on joint embedding typically first use specific neural network modules to obtain image and text representations, then use feature fusion strategies to map these features to a common feature space, and finally construct a classifier, usually composed of multilayer perceptrons, for answer prediction. Researchers have attempted to innovate on the image and text representation modules and feature fusion strategies. In this type of method, both text and image representations are treated as holistic features, which leads to poor performance in tasks requiring attention to key objects. Attention-based methods improve the model's ability to capture key information by learning different levels of attention to different parts of the input data, and then perform finer-grained fusion of the input data to predict the answer. These methods typically first use object detection models to learn the representation features of objects, then use bilinear attention networks to fuse multimodal feature representations. The attention mechanism of this network has been used to propose a series of new visual question answering methods. These methods improve performance and lay the foundation for methods based on pre-trained models. Methods based on pre-trained models utilize the loss from the pre-training task and the contrastive loss between object labels for optimization. In the field of natural language processing, the BERT language model, based on the Transformer model, has achieved excellent results. Inspired by this, to enhance the generalization ability of visual question answering models, researchers attempted to design the same paradigm for visual question answering tasks: first, pre-train the Transformer model on a large-scale dataset, and then fine-tune it using downstream tasks such as visual question answering to improve the model's accuracy.

[0004] Existing methods fail to effectively extract and understand the rich structural information between elements in textbook illustrations. Textbook illustrations often contain numerous elements with complex relationships and dependencies. These relationships may manifest in various forms, such as line connections, color distinctions, and shape cues. Furthermore, the drawing style, symbol usage, and annotation methods of illustrations vary depending on the textbook, author, and publication year, lacking unified standards and norms. This increases the difficulty of analysis. While existing image recognition technologies can identify some basic graphic elements, they are insufficient for handling complex structures and relationships. In particular, existing technologies often struggle to accurately identify symbols, lines, and color combinations with specific meanings. Analyzing illustrations typically requires combining them with relevant textual descriptions or annotations. However, existing natural language processing technologies still fall short in handling technical terminology and contextual relationships, making it difficult to accurately extract and understand key information from illustrations. Moreover, existing analysis methods are often too general, lacking specific strategies for textbook illustrations. This makes it difficult to accurately grasp the key information and structural relationships within the illustrations during the analysis process. While manual intervention can improve the accuracy of interpretation, its large-scale application is often limited by labor costs and the level of expertise required. Furthermore, manual interpretation suffers from strong subjectivity and a high susceptibility to errors. Meanwhile, textbook illustrations often involve specific subject knowledge and terminology. Interpreters lacking relevant subject background knowledge often struggle to accurately understand and interpret the information in these illustrations. Moreover, textbook illustrations are often designed to help students understand complex concepts and processes. Therefore, the interpretation of illustrations must consider educational objectives and students' learning needs, further increasing the complexity of the process. Summary of the Invention

[0005] The purpose of this invention is to address the problems in the prior art by providing a method, system, and storage medium for parsing structured information from textbook diagrams. Based on the inherent rules of the diagrams, it achieves structured information parsing of diagrams based on synthetic data, enhancing the understanding of complex semantic information even with small sample sizes and limited annotation information on relationships between elements.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a method for parsing structured information from textbook diagrams is provided, including:

[0008] Based on the relationships between different visual objects in the textbook illustrations, generate composite illustrations according to predefined relationship types;

[0009] A pre-established relation predictor is trained using synthetic schematics and the positions of visual objects.

[0010] Use a trained relation predictor to detect visual objects and predict relations in textbook illustrations;

[0011] Based on the visual object detection and relationship prediction results of the textbook illustrations, the relationship scores between the visual objects in the textbook illustrations are calculated, and the relationship content is generated based on the relationship scores, thereby realizing the structured information parsing of the textbook illustrations.

[0012] As a preferred embodiment, in the step of generating a composite diagram based on the relationships between different visual objects in the textbook illustration according to predefined relationship types, the predefined relationship types include parallel relationships, inclusion relationships, connection relationships, and similar relationships.

[0013] As a preferred embodiment, in the step of generating a composite diagram according to a predefined relationship type based on the relationship between different visual objects in the textbook diagram, several visual objects and the relationships between them are randomly generated based on the visual object relationship restriction rules: taking a set of visual objects that do not contain any visual objects as the starting point, the sets are aggregated layer by layer, new visual objects are obtained iteratively and added to the object set, and connection relationships are assigned to the objects in the object set according to random probability, that is, the relationships between different visual objects are randomly generated.

[0014] As a preferred embodiment, for different visual objects A, B, and C, the visual object relationship restriction rules include:

[0015] If A contains B, then B cannot contain A;

[0016] If A and B are listed together, then B cannot contain A.

[0017] If A and B are listed together, then A cannot contain B;

[0018] If A contains B, then B cannot be connected to A;

[0019] If A contains B, then A cannot be connected to B;

[0020] If A contains B and C does not contain any elements, then A and C cannot be of the same class.

[0021] If A and B are of the same kind, and B and C are of the same kind, then A and C are of the same kind;

[0022] If A contains B, and B contains C, then A contains C;

[0023] If A contains B, and A and C are parallel, then C cannot contain B;

[0024] If A contains B and A contains C, then B and C are parallel.

[0025] If neither A nor B is contained in any element, then A and B are parallel.

[0026] As a preferred approach, after randomly generating several visual objects and the relationships between them, each relationship is converted into code that conforms to the Mermaid specification of the declarative diagram definition language data visualization library, and then spliced ​​together to generate a composite diagram that conforms to the corresponding relationships.

[0027] As a preferred embodiment, when training a pre-established relation predictor using the synthetic schematic diagram and the position of the visual object, the relation predictor uses a ResNet residual network with residual blocks to extract the visual features of the visual object, and uses a multilayer perceptron as a position encoder to map the position information to a high-dimensional space and splice it with the visual features, so that the position information of the visual object is fused with the visual information.

[0028] Assumption Illustration for textbook A collection of visual objects in a collection Each of them It is a visual information representation of a visual object; assuming For each visual object, there is a set of positional information. Representing visual objects In the textbook illustration The position in the middle, then each visual object visual features and the position information of each visual object Mapping to a high-dimensional space yields encoded location features. Represented as:

[0029]

[0030]

[0031]

[0032] In the formula, These are the parameters that need to be trained;

[0033] This is a fusion feature that integrates visual and location information. Represented as:

[0034]

[0035] Here, [;] represents a concatenation operation;

[0036] For each visual object This yields a fused feature that combines visual and positional features. ;

[0037] For each pair of visual objects ,definition For visual objects and Visual information between them, namely and The image content between them; a multilayer perceptron is used as the relation predictor:

[0038]

[0039]

[0040] In the formula, The parameters to be trained; obtain each visual object With visual objects Relationship feature representation Relationship prediction using a linear fully connected layer:

[0041]

[0042] In the formula, The parameters to be trained are used; the loss of the prediction results is calculated using multi-label binary classification cross-entropy.

[0043]

[0044] In the formula, It is the sample size; It balances the weights of positive and negative samples; Visual objects With visual objects Relationship tags between them; Visual objects With visual objects Predicting the relationship between them; Represents the sigmoid function, which has .

[0045] As a preferred embodiment, the steps of using a trained relation predictor to detect visual objects and predict relations in textbook illustrations include:

[0046] Visual object detection is performed using a pre-trained end-to-end YOLO network for arbitrary textbook illustrations. Get the set of bounding boxes of visual objects ,have Using visual object bounding boxes From any textbook illustration Extracting visual information sets from visual objects At this point, the trained relation predictor network parameters are used to predict relations for specific visual objects. With visual objects have:

[0047] ;

[0048] ;

[0049] At the same time:

[0050]

[0051]

[0052] in,

[0053]

[0054]

[0055] In the formula, , The parameters of the trained relation predictor network;

[0056] Regarding the first For each relationship, if the final score is higher than 0.5, a corresponding relationship is determined to exist; otherwise, no corresponding relationship exists.

[0057]

[0058] In the formula, Representing visual objects With visual objects Does the first one exist? A relation is defined as follows: a value of 1 indicates the existence of a relation, while a value of 0 indicates the non-existence of a relation. Representation tensor The Dimension; After calculating the relationship scores between all visual objects, the visual object relationship analysis for the corresponding textbook illustration is completed.

[0059] As a preferred embodiment, the step of calculating the relationship score between visual objects in the textbook illustrations based on the visual object detection and relationship prediction results, and generating relationship content based on the relationship score, includes: for each textbook illustration... Retaining the relationship between the top five scorers:

[0060]

[0061] And it satisfies:

[0062]

[0063] The MiniGPT multimodal pre-trained large language model is used as the relation content generator, and the input template is defined as follows:

[0064] [INST] <imgfeature>Please describe the visual objects in this image. <box1> <box2>Having <relation>Relationship;

[0065] In this template, [INST]⋯[ / INST] represents user input; ...represents that the content is an image. Visual features; <imgfeature>This is a schematic diagram of the feature results after passing through a visual encoder; <relation>The name of the relation; <box>Represents the position coordinates of the visual object; <box>Including the values ​​at four positions: left, right, top, and bottom. Input the above template into the MiniGPT model to generate a description of the relationships; denote the MiniGPT model as... Record the corresponding template as Then the description of the relationship for:

[0066]

[0067] After obtaining the relationship descriptions of all selected options, the relationship content is generated to realize the structured information parsing of the textbook diagram.

[0068] Secondly, a structured information parsing system for textbook diagrams is provided, including:

[0069] The composite diagram generation module is used to generate composite diagrams based on the relationships between different visual objects in textbook illustrations, according to predefined relationship types.

[0070] The relation predictor training module is used to train a pre-built relation predictor using a synthetic schematic diagram and the position of visual objects.

[0071] The visual object detection and relation prediction module is used to perform visual object detection and relation prediction for textbook illustrations using a trained relation predictor.

[0072] The relation content generation module is used to calculate the relation scores between visual objects in textbook illustrations based on the detection and relation prediction results, and generate relation content based on the relation scores to realize the structured information parsing of textbook illustrations.

[0073] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the textbook schematic diagram structured information parsing method as described in the first aspect.

[0074] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects:

[0075] This invention utilizes unsupervised synthetic diagram data and a multimodal large language model to propose a Structural Parsing Network (SPN). Based on the inherent patterns of diagrams, it realizes a method for parsing structured information from synthetic diagrams. By automatically generating synthetic diagrams and predefining the possible relationship types between visual objects in the diagrams, relationships between visual objects are randomly generated. A pre-established relationship predictor is then trained using the synthetic diagrams and the positions of the visual objects. By modeling the visual objects and their relationships, predictions for relationship features are obtained. The trained relationship predictor is then used to detect visual objects and predict relationships in textbook diagrams. By calculating relationship scores between visual objects in textbook diagrams, relationship content is generated based on these scores, achieving structured information parsing from textbook diagrams. This invention generates relationships between visual objects and converts them into a diagram definition language to generate diagrams. Then, a deep learning model is used to extract visual features and positional information to predict relationships between visual objects and generate descriptions of these relationships. This invention effectively solves the problems of small sample size and limited annotation information between elements in textbook diagrams through unsupervised synthetic diagram methods, improving the model's ability to capture structural information in diagrams and enhancing the understanding of complex semantic information. Experimental results show that the present invention achieves high accuracy in relation prediction and has a significant performance improvement over existing technologies in textbook question-and-answer tasks, proving the effectiveness and practicality of the proposed method. Attached Figure Description

[0076] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1(a) is a schematic diagram of the pre-training process of the relation predictor of the structured information parsing method based on synthetic data according to an embodiment of the present invention;

[0078] Figure 1(b) is a schematic diagram of the prediction process of the relation predictor of the structured information parsing method based on synthetic data according to an embodiment of the present invention.

[0079] Figure 2 (a) A schematic diagram of parallel relationships in the predefined relationship types of this invention;

[0080] Figure 2 (b) The predefined relationship types in the embodiments of the present invention include relationship diagrams;

[0081] Figure 2 (c) A schematic diagram of connection relationships in the predefined relationship types of this invention;

[0082] Figure 2 (d) A schematic diagram of similar relationships in the predefined relationship types of this invention;

[0083] Figure 3 In this embodiment of the invention, visual objects are displayed independently when determining the relationship between them.

[0084] Figure 4 A schematic diagram of the image generated after each relationship in this embodiment of the invention is converted into icon customization code;

[0085] Figure 5 Example diagram of the synthesis schematic generated in the embodiments of the present invention;

[0086] Figure 6(a) is an example diagram showing how the relational description generated by the method of this embodiment of the invention is helpful in completing textbook question-and-answer tasks;

[0087] Figure 6(b) is an example diagram showing that the relational descriptions generated by the method of this embodiment of the invention cannot fully help with textbook question-and-answer tasks. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, those skilled in the art can obtain other embodiments without creative effort.

[0089] Please refer to Figures 1(a) and 1(b). The textbook schematic diagram structured information parsing method of this embodiment includes:

[0090] Based on the relationships between different visual objects in the textbook illustrations, generate composite illustrations according to predefined relationship types;

[0091] A pre-established relation predictor is trained using synthetic schematics and the positions of visual objects.

[0092] Use a trained relation predictor to detect visual objects and predict relations in textbook illustrations;

[0093] Based on the visual object detection and relationship prediction results of the textbook illustrations, the relationship scores between the visual objects in the textbook illustrations are calculated, and the relationship content is generated based on the relationship scores, thereby realizing the structured information parsing of the textbook illustrations.

[0094] In one possible implementation, the synthetic schematic diagram of the present invention is automatically generated. Generally, the performance of neural networks in deep learning is closely related to the size and diversity of their training data. The larger and more diverse the dataset, the better the training results can usually be obtained.

[0095] Before generating the synthesis schematic diagram, the embodiments of the present invention define as follows: Figure 2 (a) Figure 2 (b) Figure 2 (c) Figure 2 (d) shows the parallel, containment, connection and similar relationships, and a schematic diagram is designed based on these relationships.

[0096] In fact, to ensure that the generated diagram is reasonable, the relationship between any two visual objects cannot be generated completely randomly. Certain restrictions must be met. For example, if visual object A contains visual object B, then visual object B cannot contain visual object A. The specific rules are defined in Table 1.

[0097] Table 1

[0098]

[0099] Based on the rules listed in Table 1, an algorithm for generating random relationships between visual objects can be designed. In this embodiment of the invention, the algorithm for generating random relationships between visual objects starts with a set of visual objects that do not contain any specific visual object. It aggregates layer by layer, iteratively obtaining new visual objects and adding them to the object set. Then, according to a predetermined random probability, connection relationships are assigned to the objects in the object set, thus generating relationships between different visual objects. The specific algorithm is shown in Table 2.

[0100] After randomly generating several visual objects and the relationships between them using an algorithm, the result is converted into code conforming to the Mermaid specification, a data visualization library that uses a declarative chart definition language. This generates a diagram that represents the relationships. Specifically, each relationship needs to be declared and then concatenated to obtain the final chart definition code.

[0101] Mermaid is an open-source, interactive data visualization library that uses the Markdown markup language to generate charts and flowcharts. Mermaid is a text-based graphical tool that allows users to create complex diagrams using simple text code. It is commonly used to generate diagrams for websites or documents, such as flowcharts, sequence diagrams, and Gantt charts.

[0102] Flowcharts use square brackets [], parentheses (), double parentheses [[]], double parentheses with database identifier [(Database)], and parentheses with text ((text)). Edges use arrows -->, ---, etc., to indicate flow from one node to another. The direction of the flowchart can be declared, such as top-down (TB / TD), bottom-up (BT), left-to-right (LR), and right-to-left (RL). Various node shapes are supported, such as rounded rectangles, circles, rhombuses, hexagons, parallelograms, and trapezoids. Different types of links are supported, such as links with arrows, links without arrows, links with text, dashed links, and thick links. Sequence diagrams describe the chronological order of a series of events. They are defined using the `sequenceDiagram` keyword, with participants defined by `participant`, and `->>` used to indicate message passing. Gantt charts are used for project management and task tracking. They are defined using the `gantt` keyword, defining the start time, end time, and duration of a task.

[0103] Table 2

[0104]

[0105] In one possible implementation, when training a pre-established relation predictor using a synthesized schematic diagram and the positions of visual objects, the input is the schematic diagram and the position of each visual object, and the output is a relation score matrix between each pair of visual objects. The visual object feature representation uses a ResNet network with residual blocks to extract visual features of the visual objects, and a multilayer perceptron is used as a position encoder to map the position information to a high-dimensional space and concatenate it with the visual features, thus fusing the position information of the visual objects with the visual information.

[0106] In formal terms, assuming Illustration for textbook A collection of visual objects in a collection Each of them It is a visual information representation of a visual object; assuming For each visual object, there is a set of positional information. Representing visual objects In the textbook illustration The position in the middle, then each visual object visual features and the position information of each visual object Mapping to a high-dimensional space yields encoded location features. Represented as:

[0107]

[0108]

[0109]

[0110] In the formula, These are the parameters that need to be trained;

[0111] So, the fusion feature combines visual and location information. It can be represented as:

[0112]

[0113] In the formula, [;] represents the splicing operation. Ultimately, for each visual object... This yields a fused feature that combines visual and positional features. .

[0114] Furthermore, in general visual relationship detection models, the model predicts whether there is a relationship between two visual objects by calculating the fused features of the two objects; however, for the four predefined relationships mentioned above, especially the connection relationship, it is difficult to determine the connection relationship based solely on the visual information and positional information of each visual object. For example... Figure 3 As shown, determining connectivity often relies on visual information between visual objects, such as arrows and lines. Therefore, when judging whether two visual objects are related, we must consider not only the visual object pair itself but also all visual information between their positions. The formal definition is as follows: For each pair of visual objects… ,definition For visual objects and Visual information between them, namely and The image content between them. Using a multilayer perceptron as the relation predictor, we have:

[0115]

[0116]

[0117] In the formula, The parameters to be trained; obtain each visual object With visual objects Relationship feature representation Then, a fully connected linear layer is used for relation prediction:

[0118]

[0119] In the formula, These are the parameters that need to be trained; in this embodiment, multi-label binary classification cross-entropy is used to calculate the loss of the prediction results:

[0120]

[0121] In the formula, It is the sample size; It balances the weights of positive and negative samples; Visual objects With visual objects Relationship tags between them; Visual objects With visual objects Predicting the relationship between them; Represents the sigmoid function, which has .

[0122] After pre-training the relationship prediction model, keeping the parameters of the relationship prediction module and the position encoding module unchanged, relationship prediction for textbook illustrations can be performed. This part mainly includes two modules: object detection and relationship prediction. This embodiment uses a pre-trained YOLO network for visual object detection. Specifically, for any textbook illustration... Get the set of bounding boxes of visual objects ,have Using visual object bounding boxes From any textbook illustration Extracting visual information sets from visual objects At this point, the trained relation predictor network parameters are used to predict relations for specific visual objects. With visual objects have:

[0123] ;

[0124] ;

[0125] At the same time:

[0126]

[0127]

[0128] in,

[0129]

[0130]

[0131] In the formula, , The parameters of the trained relation predictor network.

[0132] Regarding the first This relationship can be defined as follows: if the final score is higher than 0.5, the model tends to believe that the relationship exists; otherwise, it believes that the relationship does not exist.

[0133]

[0134] In the formula, Representing visual objects With visual objects Does the first one exist? A relation is defined as follows: a value of 1 indicates the existence of a relation, while a value of 0 indicates the non-existence of a relation. Representation tensor The Dimension; After calculating the relationship scores between all visual objects, the visual object relationship analysis for the corresponding textbook illustration is completed.

[0135] After obtaining the relationship scores for all sub-schematic diagrams, relationship content can be generated for these relationships. In practice, textbook question-and-answer diagrams typically have eight or more objects. Taking an eight-object textbook question-and-answer diagram as an example, the relationship scores between all objects will reach 8*7*4=224, and this number will increase exponentially with the number of objects. To prevent excessive information from causing noise and interfering with question answers, this embodiment of the invention addresses the relationship scores for each textbook diagram. Retain the relationships among the top five scorers, i.e., based on the score matrix extracted above. ,take out:

[0136]

[0137] And it satisfies:

[0138]

[0139] Next, to generate content for each relation based on the diagram itself, this embodiment of the invention uses MiniGPT as the relation content generator. MiniGPT is an open-source multimodal pre-trained large language model that uses an LLaMA model as the text decoder, employs pre-trained ViT and Q-Former networks to extract visual features, and utilizes a single linear projection layer to align the input visual features with the input of the text decoder. The pre-trained ViT and Q-Former networks, together with the linear projection layer, serve as the model's visual encoder (hereinafter referred to as VE).

[0140] The specific input template is defined as follows:

[0141] [INST] <imgfeature>Please describe the visual objects in this image. <box1> <box2>Having <relation>Relationship;

[0142] In this template, [INST]⋯[ / INST] represents user input; ...represents that the content is an image. Visual features; <imgfeature>This is a schematic diagram of the feature results after passing through a visual encoder; <relation>The name of the relation; <box>Represents the position coordinates of the visual object; <box>Including the values ​​at four positions: left, right, top, and bottom. Input the above template into the MiniGPT model to generate a description of the relationships; denote the MiniGPT model as... Record the corresponding template as Then the description of the relationship for:

[0143]

[0144] After obtaining all the selected relation descriptions, these descriptions can be used as background knowledge to assist in answer prediction. The specific answer prediction method depends on the chosen testing method.

[0145] The effectiveness of the embodiments of the present invention is verified through experiments. The embodiments of the present invention use the TQA dataset to evaluate the generated relation descriptions. The following content first briefly shows the relevant statistics of the schematic diagram synthesis results and some example images, then introduces the experimental platform and experimental settings, then introduces the comparison method and experimental results, and finally conducts experimental analysis.

[0146] This invention uses Mermaid for diagram composition. Mermaid is a text-based diagramming syntax that allows developers to create and maintain diagrams using both text and code. By utilizing randomly generated visual objects and the relationships between them, and following the Mermaid syntax, these randomly generated relationships can be transformed into rule-compliant custom diagram code. Figure 4 The image shown is a simple example demonstrating the image generated after each relationship is converted into a custom icon code.

[0147] The textbook schematic diagram structured information parsing method of this invention generated 100,000 synthetic schematic diagrams during the construction of the synthetic dataset. The number of visual objects ranged from 8 to 15, the number of connections in each image ranged from 0 to 10, and the number of visual object categories ranged from 1 to 5. The total number of each relationship in all schematic diagrams is shown in Table 3.

[0148] Table 3

[0149]

[0150] Figure 5 The diagrams shown are composite schematics from some embodiments of the present invention. It can be seen that the images in the composite schematic data constructed by the methods of the present invention are composed of simple geometric shapes such as line segments, rectangles, and arrows. These schematics are extremely lacking in visual information but rich in structural information, and can clearly convey the relationships between elements. Pre-training using such schematics can effectively enhance the model's ability to capture structural information in the schematics, thereby aiding in the structured analysis of the schematics.

[0151] The following content mainly showcases the results of the relation predictor pre-training on a synthetic schematic dataset and the results of textbook question answering using the generated relation descriptions, along with generated examples and analysis.

[0152] (1) Experimental results during the training phase of the relation predictor

[0153] The pre-training results of the relation predictor are shown in Table 4. It can be observed that for all four proposed relations, the prediction accuracy is above 80%. Value. However, among these four relationships, the connection relationship is more difficult to predict than the parallel, similar, and containment relationships. This may be because the other three relationships are more closely related to the positional information of the visual object, while the connection relationship is relatively more flexible.

[0154] Table 4

[0155]

[0156] (2) Experimental results in the textbook question-and-answer phase

[0157] The experimental results are shown in Table 5, comparing the state-of-the-art models on the TQA datasets at each stage. It can be seen that SPN achieves a 1.1% performance improvement compared to the ISAAQ model, demonstrating the overall effectiveness of the model in this embodiment. Furthermore, the absence of pre-trained ResNet demonstrates the effectiveness of the generated relation descriptions in this embodiment; simply adding the generated relation descriptions results in a 0.97% performance improvement, with the pre-trained ResNet providing an additional boost.

[0158] Table 5

[0159]

[0160] Figures 6(a) and 6(b) are schematic diagrams of some examples of relation description generation, showing some high-scoring relations and the relation descriptions generated based on them. Figure 6(a) shows an example where the generated relation description is helpful in completing a textbook question-and-answer task. In this example, the relation description effectively helps answer the question. The description of the connecting relation is "marks the direct fission of uranium-236 into barium and krypton," which directly helps answer the question "What two smaller particles were produced by the fission reaction in the figure?" This also proves the effectiveness of this part of the content. Figure 6(b) shows a less successful example. Although the relation was successfully described in this example, and an introduction to the phases of the moon was obtained, it did not help answer the question. The reason is that the relation description in the relevant study mainly focuses on the relationship between two visual objects, while the relevant question "How many phases of the moon are shown in the figure?" requires counting all visual objects. This may require more complex relation design or relation synthesis operations. Therefore, it can be further studied as a future research topic.

[0161] Another embodiment of the present invention also proposes a structured information parsing system for textbook diagrams, comprising:

[0162] The composite diagram generation module is used to generate composite diagrams based on the relationships between different visual objects in textbook illustrations, according to predefined relationship types.

[0163] The relation predictor training module is used to train a pre-built relation predictor using a synthetic schematic diagram and the position of visual objects.

[0164] The visual object detection and relation prediction module is used to perform visual object detection and relation prediction for textbook illustrations using a trained relation predictor.

[0165] The relation content generation module is used to calculate the relation scores between visual objects in textbook illustrations based on the detection and relation prediction results, and generate relation content based on the relation scores to realize the structured information parsing of textbook illustrations.

[0166] Another embodiment of the present invention provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instructions stored in the memory to implement the textbook schematic diagram structured information parsing method.

[0167] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the textbook schematic diagram structured information parsing method.

[0168] For example, the instructions stored in the memory can be divided into one or more modules / units. These modules / units are stored in a computer-readable storage medium and executed by the processor to complete the textbook schematic diagram structured information parsing method described in this invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program on the server.

[0169] The electronic device may be a smartphone, laptop, PDA, or cloud server, among other computing devices. It may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the electronic device may also include more or fewer components, or combinations of certain components, or different components; for example, it may also include input / output devices, network access devices, buses, etc.

[0170] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0171] The memory can be an internal storage unit of the server, such as a hard drive or RAM. It can also be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory can include both internal and external storage units. The memory is used to store computer-readable instructions and other programs and data required by the server. It can also be used to temporarily store data that has been output or will be output.

[0172] It should be noted that the information interaction and execution process between the above-mentioned module units are based on the same concept as the method embodiment. For details on their specific functions and technical effects, please refer to the method embodiment section. They will not be repeated here.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0175] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0176] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.< / box> < / box> < / relation> < / imgfeature> < / relation> < / box1> < / imgfeature> < / box> < / box> < / relation> < / imgfeature> < / relation> < / box1> < / imgfeature>

Claims

1. A method for parsing structured information from textbook diagrams, characterized in that, include: Based on the relationships between different visual objects in the textbook illustrations, generate composite illustrations according to predefined relationship types; A pre-established relation predictor is trained using synthetic schematics and the positions of visual objects. Use a trained relation predictor to detect visual objects and predict relations in textbook illustrations; Based on the visual object detection and relationship prediction results of the textbook illustrations, the relationship scores between the visual objects in the textbook illustrations are calculated, and the relationship content is generated based on the relationship scores to realize the structured information parsing of the textbook illustrations; When training a pre-established relation predictor using the synthetic schematic diagram and the position of the visual object, the relation predictor uses a ResNet residual network with residual blocks to extract the visual features of the visual object, and uses a multilayer perceptron as a position encoder to map the position information to a high-dimensional space and splice it with the visual features, so that the position information of the visual object is fused with the visual information. Assumption Illustration for textbook A collection of visual objects in a collection Each of them It is a visual information representation of a visual object; assuming For each visual object, there is a set of positional information. Representing visual objects In the textbook illustration The position in the middle, then each visual object visual features and the position information of each visual object Mapping to a high-dimensional space yields encoded location features. Represented as: In the formula, These are the parameters that need to be trained; This is a fusion feature that integrates visual and location information. Represented as: Here, [;] represents a concatenation operation; For each visual object This yields a fused feature that combines visual and positional features. ; For each pair of visual objects ,definition For visual objects and Visual information between them, namely and The image content between them; a multilayer perceptron is used as the relation predictor: In the formula, The parameters to be trained; obtain each visual object With visual objects Relationship feature representation Relationship prediction using a linear fully connected layer: In the formula, The parameters to be trained are used; the loss of the prediction results is calculated using multi-label binary classification cross-entropy. In the formula, It is the sample size; It balances the weights of positive and negative samples; Visual objects With visual objects Relationship tags between them; Visual objects With visual objects Predicting the relationship between them; Represents the sigmoid function, which has .

2. The method for parsing structured information from textbook diagrams according to claim 1, characterized in that, In the step of generating a composite diagram based on the relationships between different visual objects in the textbook illustration according to predefined relationship types, the predefined relationship types include parallel relationships, inclusion relationships, connection relationships, and similar relationships.

3. The method for parsing structured information from textbook diagrams according to claim 1, characterized in that, In the step of generating a composite diagram based on the relationship between different visual objects in the textbook diagram and according to the predefined relationship type, several visual objects and the relationship between visual objects are randomly generated based on the visual object relationship restriction rules: taking the visual objects that do not contain any visual objects as the starting set, the aggregation is carried out layer by layer, new visual objects are obtained iteratively and added to the object set, and connection relationships are assigned to the objects in the object set according to random probability, that is, the relationship between different visual objects is randomly generated.

4. The method for parsing structured information from textbook diagrams according to claim 3, characterized in that, For different visual objects A, B, and C, the visual object relationship restriction rules include: If A contains B, then B cannot contain A; If A and B are listed together, then B cannot contain A. If A and B are listed together, then A cannot contain B; If A contains B, then B cannot be connected to A; If A contains B, then A cannot be connected to B; If A contains B and C does not contain any elements, then A and C cannot be of the same class. If A and B are of the same kind, and B and C are of the same kind, then A and C are of the same kind; If A contains B, and B contains C, then A contains C; If A contains B, and A and C are parallel, then C cannot contain B; If A contains B and A contains C, then B and C are parallel. If neither A nor B is contained in any element, then A and B are parallel.

5. The method for parsing structured information from textbook diagrams according to claim 3, characterized in that, After randomly generating several visual objects and the relationships between them, each relationship is converted into code that conforms to the Mermaid specification of the declarative diagram definition language data visualization library, and then spliced ​​together to generate a composite diagram that conforms to the corresponding relationship.

6. The method for parsing structured information from textbook diagrams according to claim 1, characterized in that, The steps of using the trained relation predictor to detect visual objects and predict relations in textbook illustrations include: Visual object detection is performed using a pre-trained end-to-end YOLO network for arbitrary textbook illustrations. Get the set of bounding boxes of visual objects ,have Using visual object bounding boxes From any textbook illustration Extracting visual information sets from visual objects At this point, the trained relation predictor network parameters are used to predict relations for specific visual objects. With visual objects have: ； ； At the same time: in, In the formula, , The parameters of the trained relation predictor network; Regarding the first For each relationship, if the final score is higher than 0.5, a corresponding relationship is determined to exist; otherwise, no corresponding relationship exists. In the formula, Represents visual objects With visual objects Does the first one exist? A relation is defined as follows: a value of 1 indicates the existence of a relation, while a value of 0 indicates the non-existence of a relation. Representation tensor The Dimension; After calculating the relationship scores between all visual objects, the visual object relationship analysis for the corresponding textbook illustration is completed.

7. The method for parsing structured information from textbook diagrams according to claim 6, characterized in that, The step of calculating the relationship score between visual objects in textbook illustrations based on the visual object detection and relationship prediction results, and generating relationship content based on the relationship score, includes: for each textbook illustration... Retaining the relationship between the top five scorers: And it satisfies: The MiniGPT multimodal pre-trained large language model is used as the relation content generator, and the input template is defined as follows: [INST] <imgfeature>Please describe the visual objects in this image. <box1> <box2>Having <relation> Relationship;< / relation> < / box1> < / imgfeature> In this template, [INST] [ / INST] represents user input; This indicates that the content is an image. Visual features; <imgfeature>This is a schematic diagram of the feature results after passing through a visual encoder; <relation>The name of the relation; <box>Represents the position coordinates of the visual object; <box>Including the values ​​at four positions: left, right, top, and bottom. Input the above template into the MiniGPT model to generate a description of the relationships; denote the MiniGPT model as... Record the corresponding template as Then the description of the relationship for:< / box> < / box> < / relation> < / imgfeature> After obtaining the relationship descriptions of all selected options, the relationship content is generated to realize the structured information parsing of the textbook diagram.

8. A structured information parsing system for textbook diagrams, characterized in that, include: The composite diagram generation module is used to generate composite diagrams based on the relationships between different visual objects in textbook illustrations, according to predefined relationship types. The relation predictor training module is used to train a pre-built relation predictor using a synthetic schematic diagram and the position of visual objects. The visual object detection and relation prediction module is used to perform visual object detection and relation prediction for textbook illustrations using a trained relation predictor. The relation content generation module is used to calculate the relation scores between visual objects in textbook illustrations based on the detection and relation prediction results of visual objects in textbook illustrations, and generate relation content based on the relation scores to realize the structured information parsing of textbook illustrations. When training a pre-established relation predictor using the synthetic schematic diagram and the position of the visual object, the relation predictor uses a ResNet residual network with residual blocks to extract the visual features of the visual object, and uses a multilayer perceptron as a position encoder to map the position information to a high-dimensional space and splice it with the visual features, so that the position information of the visual object is fused with the visual information. Assumption Illustration for textbook A collection of visual objects in a collection Each of them It is a visual information representation of a visual object; assuming For each visual object, there is a set of positional information. Representing visual objects In the textbook illustration The position in the middle, then each visual object visual features and the position information of each visual object Mapping to a high-dimensional space yields encoded location features. Represented as: In the formula, These are the parameters that need to be trained; This is a fusion feature that integrates visual and location information. Represented as: Here, [;] represents a concatenation operation; For each visual object This yields a fused feature that combines visual and positional features. ; For each pair of visual objects ,definition For visual objects and Visual information between them, namely and The image content between them; a multilayer perceptron is used as the relation predictor: In the formula, The parameters to be trained; obtain each visual object With visual objects Relationship feature representation Relationship prediction using a linear fully connected layer: In the formula, The parameters to be trained are used; the loss of the prediction results is calculated using multi-label binary classification cross-entropy. In the formula, It is the sample size; It balances the weights of positive and negative samples; Visual objects With visual objects Relationship tags between them; Visual objects With visual objects Predicting the relationship between them; Represents the sigmoid function, which has .

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the textbook schematic diagram structured information parsing method as described in any one of claims 1 to 7.

Images