Textbook schematic diagram structured information analysis method and system and storage medium

Abstract

The invention discloses a textbook schematic diagram structured information analysis method and system and a storage medium, and the method comprises the steps: generating a synthetic schematic diagram according to the relationship among different visual objects in a textbook schematic diagram and predefined relationship types, and the predefined relationship types comprise a parallel relationship, an inclusion relationship, a connection relationship and a similar relationship. Training a pre-established relation predictor by using the synthetic schematic diagram and the position of the visual object; performing textbook schematic diagram visual object detection and relation prediction by using the trained relation predictor; and according to textbook schematic diagram visual object detection and a relation prediction result, calculating a relation score between textbook schematic diagram visual objects, and according to the relation score, carrying out relation content generation so as to realize textbook schematic diagram structured information analysis. According to the internal rule of the schematic diagram, schematic diagram structured information analysis based on synthetic data is achieved, and under the conditions that the sample size is small and the relation labeling information between the elements is little, the understanding of complex semantic information is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a textbook schematic diagram structured information parsing method, system and storage medium. Background Art

[0002] In the textbook question-answering task, the parsing of schematic diagrams is crucial. In the related prior art, the analysis of Kembhavi et al. shows that in the textbook question-answering task, the vast majority (>90%) of schematic diagram-related questions require the parsing of schematic diagrams, and among these questions, most require understanding the rich structural information in the textbook schematic diagrams. Parsing the structural information of schematic diagrams is crucial to the textbook question-answering model task and is the basis for parsing the complex semantic information of schematic diagrams.

[0003] The task of parsing textbook diagrams is similar to the task of visual question answering. Existing research can be divided into three categories: joint embedding-based methods, attention-based methods, and pre-trained model-based methods. Among them, joint embedding-based methods usually use specific neural network modules to obtain image representation and text representation respectively, and then use feature fusion strategies to map the above features to a common feature space, and finally build a classifier usually composed of a multi-layer perceptron for answer prediction. On this basis, researchers will try to innovate the image and text representation modules and feature fusion strategies. In this type of method, both text representation and image representation are holistic features, which also leads to poor performance of such methods in tasks that require attention to key objects. Methods based on attention mechanisms improve the model's ability to capture key information by learning different attention levels for different parts of the input data, and on this basis, perform more fine-grained fusion of the input data to predict the answer. Such methods usually first use the target detection model to learn the representation features of the object, and then use the bilinear attention network to fuse the multimodal feature representation. The attention mechanism of this network is used to propose a series of new visual question answering methods. While improving the effect, these methods also lay the foundation for methods based on pre-trained models. The method based on the pre-trained model uses the loss of the pre-trained task and the contrast loss between object labels for optimization. In the field of natural language processing, the pre-trained language model Bert based on the Transformer model has achieved excellent results. Inspired by this, in order to enhance the generalization ability of the visual question answering model, researchers tried to design the same paradigm for the visual question answering task, that is, first pre-training the transformer model based on a large-scale dataset, and then fine-tuning it using downstream tasks such as visual question answering to improve the accuracy of the model.

[0004] Existing methods cannot effectively obtain and understand the rich structural information between elements in the diagram when parsing textbook diagrams. The diagrams in textbooks often contain a large number of elements, and there are complex associations and dependencies between these elements. These relationships may be presented in different forms, such as line connections, color distinctions, shape hints, etc. At the same time, the drawing style, symbol usage, and annotation methods of the diagrams vary depending on factors such as textbooks, authors, and publication years, and there is a lack of unified standards and specifications. This increases the difficulty of parsing. Although existing image recognition technology can recognize some basic graphic elements, it is still insufficient when dealing with complex structures and relationships. In particular, for symbols, lines, and color combinations with specific meanings, existing technologies are often difficult to accurately identify. When parsing diagrams, it is usually necessary to combine relevant text descriptions or annotations. However, existing natural language processing technology still has deficiencies in processing professional terms, contextual relationships, etc., making it difficult to accurately extract and understand the key information in the diagram. At the same time, existing parsing methods are often too general and lack specific strategies for textbook diagrams. This makes it difficult to accurately grasp the key information and structural relationships in the diagram during the parsing process. Although manual intervention can improve the accuracy of analysis, it is often difficult to achieve large-scale application of manual analysis due to labor costs and professional knowledge levels. At the same time, manual analysis also has problems such as strong subjectivity and error-proneness. At the same time, textbook diagrams often involve specific subject knowledge and professional terminology. Analysts who lack relevant subject background knowledge often find it difficult to accurately understand and analyze the information in the diagrams. And textbook diagrams are often designed to help students understand complex concepts and processes. Therefore, it is necessary to consider educational goals and students' learning needs when analyzing diagrams, which further increases the complexity of analysis. Summary of the invention

[0005] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method, system and storage medium for parsing structured information of textbook schematics. According to the inherent rules of the schematics, the structured information parsing of the schematics based on synthetic data is realized, and the understanding of complex semantic information is enhanced when the sample size is small and the relationship annotation information between elements is little.

[0006] In order to achieve the above object, the present invention has the following technical solutions: In a first aspect, a method for parsing structured information of a textbook schematic diagram is provided, comprising: Based on the relationships between different visual objects in the textbook diagram, a synthetic diagram is generated according to the predefined relationship types; Train a pre-built relationship predictor using synthetic diagrams and locations of visual objects; Use the trained relation predictor to perform textbook schematic visual object detection and relation prediction; According to the visual object detection and relationship prediction results of textbook diagrams, the relationship scores between visual objects in textbook diagrams are calculated, and the relationship content is generated according to the relationship scores to achieve structured information analysis of textbook diagrams.

[0007] As a preferred scheme, in the step of generating a synthetic schematic diagram according to predefined relationship types based on the relationship between different visual objects in the textbook schematic diagram, the predefined relationship types include parallel relationship, inclusion relationship, connection relationship and similar relationship.

[0008] As a preferred scheme, in the step of generating a synthetic schematic diagram according to the relationship between different visual objects in the textbook schematic diagram and a predefined relationship type, a number of visual objects and the relationships between visual objects are randomly generated based on the visual object relationship restriction rules: visual objects that do not contain any visual objects are taken as the starting point set, aggregated layer by layer, new visual objects are iteratively obtained and added to the object set, and connection relationships are assigned to objects in the object set according to random probability, that is, relationships between different visual objects are randomly generated.

[0009] As a preferred solution, for different visual objects A, B, and C, the visual object relationship restriction rules include: If A contains B, then B cannot contain A; If A and B are in parallel, then B cannot contain A; If A and B are in parallel, then A cannot contain B; If A contains B, then B cannot connect to A; If A contains B, then A cannot connect to B; If A contains B and C does not contain any element, then A and C cannot be of the same type; If A is of the same kind as B, and B is of the same kind as C, then A is of the same kind as C; If A contains B, and B contains C, then A contains C; If A contains B, and A and C are parallel, then C cannot contain B; If A contains B, and A contains C, then B and C are parallel; If A and B are not contained by any element, then A and B are parallel.

[0010] As a preferred solution, after randomly generating several visual objects and the relationships between them, each relationship is converted into code that conforms to the Mermaid specification of the declarative chart definition language data visualization library, and then spliced ​​to generate a synthetic schematic diagram that conforms to the corresponding relationship.

[0011] As a preferred solution, when the pre-established relationship predictor is trained using the synthetic schematic diagram and the position of the visual object, the relationship predictor applies a residual network ResNet that introduces a residual block to extract visual features of the visual object, and uses a multi-layer perceptron as a position encoder to map the position information to a high-dimensional space and concatenate it with the visual features, so that the position information of the visual object is fused with the visual information; Assumptions Diagram for textbook Visual object collection in , collection Each of is the visual information representation of a visual object; assuming For each visual object, the corresponding position information set, each Represents a visual object Schematic diagram in textbook , then each visual object Visual features And the location information of each visual object Mapping to high-dimensional space to obtain encoded position features It is expressed as:

[0012]

[0013]

[0014] In the formula, are the parameters that need to be trained; The fusion feature of visual information and position information is integrated It is expressed as:

[0015] Among them, [;] represents the splicing operation; For each visual Get a fusion feature that combines visual features and position features ; For each pair of visual objects ,definition For visual objects and The visual information between and The image content between them; a multi-layer perceptron is used as the relationship predictor:

[0016]

[0017] In the formula, is the parameter that needs to be trained; get each visual object With visual objects Relationship feature representation ; Use a linear fully connected layer for relationship prediction:

[0018] In the formula, are the parameters that need to be trained; the loss of the prediction results is calculated using multi-label binary cross entropy:

[0019] In the formula, is the sample size; is the weight to balance positive and negative samples; Is a visual object With visual objects The relationship label between them; Is a visual object With visual objects Prediction of the relationship between represents the sigmoid function, .

[0020] As a preferred solution, the step of using the trained relationship predictor to perform visual object detection and relationship prediction on the textbook schematic diagram includes: Use the pre-trained end-to-end object detection YOLO network for visual object detection, for any textbook diagram Get the bounding box collection of visual objects ,have ; Using visual object bounding boxes From any textbook diagram Extract the visual information set of the visual object At this time, the trained relationship predictor network parameters are used to predict the relationship for specific visual objects. With visual objects have: ;

[0021] ;

[0022] Also available:

[0023]

[0024] in,

[0025]

[0026] In the formula, , are the trained relationship predictor network parameters; For the If the final score is higher than 0.5, it is determined that there is a corresponding relationship, otherwise there is no corresponding relationship:

[0027] In the formula, Representing visual objects With visual objects Is there a A value of 1 indicates that a relationship exists, while a value of 0 indicates that a relationship does not exist. Represents a tensor No. dimension; after calculating the relationship scores between all visual objects, the visual object relationship analysis of the corresponding textbook diagram is completed.

[0028] As a preferred solution, the step of calculating the relationship scores between the visual objects in the textbook diagrams according to the results of the visual object detection and relationship prediction of the textbook diagrams, and generating the relationship content according to the relationship scores comprises: for each textbook diagram Keep the top five relationships by score:

[0029] And satisfy:

[0030] The multimodal pre-trained large language model MiniGPT is used as the relational content generator, and the input template is defined as follows: [INST] <imgfeature>Please describe the visual object in this image <box1> <box2>Has <relation>Relationship[ / INST]; In this template, [INST]⋯[ / INST] represents the user input; ⋯ means the content is an image Visual features; <imgfeature>This is the feature result after the schematic diagram passes through the visual encoder; <relation>is the name of the relationship; <box>Represents the position coordinates of the visual object; <box>Including the values ​​of the left, right, top and bottom positions, that is ; Input the above template into the MiniGPT model to generate a description of the relationship; The MiniGPT model is recorded as , and the corresponding template is recorded as , then the description of the relationship for:

[0031] After obtaining all selected relationship descriptions, relationship content generation is performed to achieve structured information parsing of textbook diagrams.

[0032] In a second aspect, a textbook schematic diagram structured information parsing system is provided, comprising: A synthetic diagram generation module, for generating a synthetic diagram according to a predefined relationship type based on the relationship between different visual objects in the textbook diagram; a relation predictor training module for training a pre-established relation predictor using the synthetic schematic diagram and the locations of the visual objects; Visual object detection and relationship prediction module, used to perform visual object detection and relationship prediction on textbook diagrams using trained relationship predictors; The relationship content generation module is used to calculate the relationship scores between visual objects in the textbook diagram based on the visual object detection and relationship prediction results of the textbook diagram, generate relationship content based on the relationship scores, and realize the structured information analysis of the textbook diagram.

[0033] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for parsing structured information of textbook schematic diagrams as described in the first aspect is implemented.

[0034] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: The present invention uses unsupervised synthetic schematic data and a multimodal large language model to propose a structural parsing network (SPN). According to the inherent laws of the schematic diagram, a schematic diagram structured information parsing method based on synthetic data is realized. By automatically generating a synthetic schematic diagram, the possible relationship types between visual objects in the schematic diagram are predefined, and then the relationship between visual objects is randomly generated. After that, the pre-established relationship predictor is trained using the synthetic schematic diagram and the position of the visual object. By modeling the visual object and the relationship between the visual object, a prediction for the relationship feature is obtained. The trained relationship predictor is then used to detect the visual objects and predict the relationship of the textbook schematic diagram. By calculating the relationship score between the visual objects in the textbook schematic diagram, the relationship content is generated according to the relationship score, and the textbook schematic diagram structured information parsing is realized. The present invention generates the relationship between visual objects and converts it into a diagram definition language to generate a schematic diagram, and then uses a deep learning model to extract visual features and position information, predict the relationship between visual objects, and generate a description of these relationships. The present invention effectively solves the problem of small sample size of textbook schematic diagrams and little information on the relationship annotation between elements through the method of unsupervised synthetic schematic diagrams, improves the model's ability to capture structural information in schematic diagrams, and enhances the understanding of complex semantic information. Experimental results show that the present invention achieves a higher accuracy in relationship prediction and has a significant performance improvement over the prior art in the textbook question answering task, which proves the effectiveness and practicality of the method proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0036] FIG. 1 (a) is a schematic diagram of a pre-training process of a relationship predictor of a structured information parsing method based on synthetic data according to an embodiment of the present invention; FIG1( b ) is a schematic diagram showing a prediction process of a relationship predictor of a structured information parsing method based on synthetic data according to an embodiment of the present invention; Figure 2 (a) A schematic diagram of a parallel relationship among the relationship types predefined in an embodiment of the present invention; Figure 2 (b) The relationship types predefined in the embodiment of the present invention include relationship diagrams; Figure 2 (c) a schematic diagram of connection relationships among relationship types predefined in an embodiment of the present invention; Figure 2 (d) a schematic diagram of the same type of relationships among the relationship types predefined in the embodiment of the present invention; Figure 3 The embodiment of the present invention determines the relationship between visual objects when visual objects are independently displayed; Figure 4 A schematic diagram of an image generated after each relationship in an embodiment of the present invention is converted into an icon customization code; Figure 5 An example diagram of a synthetic schematic diagram generated by an embodiment of the present invention; FIG6 (a) is an example diagram of a relationship description generated by a method according to an embodiment of the present invention that is useful for completing a textbook question answering task; FIG6 (b) is an example diagram showing that the relationship description generated by the method of an embodiment of the present invention cannot fully help the textbook question answering task. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also obtain other embodiments without making creative work.

[0038] Please refer to FIG. 1 (a) and FIG. 1 (b), the method for parsing structured information of a textbook schematic diagram according to an embodiment of the present invention includes: Based on the relationships between different visual objects in the textbook diagram, a synthetic diagram is generated according to the predefined relationship types; Train a pre-built relationship predictor using synthetic diagrams and locations of visual objects; Use the trained relation predictor to perform textbook schematic visual object detection and relation prediction; According to the visual object detection and relationship prediction results of the textbook diagram, the relationship scores between the visual objects in the textbook diagram are calculated, and the relationship content is generated according to the relationship scores to realize the structured information analysis of the textbook diagram.

[0039] In one possible implementation, the synthetic schematic diagram of the embodiment of the present invention is automatically generated. Generally, the performance of a neural network in deep learning is closely related to the scale and diversity of its training data. The larger the data set and the richer the variety, the better the training results can usually be.

[0040] Before generating the synthetic schematic diagram, the embodiment of the present invention defines Figure 2 (a) Figure 2 (b) Figure 2 (c) Figure 2 (d) The parallel relationship, inclusion relationship, connection relationship and similar relationship shown in Figure 5, and a schematic diagram is designed based on these relationships.

[0041] In fact, in order to ensure that the generated schematic diagram is reasonable, the relationship between any two visual objects cannot be generated completely randomly, but must meet certain restriction rules. For example, if visual object A contains visual object B, then visual object B cannot contain visual object A, etc. The specific rule definitions are shown in Table 1.

[0042] Table 1

[0043] Based on the rules listed in Table 1, a random relationship generation algorithm between visual objects can be designed. The random relationship generation algorithm between visual objects in the embodiment of the present invention takes the visual objects that do not contain any visual objects as the starting point set, aggregates layer by layer, iteratively obtains new visual objects and adds them to the object set, and then assigns connection relationships to the objects in the object set according to the specified random probability, so as to generate relationships between different visual objects. The specific algorithm is shown in Table 2.

[0044] After randomly generating several visual objects and the relationships between them through the algorithm, the result is converted into code that conforms to the Mermaid specification of the declarative chart definition language data visualization library to generate a schematic diagram that conforms to the relationship. Specifically, it is necessary to declare each relationship and splice them together to obtain the final chart definition code.

[0045] Mermaid is an open source interactive data visualization library that uses Markdown markup language to generate charts and flowcharts. Mermaid is a text-grammar graphical tool that allows users to draw complex charts through simple text codes. It is often used to generate charts in websites or documents, such as flowcharts, time charts, Gantt charts, etc.

[0046] Flowchart, nodes are described by text surrounded by square brackets [], round brackets (), double round brackets [[]], double round brackets plus database identifier [(Database)], round brackets plus text ((text)), etc. Edges use arrow symbols -->, ---, etc. to indicate the flow direction from one node to another. You can declare the direction of the flowchart, such as from top to bottom (TB / TD), from bottom to top (BT), from left to right (LR), from right to left (RL). Supports a variety of node shapes, such as rounded rectangles, circles, diamonds, hexagons, parallelograms, trapezoids, etc. Supports different types of links, such as links with arrows, links without arrows, links with text, dotted links, thick links, etc. Sequence diagram, used to describe the time sequence of a series of events, starts with the sequenceDiagram keyword, defines participants through participant, and uses ->> to indicate message passing. Gantt chart, used for project management and task tracking, starts with the gantt keyword, defines the start time, end time, and duration of a task.

[0047] Table 2

[0048] In a possible implementation, when the embodiment of the present invention uses the synthetic schematic diagram and the position of the visual object to train a pre-established relationship predictor, the input is the schematic diagram and the position of each visual object, and the output is each relationship score matrix between two visual objects. The visual object feature representation application introduces a ResNet network with a residual block to extract the visual features of the visual object, and uses a multi-layer perceptron as a position encoder to map the position information to a high-dimensional space and concatenate it with the visual features, so that the position information of the visual object is fused with the visual information.

[0049] Formally speaking, assuming Diagram for textbook Visual object collection in , collection Each of is the visual information representation of a visual object; assuming For each visual object, the corresponding position information set, each Represents a visual object Schematic diagram in textbook , then each visual object Visual features And the location information of each visual object Mapping to high-dimensional space to obtain encoded position features It is expressed as:

[0050]

[0051]

[0052] In the formula, are the parameters that need to be trained; Then the fusion feature of visual information and position information is integrated It can be expressed as:

[0053] In the formula, [;] represents the concatenation operation. Finally, for each visual object A fusion feature that combines visual features and position features can be obtained .

[0054] Furthermore, in general visual relationship detection models, the model predicts whether there is a relationship between visual objects by calculating the fusion features of two visual objects; however, for the four predefined relationships above, especially the connection relationship, it is difficult to judge the connection relationship only by relying on the visual information and position information of each visual object. Figure 3 As shown in , the judgment of connection relationship often relies on visual information between visual objects, such as arrows, straight lines, etc. Therefore, when judging whether two visual objects have a relationship, not only the visual object pair itself is considered, but also all the visual information between the positions of the two visual objects is taken into consideration. The formal definition is as follows: For each pair of visual objects ,definition For visual objects and The visual information between and The image content between them. Using a multi-layer perceptron as a relationship predictor, we have:

[0055]

[0056] In the formula, is the parameter that needs to be trained; get each visual object With visual objects Relationship feature representation ; Then use the linear fully connected layer to predict the relationship:

[0057] In the formula, is the parameter that needs to be trained; this embodiment uses multi-label binary classification cross entropy to calculate the loss of the prediction result:

[0058] In the formula, is the sample size; is the weight to balance positive and negative samples; Is a visual object With visual objects The relationship label between them; Is a visual object With visual objects Prediction of the relationship between represents the sigmoid function, .

[0059] After completing the pre-training of the relationship prediction model, keep the parameters of the relationship prediction module and the position encoding module unchanged, and then perform the textbook diagram relationship prediction. This part mainly includes two modules, namely object detection and relationship prediction. This embodiment uses the pre-trained YOLO network to perform visual object detection. Specifically, for any textbook diagram Get the bounding box collection of visual objects ,have ; Using visual object bounding boxes From any textbook diagram Extract the visual information set of the visual object At this time, the trained relationship predictor network parameters are used to predict the relationship for specific visual objects. With visual objects have: ;

[0060] ;

[0061] Also available:

[0062]

[0063] in,

[0064]

[0065] In the formula, , are the parameters of the trained relationship predictor network.

[0066] For the If the final score is higher than 0.5, the model tends to believe that the relationship exists, otherwise it does not exist, that is:

[0067] In the formula, Representing visual objects With visual objects Is there a A value of 1 indicates that a relationship exists, while a value of 0 indicates that a relationship does not exist. Represents a tensor No. dimension; after calculating the relationship scores between all visual objects, the visual object relationship analysis of the corresponding textbook diagram is completed.

[0068] After obtaining all the relationship scores of a sub-diagram, the relationship content can be generated for these relationships. In practice, a textbook question-and-answer diagram usually has more than eight objects. Taking a textbook question-and-answer diagram with eight objects as an example, the relationship scores between all objects will reach 8*7*4=224, and this value will increase at a square rate with the number of objects. In order to prevent too much information from causing noise and interfering with the answer to the question, the embodiment of the present invention generates a relationship score for each textbook diagram. Keep the top five relationships in terms of scores, that is, the score matrix extracted above ,take out:

[0069] And satisfy:

[0070] Next, in order to generate content for each relationship based on the schematic diagram itself, the embodiment of the present invention uses MiniGPT as a relationship content generator. MiniGPT is an open source multimodal pre-trained large language model that uses the LLaMA model as a text decoder, uses pre-trained ViT and Q-Former networks to extract visual features, and uses a single linear projection layer to align the input visual features with the input of the text decoder. The pre-trained ViT and Q-Former networks and the linear projection layer together serve as the visual encoder (Visual Encoder, hereinafter referred to as VE) of the model.

[0071] The specific input template is defined as follows: [INST] <imgfeature>Please describe the visual object in this image <box1> <box2>Has <relation>Relationship[ / INST]; In this template, [INST]⋯[ / INST] represents the user input; ⋯ means the content is an image Visual features; <imgfeature>This is the feature result after the schematic diagram passes through the visual encoder; <relation>is the name of the relationship; <box>Represents the position coordinates of the visual object; <box>Including the values ​​of the left, right, top and bottom positions, that is ; Input the above template into the MiniGPT model to generate a description of the relationship; The MiniGPT model is recorded as , and the corresponding template is recorded as , then the description of the relationship for:

[0072] After obtaining all selected relationship descriptions, the corresponding relationship descriptions can be used as background knowledge to assist answer prediction. The specific answer prediction method is related to the selected test method.

[0073] The effect of the embodiment of the present invention is verified by experiments. The embodiment of the present invention uses the TQA dataset to evaluate the generated relationship description. The following content first briefly shows the relevant statistics of the schematic synthesis results and some example images, then introduces the experimental platform and experimental settings, then introduces the comparison method and experimental results, and finally conducts experimental analysis.

[0074] The embodiment of the present invention uses Mermaid to synthesize schematic diagrams. Mermaid is a text-based chart drawing syntax that allows developers to create and maintain charts in the form of text and code. By randomly generating visual objects and the relationships between visual objects, these randomly generated relationships can be converted into chart customization codes that comply with the rules by following the Mermaid syntax. Figure 4 Shown is a simple example showing the resulting image after each relationship is converted into icon customization code.

[0075] In the process of constructing the synthetic data set, the method for parsing structured information of textbook schematic diagrams in the embodiment of the present invention generated a total of 100,000 synthetic schematic diagrams, in which the number of visual objects ranged from 8 to 15, the number of connection relationships in each image ranged from 0 to 10, and the number of visual object categories ranged from 1 to 5. The total number of each relationship in all schematic diagrams is shown in Table 3.

[0076] Table 3

[0077] Figure 5 Some synthetic schematic diagrams in the method of the embodiment of the present invention are shown. It can be seen that the images in the synthetic schematic diagram data constructed by the method of the embodiment of the present invention are composed of simple geometric shapes such as line segments, rectangles, and arrows. These schematic diagrams are extremely lacking in visual information but rich in structural information, and can clearly convey the relationship between elements. Using such schematic diagrams for pre-training can effectively enhance the model's ability to capture structural information in schematic diagrams, thereby providing assistance for structural analysis of schematic diagrams. The following content mainly shows the results of pre-training the relationship predictor on a set of synthetic diagrams and the results of using the generated relationship descriptions for textbook question answering, and shows the generated examples and analysis.

[0078] (1) Experimental results of the relationship predictor training phase The pre-training results of the relationship predictor are shown in Table 4. It can be found that for the four proposed relationships, all have more than 80% However, among these four relationships, the connection relationship is more difficult to predict than the parallel, similar, and inclusion relationships. This may be because the other three relationships are more closely related to the position information of the visual object, while the connection relationship is relatively free.

[0079] Table 4

[0080] (2) Experimental results of the textbook question-answering phase The most advanced models on the TQA datasets at each stage are used for comparison, and the experimental results are shown in Table 5. It can be seen that compared with the ISAAQ model, SPN has a 1.1% improvement in effect, which illustrates the overall effectiveness of the model of the embodiment of the present invention; while w / o pre-train ResNet illustrates the effectiveness of the relationship description generated in the method of the embodiment of the present invention. Only adding the generated relationship description has a 0.97% improvement in effect, and the pre-trained ResNet brings an additional improvement.

[0081] Table 5

[0082] Figure 6 (a) and Figure 6 (b) are some sample diagrams of relation description generation, showing some relations with high scores and relation descriptions generated based on them. Figure 6 (a) shows an example of the generated relation description being helpful in completing the textbook question-answering task. The relation description in this example provides effective help in answering the question, where the description of the connecting relation is "marks that uranium-236 directly splits into barium and krypton", which directly helps answer the question "Which two smaller particles are produced by the fission reaction in the figure", and this also proves the effectiveness of this part of the content. Figure 6 (b) shows an example that is not completely successful. Although in this example, the relationship is successfully described and an introduction to the change of the moon phase is obtained, this does not help answer the question. The reason is that the relation description in the corresponding study is mainly aimed at the relationship between two visual objects, while the related question "How many moon phases are shown in the figure" requires counting all visual objects, which may require more complex relation design or operations such as synthesis of relations. Therefore, it can be further studied as a future research content.

[0083] Another embodiment of the present invention further provides a textbook schematic diagram structured information parsing system, comprising: A synthetic diagram generation module, for generating a synthetic diagram according to a predefined relationship type based on the relationship between different visual objects in the textbook diagram; a relation predictor training module for training a pre-established relation predictor using the synthetic schematic diagram and the locations of the visual objects; Visual object detection and relationship prediction module, used to perform visual object detection and relationship prediction on textbook diagrams using trained relationship predictors; The relationship content generation module is used to calculate the relationship scores between visual objects in the textbook diagram based on the visual object detection and relationship prediction results of the textbook diagram, generate relationship content based on the relationship scores, and realize the structured information analysis of the textbook diagram.

[0084] Another embodiment of the present invention further provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the textbook schematic diagram structured information parsing method.

[0085] Another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for parsing structured information of textbook schematic diagrams is implemented.

[0086] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the textbook diagram structured information parsing method of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the server.

[0087] The electronic device may be a computing device such as a smart phone, a notebook, a PDA, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may also include more or fewer components, or a combination of certain components, or different components, for example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0088] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0089] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0090] It should be noted that the information interaction, execution process and other contents between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0091] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0093] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.< / box> < / box> < / relation> < / imgfeature> < / relation> < / box1> < / imgfeature> < / box> < / box> < / relation> < / imgfeature> < / relation> < / box1> < / imgfeature>

Claims

1. A method for analyzing structured information of textbook schematic diagrams, characterized in that: include: Based on the relationships between different visual objects in the textbook diagram, a synthetic diagram is generated according to the predefined relationship types; Train a pre-built relationship predictor using synthetic diagrams and locations of visual objects; Use the trained relation predictor to perform textbook schematic visual object detection and relation prediction; According to the visual object detection and relationship prediction results of the textbook diagram, the relationship scores between the visual objects in the textbook diagram are calculated, and the relationship content is generated according to the relationship scores to realize the structured information analysis of the textbook diagram.

2. The method for analyzing structured information of textbook schematic diagrams according to claim 1, characterized in that: In the step of generating a synthetic schematic diagram according to the relationship between different visual objects in the textbook schematic diagram according to predefined relationship types, the predefined relationship types include parallel relationship, inclusion relationship, connection relationship and similar relationship.

3. The method for analyzing structured information of textbook schematic diagrams according to claim 1, characterized in that: In the step of generating a synthetic schematic diagram according to the relationship between different visual objects in the textbook schematic diagram and the predefined relationship type, a number of visual objects and the relationships between visual objects are randomly generated based on the visual object relationship restriction rules: visual objects that do not contain any visual objects are used as the starting point set, and they are aggregated layer by layer, and new visual objects are iteratively obtained and added to the object set, and connection relationships are assigned to objects in the object set according to random probability, that is, relationships between different visual objects are randomly generated.

4. The method for analyzing structured information of textbook schematic diagrams according to claim 3, characterized in that: For different visual objects A, B, and C, the visual object relationship restriction rules include: If A contains B, then B cannot contain A; If A and B are in parallel, then B cannot contain A; If A and B are in parallel, then A cannot contain B; If A contains B, then B cannot connect to A; If A contains B, then A cannot connect to B; If A contains B and C does not contain any element, then A and C cannot be of the same type; If A is of the same kind as B, and B is of the same kind as C, then A is of the same kind as C; If A contains B, and B contains C, then A contains C; If A contains B, and A and C are parallel, then C cannot contain B; If A contains B, and A contains C, then B and C are parallel; If A and B are not contained by any element, then A and B are parallel.

5. The method for analyzing structured information of textbook schematic diagrams according to claim 3, characterized in that: After randomly generating several visual objects and the relationships between them, each relationship is converted into code that conforms to the Mermaid specification of the declarative chart definition language data visualization library, and then spliced ​​to generate a synthetic schematic diagram that conforms to the corresponding relationship.

6. The method for analyzing structured information of textbook schematic diagrams according to claim 1, characterized in that: When the pre-established relationship predictor is trained by using the synthetic schematic diagram and the position of the visual object, the relationship predictor applies a residual network ResNet that introduces a residual block to extract visual features of the visual object, and uses a multi-layer perceptron as a position encoder to map the position information to a high-dimensional space and concatenate it with the visual features, so that the position information of the visual object is fused with the visual information; Assumptions Diagram for textbook Visual object collection in , collection Each of is the visual information representation of a visual object; assuming For each visual object, the corresponding position information set, each Represents a visual object Schematic diagram in textbook , then each visual object Visual features And the location information of each visual object Mapping to high-dimensional space to obtain encoded position features It is expressed as: In the formula, are the parameters that need to be trained; The fusion feature of visual information and position information is integrated It is expressed as: Among them, [;] represents the splicing operation; For each visual Get a fusion feature that combines visual features and position features ; For each pair of visual objects ,definition For visual objects and The visual information between and The image content between them; a multi-layer perceptron is used as the relationship predictor: In the formula, is the parameter that needs to be trained; get each visual object With visual objects Relationship feature representation ; Use a linear fully connected layer for relationship prediction: In the formula, are the parameters that need to be trained; the loss of the prediction results is calculated using multi-label binary cross entropy: In the formula, is the sample size; is the weight to balance positive and negative samples; Is a visual object With visual objects The relationship label between them; Is a visual object With visual objects Prediction of the relationship between represents the sigmoid function, .

7. The method for analyzing structured information of textbook schematic diagrams according to claim 6, characterized in that: The steps of using the trained relationship predictor to perform textbook schematic visual object detection and relationship prediction include: Use the pre-trained end-to-end object detection YOLO network for visual object detection, for any textbook diagram Get the bounding box collection of visual objects ,have ; Using visual object bounding boxes From any textbook diagram Extract the visual information set of the visual object At this time, the trained relationship predictor network parameters are used to predict the relationship for specific visual objects. With visual objects have: ； ； Also available: in, In the formula, , are the trained relationship predictor network parameters; For the If the final score is higher than 0.5, it is determined that there is a corresponding relationship, otherwise there is no corresponding relationship: In the formula, Representing visual objects With visual objects Is there a A value of 1 indicates that a relationship exists, while a value of 0 indicates that a relationship does not exist. Represents a tensor No. dimension; after calculating the relationship scores between all visual objects, the visual object relationship analysis of the corresponding textbook diagram is completed.

8. The method for analyzing structured information of textbook schematic diagrams according to claim 7, characterized in that: The step of calculating the relationship scores between the visual objects in the textbook schematic diagram according to the results of the visual object detection and relationship prediction of the textbook schematic diagram, and generating the relationship content according to the relationship scores comprises: for each textbook schematic diagram Keep the top five relationships by score: And satisfy: The multimodal pre-trained large language model MiniGPT is used as the relational content generator, and the input template is defined as follows: [INST] <imgfeature>Please describe the visual object in this image <box1> <box2>Has <relation> Relationship[ / INST];< / relation> < / box1> < / imgfeature> In this template, [INST]⋯[ / INST] represents the user input; ⋯ means the content is an image Visual features; <imgfeature>This is the feature result after the schematic diagram passes through the visual encoder; <relation>is the name of the relationship; <box>Represents the position coordinates of the visual object; <box>Including the values ​​of the left, right, top and bottom positions, that is ; Input the above template into the MiniGPT model to generate a description of the relationship; The MiniGPT model is recorded as , and the corresponding template is recorded as , then the description of the relationship for:< / box> < / box> < / relation> < / imgfeature> After obtaining all selected relationship descriptions, relationship content generation is performed to achieve structured information parsing of textbook diagrams.

9. A textbook schematic diagram structured information parsing system, characterized in that: include: A synthetic diagram generation module, for generating a synthetic diagram according to a predefined relationship type based on the relationship between different visual objects in the textbook diagram; a relation predictor training module for training a pre-established relation predictor using the synthetic schematic diagram and the locations of the visual objects; Visual object detection and relationship prediction module, used to perform visual object detection and relationship prediction on textbook diagrams using trained relationship predictors; The relationship content generation module is used to calculate the relationship scores between visual objects in the textbook diagram based on the visual object detection and relationship prediction results of the textbook diagram, generate relationship content based on the relationship scores, and realize the structured information analysis of the textbook diagram.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for parsing structured information of textbook schematic diagrams as claimed in any one of claims 1 to 8 is implemented.

Images