A sketch representation enhancement method and system based on self-supervised learning

Abstract

The application discloses a sketch representation enhancement method and system based on self-supervised learning, in the image and text feature extraction stage, the sketch is augmented and transformed, the text box content in the sketch is recognized, and the text content and the sketch are encoded respectively to obtain text features and image features; in the sketch representation enhancement stage under the guidance of the text, the text features are used as the basis, and a guide attention unit is applied to enhance the image features; in the contrast self-supervised learning stage, the original image and the augmented sketch enhancement features are mapped into a low-dimensional vector space through a projection function, and the loss is calculated by the low-dimensional vector and the model is optimized.

Description

Technical Field

[0001] This invention belongs to the field of schematic diagram processing technology, specifically relating to a method and system for enhancing schematic diagram representations based on self-supervised learning. Background Technology

[0002] Traditional online learning is merely a simple human-computer interaction method, failing to provide learners with personalized and intelligent instruction. Therefore, smart education has emerged, and intelligent question-answering systems are a crucial component. They not only act as teachers in traditional education but also serve as learning assistants for users. Intelligent question-answering systems require machines to possess reading comprehension capabilities. Machine reading comprehension requires the machine to automatically generate answers to questions, and its natural language questions are typically related to a given text. Unlike machine reading comprehension, visual question answering adds image information to the input. A visual question answering task presents an image and a natural language question, requiring the user to infer the correct answer based on the image's visual elements and common sense.

[0003] In recent years, with the increasing popularity of research on text, image, and knowledge reasoning, visual question answering (VQA) has become increasingly important. In VQA, the answers are not given through natural language; machines often need to find features in a given image and infer the correct answer. To correctly answer VQA questions, machines not only need natural language processing and computer vision capabilities, but also the ability to fuse cross-modal knowledge.

[0004] Traditional visual question answering tasks only accept images and questions as input, while the educational domain contains numerous diagrams, often requiring reference to relevant professional knowledge to answer questions. It typically uses abstract graphical symbols rather than real images to represent elements and scenes, and the representation of elements varies significantly across different subject areas, as seen in datasets like CSDia, AI2D, and Plotqa. Therefore, traditional visual question answering datasets and related research cannot meet the needs of the educational domain. Textbook question answering research combines machine reading comprehension and visual question answering tasks, and is also an extension of these two tasks. Diagrams in the curriculum domain are not entirely equivalent to natural images, and the multimodal information fusion in textbook question answering differs significantly from that in visual question answering. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for enhancing the representation of schematic diagrams based on self-supervised learning, which addresses the shortcomings of the prior art. It utilizes the diversity of schematic diagram representations and the presence of text boxes in the diagrams to solve the technical problems of schematic diagram understanding and feature extraction.

[0006] The present invention adopts the following technical solution:

[0007] A graph representation enhancement method based on self-supervised learning includes the following steps:

[0008] S1. Perform image augmentation on the schematic diagram to obtain the corresponding augmented schematic diagram. Use a recurrent neural network to extract the text features of multiple text boxes in the schematic diagram, and use a convolutional neural network to extract the image features of the schematic diagram to obtain the schematic diagram features.

[0009] S2. Use self-attention units to enhance the text features obtained in step S1 to obtain text-enhanced features. Use the text-enhanced features as guidance to enhance the schematic features obtained in step S1 by the attention of the text-enhanced features on the schematic features to obtain the schematic representation enhancement network.

[0010] S3. Using the schematic representation enhancement network obtained in step S2, feature extraction is performed on the schematic diagram and the corresponding augmented schematic diagram in step S1 to obtain an enhanced feature set. The enhanced feature set is projected from high-dimensional features to low-dimensional feature vectors. The loss function is calculated based on the low-dimensional feature vectors. The parameters of the schematic representation enhancement network are optimized by minimizing the loss value of the loss function, thereby realizing schematic representation enhancement based on self-supervised learning.

[0011] Specifically, step S1 is as follows:

[0012] S101. Augment the schematic diagram to obtain the corresponding augmented schematic diagram;

[0013] S102. When extracting schematic features, the ResNet convolutional neural network relies on training to retain effective shallow convolutional features, so that the final output schematic features contain comprehensive and diverse information. ResNet is used to extract schematic features and generate image feature vectors.

[0014] S103. Use the word vectors pre-trained by BERT for word embedding representation, and use the extended network recurrent gating unit of the long short-term memory network to extract the text box features in the schematic diagram. Take the hidden layer features of the last word in the text box as the feature vector of the text box content, and concatenate the text box feature vectors to form the text features in the corresponding schematic diagram.

[0015] Furthermore, in step S101, the augmentation methods include converting to grayscale, channel elimination, horizontal flipping, low-angle rotation, random occlusion, and / or skewing.

[0016] Specifically, step S2 is as follows:

[0017] S201. Use self-attention units to enhance the text features obtained in step S1 to obtain text-enhanced features;

[0018] S202. Using the corresponding enhanced text features as guidance, the schematic diagram feature vector obtained in step S1 is enhanced to represent the object features referred to by the text in the schematic diagram. Combining the convolutional neural network and recurrent neural network in step S1, the text feature enhancement obtained in step S201 is used to construct the schematic diagram representation enhancement network.

[0019] Furthermore, step S201 specifically includes:

[0020] Set the query, key, and value to the same dimension d, calculate the dot product attention of a query with all keys, and concatenate multiple query vectors to obtain the query matrix Q; learn the relationship between vectors within the text feature matrix through a multi-head attention mechanism, and obtain the reconstructed features by weighted summation of all feature vectors, summing them to obtain the original enhanced features, and then normalizing them; after normalization, use a fully connected network as a feedforward layer, and then add the original enhanced features with the output features of the fully connected network and normalize them to obtain the final enhanced features of the text in the text box in the diagram.

[0021] Furthermore, step S202 specifically includes:

[0022] The schematic diagram features are expanded and concatenated to obtain schematic diagram extended features. After multi-head attention, addition and normalization are used. The inputs are text enhancement features and schematic diagram extended features. The schematic diagram features are enhanced by the attention of text enhancement features on schematic diagram features. The output is schematic diagram enhanced features. Combining the convolutional neural network and recurrent neural network in step S1, the text feature enhancement obtained in step S201 constructs a schematic diagram representation enhancement network.

[0023] Specifically, step S3 is as follows:

[0024] S301. Randomly select a schematic augmentation method obtained in step S1 as the comparison object of the original image, denote the schematic representation enhancement network as f, and obtain the set of enhanced features in the original image and the augmented schematic based on the schematic representation enhancement network f.

[0025] S302. Using a fully connected network with one hidden layer as a projection head, the enhanced feature set obtained in step S1 is projected from high dimension to low dimension vector space to obtain the projection vector of high dimension feature vector.

[0026] S303. Denote the feature vectors corresponding to the same text box in the original image and the augmented image as positive sample pairs, and the feature vectors corresponding to different text boxes as negative sample pairs. Calculate the positive sample pair z using cosine similarity. i z j The similarity of the positive sample pairs z i z jAfter swapping positions, the loss is calculated again, and the average loss of all positive sample pairs is taken as the final loss. By minimizing the final loss L, backpropagation is used to update and optimize the schematic feature learning network f and the projection network g. The schematic representation enhancement network f is optimized as the image feature extraction module, thus realizing schematic representation enhancement based on self-supervised learning.

[0027] Furthermore, in step S303, the final loss L is minimized as follows:

[0028]

[0029] Where m is the number of text boxes in the diagram, l is the entropy noise comparison estimate, and k is a variable.

[0030] Furthermore, positive sample pairs z i z j similarity (z i ,z j Specifically:

[0031] similarity(z i ,z j )=z i T z j / (||z i ||||z j ||)

[0032] Where ||·|| is the magnitude of the vector, and T is the transpose of the vector.

[0033] Secondly, embodiments of the present invention provide a schematic representation enhancement system based on self-supervised learning, comprising:

[0034] The feature module performs image augmentation on the schematic diagram to obtain the corresponding augmented schematic diagram. It uses a recurrent neural network to extract the text features of multiple text boxes in the schematic diagram and a convolutional neural network to extract the image features of the schematic diagram to obtain the schematic diagram features.

[0035] The enhancement module uses self-attention units to enhance the text features obtained by the feature module, resulting in text-enhanced features. Using these text-enhanced features as guidance, the schematic representation enhancement network is obtained by enhancing the schematic representation features obtained by the feature module based on the attention of the text-enhanced features to the schematic representation features.

[0036] The representation module uses the schematic representation enhancement network obtained from the enhancement module to extract features from the schematic diagram of the feature module and the corresponding augmented schematic diagram, respectively, to obtain an enhanced feature set. The enhanced feature set is projected from high-dimensional features to low-dimensional feature vectors, and a loss function is calculated based on the low-dimensional feature vectors. The parameters of the schematic representation enhancement network are optimized by minimizing the loss value of the loss function, thereby realizing schematic representation enhancement based on self-supervised learning.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] A self-supervised learning-based schematic representation enhancement method is proposed. Image augmentation increases the amount of training data and diversifies the data presentation, improving the generalization ability of the schematic representation enhancement network. By designing text feature enhancement steps and text-guided schematic feature representation enhancement steps, text features and schematic features are effectively fused. A self-supervised training method using contrastive learning is designed and implemented to pre-train the schematic representation enhancement network. Utilizing text boxes present in the schematic as an aid, a self-supervised learning-based schematic representation enhancement method is designed and implemented, capable of extracting feature vectors of multiple objects in the schematic, expanding a single schematic feature vector into feature vectors of multiple objects, effectively enhancing the schematic representation, providing rich schematic feature representations for downstream tasks, and effectively improving the performance of downstream tasks.

[0039] Furthermore, ResNet is used to extract image features. The ResNet network, during schematic feature extraction, can retain effective shallow convolutional features through training, ensuring that the final output schematic features contain comprehensive and diverse information. All text boxes and content in the original schematic have significant guiding significance for subsequent schematic representation reinforcement learning. To more effectively store word features and preserve the relationships between words, word embedding is commonly used in natural language processing, mapping each word to a vector. Currently, the best-performing word vector representations typically use semantic space mapping methods pre-trained by language models on large corpora. This invention uses BERT pre-trained word vectors for word embedding representation. A gated recurrent unit (GRU), an extension of the Long Short-Term Memory (LSTM) network, is used to extract text box features from the schematic.

[0040] Furthermore, image augmentation techniques discovered and applied on certain specific datasets cannot often be directly and effectively applied to other datasets because different datasets have different characteristics; illustration Figure 1Generally, it includes logical symbols such as images, arrows, text information, and basic graphics, expressing information with high purity and compactness, and the content serves the same theme; augmentation methods on natural images are not all applicable to the augmentation of schematic diagrams, so this invention designs an augmentation method suitable for schematic diagrams based on their characteristics, which facilitates subsequent self-supervised learning.

[0041] Furthermore, text and image information are different modalities and cannot be fused through simple addition or concatenation. Therefore, this invention designs text feature enhancement and text-guided schematic diagram feature representation enhancement steps to effectively fuse multimodal information.

[0042] Furthermore, the text in the illustrations may contain multiple meanings; for example, "South Pole" in an illustration of Earth's structure may have different meanings than "South Pole" in an illustration of a magnet. By analyzing the relationships between text boxes in the illustrations, the precise literal meaning of a particular text box can be inferred. For instance, "South Pole" appearing alongside "equator" represents the South Pole on Earth, while appearing alongside "magnetic field" represents the South Pole on a magnetic material. In addition, there may be relationships between the text box labels in the illustrations. For example, relationships with obvious literal meanings, such as "branch" and "root," can be directly extracted through the feature representation of the text boxes without requiring image information. Therefore, this invention uses a self-guided attention (SA) unit to enhance the text features.

[0043] Furthermore, the global features obtained using image feature extraction models are insufficient to represent all objects and their complex relationships in a schematic diagram. Textual features are of significant guiding importance for schematic diagram feature learning. Using text as guidance to enhance schematic diagram features and represent the features of the objects referred to by the text boxes in the schematic diagram is crucial for schematic diagram feature extraction and understanding. Therefore, text-guided schematic diagram representation enhancement utilizes Guide Attention (GA) units to enhance schematic diagram features based on text content.

[0044] Furthermore, current datasets for schematic diagrams are limited, and existing datasets have limited data volume. Downstream tasks such as textbook question answering typically use schematic diagram enhancement features extracted by representation augmentation networks, which are then fused across multiple modalities. The effectiveness of these enhancements can only be verified by evaluating question-answering accuracy. Training representation augmentation networks using downstream tasks generally yields mediocre results. Therefore, this invention designs and implements self-supervised training using contrastive learning.

[0045] Furthermore, because the projection vectors are calculated using the same projection network, the similarity between the projection vectors reflects the similarity between the original feature vectors. Ideally, after training, the projection vectors of the same object in the original image and the augmented schematic have high similarity, while the projection vectors of different objects have low similarity. The projection network pre-trained in this way will be discarded, and the schematic feature extraction network can be directly used in downstream tasks.

[0046] Furthermore, the similarity between two vectors is measured by taking the cosine of the angle between them. The cosine of the angle determines whether the two vectors point in roughly the same direction. When the two vectors point in the same direction, the cosine similarity is 1; when the angle between them is 90°, the cosine similarity is 0; and when the two vectors point in completely opposite directions, the cosine similarity is -1. This result is independent of the length of the vectors and depends only on their direction of travel.

[0047] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0048] In summary, this invention maps the enhanced features of the original image and the augmented schematic to a low-dimensional vector space using a projection function, and calculates the loss and optimizes the model using the low-dimensional vector.

[0049] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0050] Figure 1 This is a framework diagram of the self-supervised learning-based schematic representation enhancement method of the present invention;

[0051] Figure 2 This is a schematic diagram of the augmentation method of the present invention;

[0052] Figure 3 This is a schematic diagram of the text feature enhancement method of the present invention;

[0053] Figure 4 This is a schematic diagram of the characterization enhancement method of the present invention;

[0054] Figure 5 This is a diagram illustrating the similarity of object features between the original image and the augmented image. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0059] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0060] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0061] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0062] This invention provides a schematic representation enhancement method based on self-supervised learning. In the image and text feature extraction stage, the schematic diagram is augmented and transformed to identify the text box content in the schematic diagram. The text content and the schematic diagram are encoded to obtain text features and image features, respectively. In the text-guided schematic representation enhancement stage, the image features are enhanced by applying a guided attention unit based on the text features. In the contrastive self-supervised learning stage, this invention maps the enhanced features of the original image and the augmented schematic diagram to a low-dimensional vector space through a projection function, and calculates the loss and optimizes the model using the low-dimensional vector.

[0063] This invention primarily focuses on methods for understanding diagrams in textbook Q&A sections and fusing multimodal information. It enables machines to understand the text content and diagrams in textbook Q&A sections and correctly answer related questions containing image information. This expands the information receiving and processing capabilities of intelligent Q&A assistants from a single-modality format containing only text to a multimodal format containing both text and images.

[0064] Please see Figure 1 This invention presents a diagram feature enhancement method based on self-supervised learning. Leveraging the crucial guiding role of text box content in understanding diagrams, it proposes a Diagram Feature Enhancement Network (Dia-FEN). Utilizing the diverse expressive forms of diagrams, it achieves diagram feature enhancement based on self-supervised learning, comprising the following steps:

[0065] S1. Image and Text Feature Extraction

[0066] Based on the image augmentation methods on natural images, an augmentation method suitable for schematic diagrams is proposed. OCR is used to identify all text box contents in the original image, providing a basis for subsequent representation enhancement. Secondly, the features of the original image and the augmented image are obtained by using a convolutional neural network ResNet-50. For each text box content in the schematic diagram, word embedding is performed using BERT pre-trained word vectors and then input into a recurrent neural network GRU to obtain text features.

[0067] S101. Augment the schematic diagram to obtain the corresponding augmented schematic diagram;

[0068] Please see Figure 2 The augmentation methods include the following six:

[0069] 1) Convert to grayscale image

[0070] 2) Channel elimination

[0071] Typically, an RGB image contains three color channels: red, green, and blue. Removing one color channel from the illustration while retaining the other two results in an augmented illustration.

[0072] 3) Horizontal flip

[0073] The schematic augmentation method only uses left-right flip transformation, resulting in minimal loss of information in the original image.

[0074] 4) Low-angle rotation

[0075] The diagram augmentation method uses only small angles (less than 45 degrees) of rotation, and the blank areas in the original image after rotation are filled with white.

[0076] 5) Random occlusion

[0077] Random occlusion involves randomly adding occlusion blocks to an image. Using small blank occlusion blocks and randomly adding them to the illustration is also a method of illustration augmentation.

[0078] 6) Oblique cut

[0079] Skewing typically stretches and transforms an image in a specific way. Intuitively, it's like the same image seen by the human eye from different angles, with the blank spaces after skewing filled with white.

[0080] S102, Image Feature Representation;

[0081] When extracting schematic features, the ResNet network relies on training to retain effective shallow convolutional features, so that the final output schematic features contain comprehensive and diverse information. ResNet is used to extract schematic features and generate feature vectors.

[0082] S103, Text Feature Representation;

[0083] Word embeddings are performed using pre-trained word vectors from BERT (Bidirectional Encoder Representation from Transformers). In the original diagram, the text box content is represented by querying the pre-trained BERT word vectors to obtain the word vectors of all words within the text box. This invention uses a Gated Recurrent Unit (GRU), an extension of the Long Short-Term Memory (LSTM) network, to extract features from the text boxes in the diagram. The diagram contains m text boxes; the hidden layer features of the last word in each text box are taken as the feature vector of that text box content. Finally, the feature vectors of the m text boxes are concatenated to form the text features in the diagram.

[0084] S2. Text-guided graphical representation enhancement

[0085] Self-Guide Attention (SA) is used to enhance text features. The input is text features; the self-attention mechanism mines the relationships between each text box and uses these relationships to enhance the original text features, ensuring they contain correct semantic information. The output is the enhanced text features. Then, the text is used as guidance to enhance the schematic representation features, representing the object features the text refers to in the schematic. The schematic features are expanded by concatenating m object features to obtain extended schematic features. Text-guided schematic representation enhancement uses a similar attention unit (Guide Attention, GA) to text representation enhancement. To highlight the image features of the schematic, image features are summed and normalized after multi-head attention. Its input is the enhanced text features and the extended schematic features; the enhanced schematic features are enhanced based on the attention given to them by the enhanced text features.

[0086] S201. Use self-guided attention (SA) units to enhance text features;

[0087] The input to an attention unit consists of a query, a key, and a value. For computational convenience, the query, key, and value are typically set to the same dimension d. First, the dot product attention of one query with all keys is computed. The multi-head attention mechanism consists of multiple parallel attention heads, which can further enhance the representational ability of the features involved in the attention mechanism. This approach prevents the multi-head attention network from becoming too complex and converging too slowly. Each attention head corresponds to a different dot product attention function, and the multi-head attention function ultimately outputs the features.

[0088] Please see Figure 3 In actual calculations, a set of queries is usually calculated simultaneously, that is, multiple query vectors are concatenated to obtain the query matrix Q.

[0089] In the text feature enhancement of the schematic diagram, the relationship between the vectors within the text feature matrix is ​​learned through a multi-head attention mechanism, and the reconstructed features are obtained by weighted summation of all feature vectors. The original enhanced features are then obtained by summing them and normalizing them.

[0090] After normalization, a fully connected layer (FC) is used as the feedforward layer. Finally, the original enhanced features are added to the FC output features and normalized to obtain the final enhanced features of the text within the text box in the diagram.

[0091] S202. Using text enhancement features as guidance, enhance the features of the schematic diagram to represent the object features referred to by the text enhancement features in the schematic diagram;

[0092] Please see Figure 4 First, the schematic diagram features are expanded and concatenated to obtain the expanded schematic diagram features. The number of text boxes in the schematic diagram is the same as the number of text-enhanced feature vectors. Schematic diagram representation enhancement guided by text-enhanced features uses a similar attention unit (Guide Attention, GA) to text representation enhancement. To highlight the image features of the schematic diagram, summation and normalization are used after multi-head attention. Its inputs are the text-enhanced features and the expanded schematic diagram features. The schematic diagram features are enhanced by the attention given to them by the text-enhanced features, and its output is the expanded schematic diagram features.

[0093] In summary, by extracting schematic enhancement features through text-guided attention units, the single feature vector in the original image is expanded into multiple feature vectors, which represent the features of the object pointed to by the schematic text box in the image.

[0094] S3, Contrastive Self-Supervised Learning

[0095] The goal is to make the objects pointed to by the same text boxes in the original image and the augmented schematic as similar as possible, while different text boxes point to different objects. Projection vectors are calculated using the same projection network, and the similarity between these projection vectors reflects the similarity between the original feature vectors. After training, the projection vectors of the same object in the original image and the schematic have high similarity, while the projection vectors of different objects have low similarity. The projection network pre-trained in this way is discarded, and the schematic feature extraction network can be directly used in downstream tasks.

[0096] The specific steps are as follows: Feature extraction is performed using a schematic representation enhancement network; high-dimensional features are projected onto low-dimensional feature vectors using a projection network; loss is calculated using the projection vectors; and the model is optimized through backpropagation.

[0097] S301. Each time, a schematic representation augmentation method is randomly selected as the comparison object of the original image. The schematic representation augmentation network is denoted as f. Based on f, the feature sets of the original image and the augmented schematic can be obtained.

[0098] S302. When calculating the loss, the high-dimensional feature vector is projected into a low-dimensional vector space. Here, a fully connected network with one hidden layer is used as the projection head to obtain the projected vector of the high-dimensional feature vector;

[0099] S303. Calculate the cosine similarity between two projection vectors z. i z j The similarity of the positive sample pairs z i z j After swapping positions, the loss is calculated again, and the average of the losses of all positive sample pairs is taken as the final loss. By minimizing the final loss L, backpropagation is used to update and optimize the schematic feature learning network f and the projection network g. After optimizing the schematic representation enhancement network f, it is used as the image feature extraction module. The extracted features can be used in downstream tasks as image features for subsequent operations, such as schematic question answering, image-text matching, etc.

[0100] Two projection vectors z i z j The similarities are specifically as follows:

[0101] similarity(z i ,z j )=z i T z j / (||z i ||||z j ||)

[0102] Where ||·|| is the magnitude of the vector.

[0103] The projection network pre-trained in this way will be discarded, and the schematic feature extraction network f can be directly used in downstream tasks.

[0104] Let z i and z j Let z be the feature vector projections of the same text box representing the object in the original image and the augmented image, respectively. i and z j These are called positive sample pairs, and z i The feature vector z of the objects referred to by other text boxes in the original and augmented imagesk These are called negative sample pairs.

[0105] The loss function for a positive sample pair (i,j) is calculated using noise contrastive estimation (NCE):

[0106]

[0107] Where τ is a constant that can expand the range of cosine similarity, exp(·) is an exponential function with base e, and log(·) is a logarithmic function.

[0108] The method for calculating the loss is called the Normalized Temperature-Scaled Cross Entropy Loss (NT-Xent):

[0109]

[0110] Where m is the number of text boxes in the diagram, l is the entropy noise comparison estimate, and k is a variable.

[0111] The entropy noise comparison estimation is as follows:

[0112]

[0113] Among them, similarity(z i ,z j ) represents cosine similarity, t is a variable parameter, and exp() is an exponential function.

[0114] In another embodiment of the present invention, a schematic representation enhancement system based on self-supervised learning is provided. This system can be used to implement the above-mentioned schematic representation enhancement method based on self-supervised learning. Specifically, the schematic representation enhancement system based on self-supervised learning includes modules, modules, modules, modules, and modules.

[0115] in,.

[0116] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0117] In the case study of the pre-trained model, a schematic representation enhancement network trained for 100 epochs was used as the pre-trained model. Object features were extracted from the schematic diagram of the human respiratory system and its augmented schematic diagram for comparison. The absolute value of the cosine similarity was used to calculate the similarity between each object feature. The similarity of object features between the original image and the augmented schematic diagram is shown below. Figure 5 As shown, the text boxes marked with an asterisk (*) represent the objects in the augmented image. The similarity heatmap clearly shows that the objects referred to by the same text box have high feature similarity in both the original image and the augmented image, while the similarity between features of different objects is low, indicating the effectiveness of self-supervised learning.

[0118] Furthermore, the similarity between the loss value and feature vector obtained using the contrastive learning algorithm is insufficient to demonstrate the effectiveness of the pre-training of the schematic representation enhancement network. Therefore, this invention applies the pre-trained model to the image feature extraction module in a textbook question-answering model and designs comparative experiments to verify the effectiveness of the schematic representation enhancement network and its pre-training. The accuracy of the textbook question-answering task is significantly improved.

[0119] In summary, the present invention provides a method and system for enhancing schematic representations based on self-supervised learning, which effectively enhances schematic representations and provides richer image information for downstream tasks such as multimodal information fusion reasoning, schematic feature extraction, and image-text matching.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for enhancing schematic representations based on self-supervised learning, characterized in that, Includes the following steps: S1. Perform image augmentation on the schematic diagram to obtain the corresponding augmented schematic diagram. Use a recurrent neural network to extract the text features of multiple text boxes in the schematic diagram, and use a convolutional neural network to extract the image features of the schematic diagram to obtain the schematic diagram features. S2. Enhance the text features obtained in step S1 to obtain text-enhanced features. Use the text-enhanced features as guidance to enhance the schematic features obtained in step S1 based on the attention of the text-enhanced features to the schematic features, thus obtaining the schematic representation enhancement network. Step S2 specifically involves: S201. Use self-attention units to enhance the text features obtained in step S1 to obtain enhanced text features; S202. Using the corresponding text enhancement features as guidance, the schematic diagram features are enhanced to represent the object features referred to by the text in the schematic diagram. A schematic diagram representation enhancement network is constructed by combining convolutional neural networks, recurrent neural networks, and the text feature enhancement mechanism of self-attention units. Specifically: The schematic diagram features are expanded and concatenated to obtain the schematic diagram extended features. After multi-head attention, addition and normalization are used. The input is the text enhancement features and the schematic diagram extended features. The schematic diagram features are enhanced by the attention of the text enhancement features on the schematic diagram features. The output is the schematic diagram enhanced features. The schematic diagram representation enhancement network is constructed by combining the convolutional neural network and recurrent neural network in step S1 with the text feature enhancement mechanism of the self-attention unit. S3. Using the schematic representation enhancement network obtained in step S2, feature extraction is performed on the schematic diagram and the corresponding augmented schematic diagram in step S1 to obtain an enhanced feature set. The enhanced feature set is projected from high-dimensional features to low-dimensional feature vectors. The loss function is calculated based on the low-dimensional feature vectors. The parameters of the schematic representation enhancement network are optimized by minimizing the loss value of the loss function, thereby realizing schematic representation enhancement based on self-supervised learning.

2. The diagram representation enhancement method based on self-supervised learning according to claim 1, characterized in that, Step S1 is as follows: S101. Augment the schematic diagram to obtain the corresponding augmented schematic diagram; S102. Use a convolutional neural network ResNet to extract image features of the schematic diagram to obtain schematic diagram features; S103. Use the word vectors pre-trained by BERT for word embedding representation. Use the extended network recurrent gating unit of the long short-term memory network to extract multiple text boxes in the diagram. Take the hidden layer features of the last word in the text box as the feature vector of the text box content. Concatenate the text box feature vectors to form the text features in the corresponding diagram.

3. The diagram representation enhancement method based on self-supervised learning according to claim 2, characterized in that, In step S101, the augmentation methods include converting to grayscale, channel elimination, horizontal flipping, low-angle rotation, random occlusion, and / or skewing.

4. The diagram representation enhancement method based on self-supervised learning according to claim 1, characterized in that, Step S201 is as follows: Set the query, key, and value to be the same dimension. The process involves calculating the dot product attention of a query with all keys, concatenating multiple query vectors to obtain the query matrix Q; learning the relationships between vectors within the text feature matrix through a multi-head attention mechanism, and obtaining reconstructed features by weighted summation of all feature vectors, summing them to obtain the original enhanced features, and then normalizing them; after normalization, using a fully connected network as a feedforward layer, adding the original enhanced features with the output features of the fully connected network and normalizing them to obtain the final enhanced features of the text within the text box in the diagram.

5. The diagram representation enhancement method based on self-supervised learning according to claim 1, characterized in that, Step S3 is as follows: S301. Randomly select one of the schematic representation enhancement methods obtained in step S1 as the comparison object of the original image, and denote the schematic representation enhancement network as... The reinforcement network is characterized according to the schematic diagram. Obtain the set of enhanced features from the original image and the augmented schematic; S302. Using a fully connected network with one hidden layer as a projection head, the enhanced feature set obtained in step S301 is projected from high dimension to low dimension vector space to obtain the projection vector of high dimension feature vector. S303. Record the feature vectors corresponding to the same text box in the original image and the augmented image as positive sample pairs, and the feature vectors corresponding to different text boxes as negative sample pairs. Calculate the positive sample pairs using cosine similarity. , The similarity between positive sample pairs , After swapping positions, the loss is calculated again, and the average loss of all positive sample pairs is taken as the final loss. The final loss is then minimized. The backpropagation update optimization diagram characterizes the augmented network. and projection network Optimize the schematic representation enhancement network As an image feature extraction module, it realizes schematic representation enhancement based on self-supervised learning.

6. The diagram representation enhancement method based on self-supervised learning according to claim 5, characterized in that, In step S303, minimize the final loss. for: in, This represents the number of text boxes in the diagram. For entropy noise comparison estimation, For variables.

7. The diagram representation enhancement method based on self-supervised learning according to claim 5, characterized in that, Positive sample pairs , similarity Specifically: in, Let the magnitude of the vector be . This is the transpose of a vector.

8. A schematic representation enhancement system based on self-supervised learning, characterized in that, include: The feature module performs image augmentation on the schematic diagram to obtain the corresponding augmented schematic diagram. It uses a recurrent neural network to extract the text features of multiple text boxes in the schematic diagram and a convolutional neural network to extract the image features of the schematic diagram to obtain the schematic diagram features. The enhancement module uses self-attention units to enhance the text features obtained from the feature module, resulting in enhanced text features. These enhanced text features are then used as guidance to further enhance the schematic representation features obtained from the feature module based on the attention these enhanced text features receive on the schematic representation features, resulting in a schematic representation enhancement network. Specifically: The obtained text features are enhanced using self-attention units to obtain text-enhanced features. For the obtained schematic features, the corresponding text-enhanced features are used as guidance to enhance the schematic features, representing the object features referred to by the text in the schematic. A schematic representation enhancement network is constructed by combining convolutional neural networks, recurrent neural networks, and the text feature enhancement mechanism of self-attention units. Specifically: The schematic diagram features are expanded and concatenated to obtain the schematic diagram extended features. After multi-head attention, addition and normalization are used. The input is the text enhancement features and the schematic diagram extended features. The schematic diagram features are enhanced by the attention of the text enhancement features on the schematic diagram features. The output is the schematic diagram enhanced features. The schematic diagram representation enhancement network is constructed by combining the text feature enhancement mechanism of the self-attention unit with the convolutional neural network and the recurrent neural network. The representation module uses the schematic representation enhancement network obtained from the enhancement module to extract features from the schematic diagram of the feature module and the corresponding augmented schematic diagram, resulting in an enhanced feature set. The enhanced feature set is projected from high-dimensional features to low-dimensional feature vectors, and a loss function is calculated based on the low-dimensional feature vectors. The parameters of the schematic representation enhancement network are optimized by minimizing the loss value of the loss function, thereby realizing schematic representation enhancement based on self-supervised learning.

Images