General reading sequence construction method based on strong visual supervision
Through a general reading order construction method based on strong visual supervision, combining text and visual information to generate reading order, the problem of insufficient understanding of existing methods in a multimodal environment is solved, and more accurate and flexible reading comprehension capabilities are achieved, and it is suitable for a variety of document types.
Patent Information
- Application Number
- CN202510388449.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-31
- Publication Date
- 2025-08-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing reading comprehension methods ignore the important role of visual information, resulting in insufficient understanding ability in a multimodal environment and the inability to effectively process multiple types of visual information.
A general reading order construction method based on strong visual supervision is adopted. By collecting large-scale data sets containing text and visual information, preprocessing, feature vectors are extracted using pre-trained language models and convolutional neural networks, text and visual features are fused, reading orders are generated using recurrent neural networks and long and short-term memory networks, and model performance is optimized through standard evaluation indicators.
It improves the accuracy of reading comprehension and the generalization ability of the model, allowing it to better understand the document structure in a multimodal environment, adapt to different types of documents, has strong adaptability and flexibility, and can handle academic papers, news reports and user-generated content.
Smart Images
Figure CN120496080A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a universal reading sequence construction method based on strong visual supervision. Background Art
[0002] Reading comprehension is a key task in natural language processing, aiming to enable computers to understand the meaning of text and answer relevant questions. Traditional reading comprehension methods primarily rely on textual information, analyzing the grammatical structure and semantic relationships of sentences to build understanding of the text. However, these methods overlook the important role of visual information in reading comprehension. In reality, humans utilize both textual and visual information, such as images and charts, to aid comprehension and reasoning during reading.
[0003] With the development of deep learning technology, the integration of computer vision and natural language processing has become a research hotspot. Some existing work has attempted to incorporate visual information into reading comprehension tasks, but most have limitations, such as relying on specific datasets and being unable to effectively process diverse types of visual information. Therefore, we propose a general reading sequence construction method based on strong visual supervision to address this issue. Summary of the Invention
[0004] The purpose of the present invention is to provide a general reading sequence construction method based on strong visual supervision to solve the problems raised in the above background technology.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A general reading sequence construction method based on strong visual supervision includes the following steps:
[0007] S1. Data collection: Collect a large-scale dataset containing text and corresponding visual information;
[0008] S2, data preprocessing: preprocess the collected text and visual information;
[0009] S3. Feature extraction: For text, a pre-trained language model is used to extract the text’s feature vector to capture the text’s semantic information. For visual information, a convolutional neural network is used to extract the visual feature vector to capture the key information of the image or chart.
[0010] S4, feature fusion: fuse the text feature vector and the visual feature vector to obtain a fused feature vector;
[0011] S5, reading sequence construction: the fused feature vector is input into the recurrent neural network and long short-term memory network to generate the reading sequence;
[0012] S6. Application Evaluation: Apply the trained model to actual reading comprehension tasks, receive input text and visual information, output a reading sequence and the corresponding answer to the question, and use standard reading comprehension evaluation metrics to evaluate and compare the model's performance.
[0013] Preferably, in said S2, the specific steps are as follows:
[0014] S201, Text Cleaning: Segment the text into sentences or paragraphs, remove words and punctuation that contribute little to the core semantics, scan the text using a pre-built Chinese dictionary, and extract substrings that match words in the dictionary as vocabulary;
[0015] S202, image cropping: using an image processing algorithm to automatically detect the main object or area of interest in the image, then determining a cropping area based on the detection result, and automatically cropping the image;
[0016] S203. Image scaling: Determine a fixed scaling size based on the size requirement of the input to the convolutional neural network, and scale the image proportionally based on the original aspect ratio of the image and the size of the target device or display area;
[0017] S204, pixel value normalization: unify the pixel value range of the image.
[0018] Preferably, in S203, when scaling the image, an interpolation algorithm is used to calculate the pixel values at the new size.
[0019] Preferably, in S3, the specific steps are:
[0020] S301. Select a pre-trained language model: Select an appropriate pre-trained language model according to task requirements;
[0021] S302. Feature input: The preprocessed text is input into a pretrained language model. The model outputs a vector representation of each word or token. The vectors are averaged, maximized, or reduced in dimension using a pooling layer. The preprocessed image is input into a pretrained convolutional neural network. The network automatically extracts image features through multiple layers of convolution and pooling operations.
[0022] S303. Output feature vector: Obtain a fixed-length feature vector from the pre-trained language model, obtain the output from the convolutional neural network, and flatten the tensor into a one-dimensional vector so that it can be merged with other feature vectors or used for subsequent tasks.
[0023] Preferably, in S4, the specific steps are:
[0024] S401. Initialization of feature vectors: import the text feature vector and image information as vectors, and introduce an attention mechanism to dynamically adjust the weights of the text and visual feature vectors;
[0025] S402, weighted summation: Use attention weights to perform weighted summation on the text and visual feature vectors to obtain the fused feature vector
[0026] S403, feature output: output the fused feature vector and use it as input for subsequent tasks.
[0027] Preferably, the specific steps in S5 are:
[0028] S501. Input the fused feature vector to ensure that it has the same scale and distribution as the training data and that the dimensions match;
[0029] S502, select a loss function and optimization algorithm, and train the model through forward propagation, loss calculation, backpropagation, and parameter update;
[0030] S503: Use the trained model, input the new fusion feature vector, and generate the corresponding reading sequence.
[0031] Preferably, in S501, after inputting the fused feature vector, confirm that the trained model file is complete and not damaged, and verify its integrity by checking the size, hash value, etc. of the model file; in S503, after producing the reading sequence, the rationality of the generated reading sequence is verified to check whether the generated result meets the expected range, data type and other requirements; according to the specific application scenario, the generated reading sequence is interpreted and applied; if it is used for text sorting tasks, it may be necessary to convert the generated reading sequence into the actual text order; if it is a task in other fields, the results also need to be further processed and interpreted according to the actual situation.
[0032] Preferably, the specific steps in S6 are as follows:
[0033] S601. Collect a test dataset: Select an appropriate, representative dataset for actual reading comprehension tasks that is similar to the training dataset. The dataset should contain text information, corresponding visual information, the correct reading order, and questions and corresponding standard answers. Ensure that the dataset is large enough to obtain reliable evaluation results. At the same time, perform preprocessing on the dataset, such as text cleaning and visual feature extraction, to ensure that it meets the model input requirements.
[0034] S602. Determine evaluation metrics: Define standard reading comprehension evaluation metrics to be used to evaluate model performance. Evaluation metrics include but are not limited to accuracy, precision, and recall.
[0035] S603. Text and visual information input: Input the text and visual information in the test dataset into the trained model according to the input format requirements during model training. If the model performs specific encoding on the text during training, ensure that the input text meets these requirements. For visual information, ensure that its format and size are consistent with those during training.
[0036] S604, obtaining model output: The model output includes the reading order and the corresponding question answer. The output is processed according to the output characteristics of the model, such as converting the probability distribution into a specific selection result and post-processing the text answer;
[0037] S605: Reading sequence evaluation: Compare the generated reading sequence with the standard reading sequence, calculate the accuracy, average the evaluation indicators of all questions, and obtain the overall question answer evaluation result
[0038] S606: Comparison with other models: Compare the evaluation results of the current model with the performance of other published similar models on the same task;
[0039] S607: Result analysis and improvement suggestions: Based on the results of evaluation and comparison, analyze the problems existing in the model and make improvement suggestions based on the analysis results, such as increasing the amount of training data, adjusting the model structure, improving the processing methods of text and visual information, etc., to further improve the performance of the model.
[0040] The beneficial effects of the present invention are:
[0041] 1. In the present invention, the method for constructing a universal reading order based on strong visual supervision can more comprehensively understand document content through a pre-trained model, thereby improving the accuracy of reading comprehension. This combination not only enhances the grasp of text meaning, but also enables the model to better capture key elements in images, such as charts and symbols, thereby providing more accurate understanding in a multimodal environment. By learning the structural characteristics of different types of documents, the model can automatically recognize and process complex text structures, such as long sentences and nested sentences, further improving overall comprehension capabilities.
[0042] 2. The present invention describes a universal reading sequence construction method based on strong visual supervision. This method uses large-scale datasets for training, giving the model strong generalization capabilities and allowing it to be applied to diverse reading comprehension tasks. This method enables the model to perform well on diverse data, including academic papers, news reports, and user-generated content. The model not only learns how to extract useful features from large amounts of data but can also adapt to new domains or tasks, demonstrating strong adaptability and flexibility.
[0043] 3. The present invention describes a universal reading order construction method based on strong visual supervision. This method dynamically adjusts the weights of text and visual information according to different task requirements, improving the model's flexibility and adaptability. The attention mechanism allows the model to focus on the most important parts, whether keywords in text or salient areas in images, enabling it to complete specific tasks more accurately. This adaptive capability enables the model to maintain efficient performance when processing different types of documents, whether they are technical documents that require detailed analysis or news articles that are quickly browsed.
[0044] 4. In the present invention, the method for constructing a universal reading sequence based on strong visual supervision can improve the performance of the model by continuously optimizing the model structure and parameters, making it more efficient and accurate in practical applications. This process involves multiple aspects such as algorithm selection, network architecture design, and hyperparameter adjustment. With the continuous development of technology, the emergence of new optimization methods and technologies has significantly improved the performance of the model. For example, by adopting more advanced neural network architecture and efficient training strategies, the model can achieve higher accuracy in less time while reducing the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a flow chart of a general reading sequence construction method based on strong visual supervision proposed by the present invention. DETAILED DESCRIPTION
[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0047] Reference Figure 1 , a general reading sequence construction method based on strong visual supervision, including the following steps:
[0048] S1. Data collection: Collect a large-scale dataset containing text and corresponding visual information;
[0049] S2, data preprocessing: preprocess the collected text and visual information;
[0050] S3. Feature extraction: For text, a pre-trained language model is used to extract the text’s feature vector to capture the text’s semantic information. For visual information, a convolutional neural network is used to extract the visual feature vector to capture the key information of the image or chart.
[0051] S4, feature fusion: fuse the text feature vector and the visual feature vector to obtain a fused feature vector;
[0052] S5, reading sequence construction: the fused feature vector is input into the recurrent neural network and long short-term memory network to generate the reading sequence;
[0053] S6. Application Evaluation: Apply the trained model to actual reading comprehension tasks, receive input text and visual information, output a reading sequence and the corresponding answer to the question, and use standard reading comprehension evaluation metrics to evaluate and compare the model's performance.
[0054] In this embodiment, in S2, the specific steps are as follows:
[0055] S201, Text Cleaning: Segment the text into sentences or paragraphs, remove words and punctuation that contribute little to the core semantics, scan the text using a pre-built Chinese dictionary, and extract substrings that match words in the dictionary as vocabulary;
[0056] S202, image cropping: using an image processing algorithm to automatically detect the main object or area of interest in the image, then determining a cropping area based on the detection result, and automatically cropping the image;
[0057] S203. Image scaling: Determine a fixed scaling size based on the size requirement of the input to the convolutional neural network, and scale the image proportionally based on the original aspect ratio of the image and the size of the target device or display area;
[0058] S204, pixel value normalization: unify the pixel value range of the image.
[0059] In this embodiment, in S203, when scaling the image, an interpolation algorithm is used to calculate the pixel values at the new size.
[0060] In this embodiment, in S3, the specific steps are:
[0061] S301. Select a pre-trained language model: Select an appropriate pre-trained language model according to task requirements;
[0062] S302. Feature input: The preprocessed text is input into a pretrained language model. The model outputs a vector representation of each word or token. The vectors are averaged, maximized, or reduced in dimension using a pooling layer. The preprocessed image is input into a pretrained convolutional neural network. The network automatically extracts image features through multiple layers of convolution and pooling operations.
[0063] S303. Output feature vector: Obtain a fixed-length feature vector from the pre-trained language model, obtain the output from the convolutional neural network, and flatten the tensor into a one-dimensional vector so that it can be merged with other feature vectors or used for subsequent tasks.
[0064] In this embodiment, in S4, the specific steps are:
[0065] S401. Initialization of feature vectors: import the text feature vector and image information as vectors, and introduce an attention mechanism to dynamically adjust the weights of the text and visual feature vectors;
[0066] S402, weighted summation: Use attention weights to perform weighted summation on the text and visual feature vectors to obtain the fused feature vector
[0067] S403, feature output: output the fused feature vector and use it as input for subsequent tasks.
[0068] In this embodiment, the specific steps in S5 are:
[0069] S501. Input the fused feature vector to ensure that it has the same scale and distribution as the training data and that the dimensions match;
[0070] S502, select a loss function and optimization algorithm, and train the model through forward propagation, loss calculation, backpropagation, and parameter update;
[0071] S503: Use the trained model, input the new fusion feature vector, and generate the corresponding reading sequence.
[0072] In this embodiment, in S501, after inputting the fused feature vector, it is confirmed that the trained model file is complete and not damaged, and its integrity is verified by checking the size, hash value, etc. of the model file. In S503, after producing the reading sequence, the rationality of the generated reading sequence is verified to check whether the generated result meets the expected range, data type and other requirements. According to the specific application scenario, the generated reading sequence is interpreted and applied. If it is used for text sorting tasks, it may be necessary to convert the generated reading sequence into the actual text order; if it is a task in other fields, the results also need to be further processed and interpreted according to the actual situation.
[0073] In this embodiment, the specific steps in S6 are as follows:
[0074] S601. Collect a test dataset: Select an appropriate, representative dataset for actual reading comprehension tasks that is similar to the training dataset. The dataset should contain text information, corresponding visual information, the correct reading order, and questions and corresponding standard answers. Ensure that the dataset is large enough to obtain reliable evaluation results. At the same time, perform preprocessing on the dataset, such as text cleaning and visual feature extraction, to ensure that it meets the model input requirements.
[0075] S602. Determine evaluation metrics: Define standard reading comprehension evaluation metrics to be used to evaluate model performance. Evaluation metrics include but are not limited to accuracy, precision, and recall.
[0076] S603. Text and visual information input: Input the text and visual information in the test dataset into the trained model according to the input format requirements during model training. If the model performs specific encoding on the text during training, ensure that the input text meets these requirements. For visual information, ensure that its format and size are consistent with those during training.
[0077] S604, obtaining model output: The model output includes the reading order and the corresponding question answer. The output is processed according to the output characteristics of the model, such as converting the probability distribution into a specific selection result and post-processing the text answer;
[0078] S605: Reading sequence evaluation: Compare the generated reading sequence with the standard reading sequence, calculate the accuracy, average the evaluation indicators of all questions, and obtain the overall question answer evaluation result
[0079] S606: Comparison with other models: Compare the evaluation results of the current model with the performance of other published similar models on the same task;
[0080] S607: Result analysis and improvement suggestions: Based on the results of evaluation and comparison, analyze the problems existing in the model and make improvement suggestions based on the analysis results, such as increasing the amount of training data, adjusting the model structure, improving the processing methods of text and visual information, etc., to further improve the performance of the model.
[0081] In this embodiment, when used, a large-scale data set containing text and corresponding visual information is first collected, the text is segmented into sentences or paragraphs, and words and punctuation marks that contribute little to the expression of core semantics are removed. A pre-built Chinese dictionary is used to scan the text, and substrings that can match words in the dictionary are extracted as vocabulary. An image processing algorithm is used to automatically detect the main objects or areas of interest in the image, and then a cropping area is determined based on the detection results, and the image is automatically cropped. A fixed scaling size is determined based on the size requirements of the input to the convolutional neural network, and the image is scaled proportionally according to the original aspect ratio of the image and the size of the target device or display area. The pixel value range of the image is unified. A pre-trained language model is selected to input the pre-processed text into the pre-trained language model. The model outputs a vector representation of each word or tag, takes the average, maximum value or uses a pooling layer to reduce the dimension of the vector, and inputs the pre-processed image into a pre-trained convolutional neural network. The network automatically extracts the features of the image through multi-layer convolution and pooling operations, imports the text feature vector and image information as a vector, and introduces an attention mechanism to dynamically adjust the weights of the text and visual feature vectors. The attention weights are used to perform weighted summation on the text and visual feature vectors to obtain the fused feature vector, which is output as the input for subsequent tasks.
[0082] Ensure that it has the same scale and distribution as the training data and matches the dimensions. Train the model through forward propagation, loss calculation, backpropagation and parameter update. Use the trained model to pass in the new fused feature vector to generate the corresponding reading sequence. Select an appropriate, representative actual reading comprehension task dataset that is similar to the training dataset. The dataset should contain text information, corresponding visual information, the correct reading sequence, as well as questions and corresponding standard answers. Clearly define the standard reading comprehension evaluation indicators used to evaluate model performance. Evaluation indicators include but are not limited to accuracy, precision, and recall. Input the text information and visual information in the test dataset into the trained model according to the input format requirements during model training. The model output includes a reading sequence and the corresponding question answer. Process the output according to the output characteristics of the model, compare the generated reading sequence with the standard reading sequence, calculate the accuracy, average the evaluation indicators of all questions, and obtain the overall question answer evaluation result. Compare the evaluation results of the current model with the performance of other published similar models on the same task. Based on the results of the evaluation and comparison, analyze the problems existing in the model and make improvement suggestions based on the analysis results.
[0083] The above is a detailed introduction to a general reading sequence construction method based on strong visual supervision provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above examples is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A general reading sequence construction method based on strong visual supervision, characterized in that: The steps include: S1. Data collection: Collect a large-scale dataset containing text and corresponding visual information; S2, data preprocessing: preprocess the collected text and visual information; S3. Feature extraction: For text, a pre-trained language model is used to extract the text’s feature vector to capture the text’s semantic information. For visual information, a convolutional neural network is used to extract the visual feature vector to capture the key information of the image or chart. S4, feature fusion: fuse the text feature vector and the visual feature vector to obtain a fused feature vector; S5, reading sequence construction: the fused feature vector is input into the recurrent neural network and long short-term memory network to generate the reading sequence; S6. Application Evaluation: Apply the trained model to actual reading comprehension tasks, receive input text and visual information, output a reading sequence and the corresponding answer to the question, and use standard reading comprehension evaluation metrics to evaluate and compare the model's performance.
2. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: In said S2, the specific steps are as follows: S201, Text Cleaning: Segment the text into sentences or paragraphs, remove words and punctuation that contribute little to the core semantics, scan the text using a pre-built Chinese dictionary, and extract substrings that match words in the dictionary as vocabulary; S202, image cropping: using an image processing algorithm to automatically detect the main object or area of interest in the image, then determining a cropping area based on the detection result, and automatically cropping the image; S203. Image scaling: Determine a fixed scaling size based on the size requirement of the input to the convolutional neural network, and scale the image proportionally based on the original aspect ratio of the image and the size of the target device or display area; S204, pixel value normalization: unify the pixel value range of the image.
3. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: In S203, when scaling the image, an interpolation algorithm is used to calculate the pixel values at the new size.
4. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: In S3, the specific steps are: S301. Select a pre-trained language model: Select an appropriate pre-trained language model according to task requirements; S302. Feature input: The preprocessed text is input into a pretrained language model. The model outputs a vector representation of each word or token. The vectors are averaged, maximized, or reduced in dimension using a pooling layer. The preprocessed image is input into a pretrained convolutional neural network. The network automatically extracts image features through multiple layers of convolution and pooling operations. S303. Output feature vector: Obtain a fixed-length feature vector from the pre-trained language model, obtain the output from the convolutional neural network, and flatten the tensor into a one-dimensional vector so that it can be merged with other feature vectors or used for subsequent tasks.
5. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: In S4, the specific steps are: S401. Initialization of feature vectors: import the text feature vector and image information as vectors, and introduce an attention mechanism to dynamically adjust the weights of the text and visual feature vectors; S402, weighted summation: Use attention weights to perform weighted summation on the text and visual feature vectors to obtain the fused feature vector S403, feature output: output the fused feature vector and use it as input for subsequent tasks.
6. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: The specific steps in S5 are: S501. Input the fused feature vector to ensure that it has the same scale and distribution as the training data and that the dimensions match; S502, select a loss function and optimization algorithm, and train the model through forward propagation, loss calculation, backpropagation, and parameter update; S503: Use the trained model, input the new fusion feature vector, and generate the corresponding reading sequence.
7. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: In S501, after inputting the fused feature vector, the trained model file is confirmed to be complete and undamaged, and its integrity is verified by checking the size, hash value, etc. of the model file. In S503, after generating the reading sequence, the rationality of the generated reading sequence is verified to check whether the generated result meets the expected range, data type and other requirements. According to the specific application scenario, the generated reading sequence is interpreted and applied. If it is used for text sorting tasks, it may be necessary to convert the generated reading sequence into the actual text order; if it is a task in other fields, the results also need to be further processed and interpreted according to the actual situation.
8. The method for constructing a universal reading sequence based on strong visual supervision according to claim 1, characterized in that: The specific steps in S6 are as follows: S601. Collect a test dataset: Select an appropriate, representative dataset for actual reading comprehension tasks that is similar to the training dataset. The dataset should contain text information, corresponding visual information, the correct reading order, and questions and corresponding standard answers. Ensure that the dataset is large enough to obtain reliable evaluation results. At the same time, perform preprocessing on the dataset, such as text cleaning and visual feature extraction, to ensure that it meets the model input requirements. S602. Determine evaluation metrics: Define standard reading comprehension evaluation metrics to be used to evaluate model performance. Evaluation metrics include but are not limited to accuracy, precision, and recall. S603. Text and visual information input: Input the text and visual information in the test dataset into the trained model according to the input format requirements during model training. If the model performs specific encoding on the text during training, ensure that the input text meets these requirements. For visual information, ensure that its format and size are consistent with those during training. S604, obtaining model output: The model output includes the reading order and the corresponding question answer. The output is processed according to the output characteristics of the model, such as converting the probability distribution into a specific selection result and post-processing the text answer; S605: Reading sequence evaluation: Compare the generated reading sequence with the standard reading sequence, calculate the accuracy, average the evaluation indicators of all questions, and obtain the overall question answer evaluation result S606: Comparison with other models: Compare the evaluation results of the current model with the performance of other published similar models on the same task; S607: Result analysis and improvement suggestions: Based on the results of evaluation and comparison, analyze the problems existing in the model and make improvement suggestions based on the analysis results, such as increasing the amount of training data, adjusting the model structure, improving the processing methods of text and visual information, etc., to further improve the performance of the model.