Text review method and system based on visual language large model
Through the method based on visual language big model, the error transmission problem of text recognition algorithm in text detection and correction under the traditional CNN framework is solved, which improves the accuracy and generalization ability of text review, and realizes text recognition in multiple scenarios Effect.
Patent Information
- Application Number
- CN202410940168.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-22
AI Technical Summary
The text recognition algorithm under the traditional CNN framework has error transmission in text detection, text box correction and text recognition links, resulting in insufficient accuracy and generalization capabilities of text review, and is unable to effectively cover text recognition in multiple scenarios.
Using a visual language model based on the visual language model, create a visual language model for text recognition through the VARY-TINY module and the VARY-BASE module, train and verify, deploy it on the server for text review, and use a large number of historical text files for pre-pre- Processing and dataset building, setting hyperparameters and loss optimizers, trained using AdamW optimizer and cosine annealing scheduler.
Improve the accuracy and generalization ability of text review, especially in scenes with high noise, complex format or handwritten text, avoid error transmission in text detection and text box correction, and can more accurately Extract text information from complex visual environments.
Smart Images

Figure CN119007229A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of text recognition technology, and in particular to a text review method and system based on a visual language large model. Background Art
[0002] In the process of using business documents such as contracts and authorization letters, they need to be managed, that is, the key information in them needs to be reviewed. However, the traditional purely manual review method has the disadvantages of high repetitiveness, boring content, long time consumption, easy omissions, etc. A slight negligence will cause serious losses. In the context of the rapid development of the Internet and digitalization, the review of texts requires more efficient technology to meet the needs of modern business.
[0003] With the rapid development of artificial intelligence technology, OCR technology (optical character recognition technology) has made significant progress. Applying OCR technology to text review scenarios has become a more efficient solution. OCR technology uses optical and computer technology to read text printed or written on paper and convert it into a format that can be accepted by computers and understood by humans.
[0004] The text recognition process under the traditional CNN (convolutional neural network) framework adopts a two-stage OCR algorithm, that is, a detection model + recognition model composition. In addition to input and output, the OCR core framework includes a text detection module, a detection frame correction module, and a text recognition module. The text detection module is used to detect the text area in the image and mark it with a text frame; the detection frame correction module is used to correct the text frame represented by four points into a rectangular frame to facilitate subsequent text recognition, and to judge and correct the text direction, such as turning the text line straight when it is inverted; the text recognition module is used to perform text recognition on the corrected text frame to obtain the text content in each text frame.
[0005] From the above links, it can be seen that the OCR recognition algorithm based on traditional CNN has error transmission in text detection, text box correction and text recognition, that is, the inaccuracy of text detection will reduce the accuracy of subsequent links; in addition, due to the relatively fixed design of the traditional CNN network structure, the model expression ability under this framework has certain limitations, which is directly manifested in the inability to cover text recognition in multiple scenarios and maintain a good recognition accuracy rate. Due to the poor accuracy and generalization ability of traditional methods in text recognition, the accuracy and generalization ability of text review are poor, that is, the review of key information based on incorrect text recognition content will inevitably lead to unsatisfactory review results.
[0006] Therefore, how to provide a text review method and system based on a large visual language model to improve the accuracy and generalization ability of text review has become a technical problem that needs to be solved urgently. Summary of the invention
[0007] The technical problem to be solved by the present invention is to provide a text review method and system based on a large visual language model, so as to improve the accuracy and generalization ability of text review.
[0008] In a first aspect, the present invention provides a text review method based on a visual language large model, comprising the following steps:
[0009] Step S1, obtaining a large number of historical text files, constructing a data set after preprocessing each of the historical text files, and dividing the data set into a training set and a validation set;
[0010] Step S2, creating a visual language model for text recognition based on the VARY-T I NY module and the VARY-BASE module, and setting the hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model;
[0011] Step S3, training the visual language model using the training set, and verifying the trained visual language model using the verification set;
[0012] Step S4, deploying the verified visual language large model on a server;
[0013] Step S5: The server obtains the text file to be reviewed and the review information, inputs the text file to be reviewed into the visual language model to obtain text recognition content, matches the text recognition content with the review information, and generates a text review result.
[0014] Furthermore, the step S1 is specifically as follows:
[0015] Obtain a large number of historical text files in word, PDF or image formats;
[0016] The chart annotations in the historical text file in word format are converted into Python dictionaries through Matplotlib and Pyecharts; the text information of the historical text file in PDF format is extracted through the fitz library of the PYMUPDF software package, and the historical text file is converted into a PNG image; the text information of the historical text file in image format is extracted through PADDLEOCR, and the historical text file is converted into a PNG image; each of the historical text files is annotated based on the Python dictionary or text information to generate corresponding tags to complete the preprocessing of each of the historical text files;
[0017] A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio.
[0018] Further, in step S2, the VARY-T I NY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer;
[0019] The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048;
[0020] The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training;
[0021] The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-T INY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1. The learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5;
[0022] The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
[0023] Furthermore, the step S3 is specifically as follows:
[0024] The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met;
[0025] The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
[0026] Furthermore, the step S4 is specifically as follows:
[0027] After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism of the API interface is set;
[0028] The step S5 is specifically as follows:
[0029] The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language big model through the API interface, inputs the text file to be reviewed into the visual language big model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into JSON format for feedback.
[0030] In a second aspect, the present invention provides a text review system based on a visual language model, comprising the following modules:
[0031] A data set construction module is used to obtain a large number of historical text files, construct a data set after preprocessing each of the historical text files, and divide the data set into a training set and a validation set;
[0032] A visual language model creation module, used to create a visual language model for text recognition based on the VARY-T I NY module and the VARY-BASE module, and set the hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model;
[0033] A visual language large model training module, used to train the visual language large model through the training set, and verify the trained visual language large model through the verification set;
[0034] A visual language large model deployment module, used to deploy the verified visual language large model on a server;
[0035] The text review module is used for the server to obtain the text file to be reviewed and the review information, input the text file to be reviewed into the visual language model to obtain the text recognition content, match the text recognition content with the review information, and generate the text review result.
[0036] Furthermore, the data set construction module is specifically used for:
[0037] Obtain a large number of historical text files in word, PDF or image formats;
[0038] The chart annotations in the historical text file in word format are converted into Python dictionaries through Matplotlib and Pyecharts; the text information of the historical text file in PDF format is extracted through the fitz library of the PYMUPDF software package, and the historical text file is converted into a PNG image; the text information of the historical text file in image format is extracted through PADDLEOCR, and the historical text file is converted into a PNG image; each of the historical text files is annotated based on the Python dictionary or text information to generate corresponding tags to complete the preprocessing of each of the historical text files;
[0039] A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio.
[0040] Furthermore, in the visual language large model creation module, the VARY-T I NY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer;
[0041] The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048;
[0042] The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training;
[0043] The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-T INY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1. The learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5;
[0044] The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
[0045] Furthermore, the visual language large model training module is specifically used for:
[0046] The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met;
[0047] The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
[0048] Furthermore, the visual language large model deployment module is specifically used for:
[0049] After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism for the API interface is set;
[0050] The text review module is specifically used for:
[0051] The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language big model through the API interface, inputs the text file to be reviewed into the visual language big model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into JSON format for feedback.
[0052] The advantages of the present invention are:
[0053] By obtaining a large number of historical text files, each historical text file is preprocessed to build a data set, and the data set is divided into a training set and a verification set; then, a visual language model for text recognition is created based on the VARY-TINY module and the VARY-BASE module, and the hyperparameters, loss optimizer, and cosine annealing scheduler of the visual language model are set; then, the visual language model is trained with the training set, and the trained visual language model is verified with the verification set, and the verified visual language model is deployed on the server; finally, the server obtains the text file to be reviewed and the review information, inputs the text file to be reviewed into the visual language model to obtain the text recognition content, matches the text recognition content with the review information, and generates the text review result; that is, based on VARY-T The large visual language model created by the INY module and the VARY-BASE module performs text recognition and has good fine-grained text encoding capabilities. It can effectively learn the context of the entire sentence or paragraph in the text file, which helps to improve recognition accuracy, especially in scenarios with high noise, complex formats or handwritten text. Compared with the traditional CNN OCR recognition algorithm, it avoids error transmission in text detection, text box correction and text recognition. The large visual language model is trained through a large number of historical text files, which makes the large visual language model more robust when facing challenges such as blur, tilt, and occlusion. It can more accurately extract text information from complex visual environments, ultimately greatly improving the accuracy and generalization ability of text review. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.
[0055] Figure 1 It is a flow chart of a text review method based on a visual language large model of the present invention.
[0056] Figure 2 It is a structural schematic diagram of a text review system based on a visual language large model of the present invention. DETAILED DESCRIPTION
[0057] The technical solution in the embodiments of the present application has the following overall idea: text recognition is performed based on the large visual language model created by the VARY-TINY module and the VARY-BASE module, which has good fine-grained text encoding capabilities and can effectively learn the context of the entire sentence or paragraph in the text file, which helps to improve recognition accuracy, especially in scenarios with high noise, complex formats or handwritten texts. Compared with the traditional CNN OCR recognition algorithm, it avoids error transmission in text detection, text box correction and text recognition links, and trains the large visual language model through a large number of historical text files, so that the large visual language model shows stronger robustness when facing challenges such as blur, tilt, occlusion, etc., and can more accurately extract text information from complex visual environments to improve the accuracy and generalization ability of text review.
[0058] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a text review method based on a visual language large model of the present invention comprises the following steps:
[0059] Step S1, obtaining a large number of historical text files, constructing a data set after preprocessing each of the historical text files, and dividing the data set into a training set and a validation set;
[0060] Step S2, creating a visual language model for text recognition based on the VARY-T I NY module and the VARY-BASE module, and setting the hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model;
[0061] Step S3, training the visual language model using the training set, and verifying the trained visual language model using the verification set;
[0062] Step S4, deploying the verified visual language large model on a server;
[0063] Step S5: The server obtains the text file to be reviewed and the review information, inputs the text file to be reviewed into the visual language model to obtain text recognition content, matches the text recognition content with the review information, and generates a text review result.
[0064] The step S1 is specifically as follows:
[0065] Obtain a large number of historical text files in word, PDF or image formats; the historical text files include blurred, tilted or blocked samples, and the content covers a variety of types and topics, such as business licenses, ID cards, authorization letters, customs declarations, etc.; that is, fully consider the compatibility and flexibility of different media to ensure automatic, accurate and efficient input of text review, and provide a solid foundation for the smooth progress of the entire review process;
[0066] The chart annotations in the historical text file in word format are converted into Python dictionaries through Matplotlib and Pyecharts; the text information of the historical text file in PDF format is extracted through the fitz library of the PYMUPDF software package, and the historical text file is converted into a PNG image; the text information of the historical text file in image format is extracted through PADDLEOCR, and the historical text file is converted into a PNG image; each of the historical text files is annotated based on the Python dictionary or text information to generate corresponding tags to complete the preprocessing of each of the historical text files;
[0067] A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio. That is, autoregressive training is performed using image (historical text file)-text (label) pairs, following the popular LVLM input format, that is, image tokens (tokens corresponding to historical text files) and text tokens (tokens corresponding to labels) are packaged in the form of prefixes; specifically, two special tags "" and "" are used to indicate the input position of the image token.
[0068] In step S2, the VARY-T I NY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer;
[0069] The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048;
[0070] The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder, that is, to align with the VARY-BASE module; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training;
[0071] The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-T INY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1. The learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5;
[0072] The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
[0073] The step S3 is specifically as follows:
[0074] The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met;
[0075] The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
[0076] The step S4 is specifically as follows:
[0077] After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism of the API interface is set;
[0078] By setting the authentication mechanism of the API interface, the visual language model can be called for text review only after the authentication is passed, thus preventing the visual language model from being illegally called;
[0079] The step S5 is specifically as follows:
[0080] The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language model through the API interface, inputs the text file to be reviewed into the visual language model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into a JSON format for feedback. That is, if there is a matching text recognition content, a text review result that passes the review is generated, and if there is no matching text recognition content, a text review result that fails the review is generated.
[0081] Before providing feedback on the text review result, the text review result is encrypted into an encrypted result, specifically:
[0082] The text review result is encrypted by the AES algorithm to obtain first encrypted data, a random character string of a specified length is randomly generated, the random character string is added to a specified position of the first encrypted data to obtain second encrypted data, the second encrypted data is encrypted by the RSA algorithm to obtain third encrypted data, the third encrypted data is divided based on a preset division ratio and the order is swapped to obtain fourth encrypted data, the fourth encrypted data is encrypted by the IDEA algorithm to obtain fifth encrypted data, each character of the fifth encrypted data is mapped by a preset mapping rule to obtain sixth encrypted data, and the sixth encrypted data is encrypted by the El Gama l algorithm to obtain an encryption result.
[0083] The text review result is encrypted by the AES algorithm to obtain the first encrypted data, a random character string of a specified length is randomly generated and added to the specified position of the first encrypted data to obtain the second encrypted data, the second encrypted data is encrypted by the RSA algorithm to obtain the third encrypted data, the third encrypted data is divided based on the division ratio and the front-to-back order is swapped to obtain the fourth encrypted data, the fourth encrypted data is encrypted by the IDEA algorithm to obtain the fifth encrypted data, each character of the fifth encrypted data is mapped by the mapping rule to obtain the sixth encrypted data, and the sixth encrypted data is encrypted by the El Gama l algorithm to obtain the encrypted result. If the corresponding encryption algorithm or data transformation rule is not known, the encryption result cannot be cracked. At least 8 security measures (AES algorithm, specified length, specified position, RSA algorithm, division ratio, IDEA algorithm, mapping rule, El Gama l algorithm) are taken to prevent the text review result from being stolen and tampered with in plain text, which greatly improves the security of the text review result transmission.
[0084] A preferred embodiment of a text review system based on a visual language large model of the present invention includes the following modules:
[0085] A data set construction module is used to obtain a large number of historical text files, construct a data set after preprocessing each of the historical text files, and divide the data set into a training set and a validation set;
[0086] A visual language model creation module, used to create a visual language model for text recognition based on the VARY-T I NY module and the VARY-BASE module, and set the hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model;
[0087] A visual language large model training module, used to train the visual language large model through the training set, and verify the trained visual language large model through the verification set;
[0088] A visual language large model deployment module, used to deploy the verified visual language large model on a server;
[0089] The text review module is used for the server to obtain the text file to be reviewed and the review information, input the text file to be reviewed into the visual language model to obtain the text recognition content, match the text recognition content with the review information, and generate the text review result.
[0090] The data set construction module is specifically used for:
[0091] Obtain a large number of historical text files in word, PDF or image formats; the historical text files include blurred, tilted or blocked samples, and the content covers a variety of types and topics, such as business licenses, ID cards, authorization letters, customs declarations, etc.; that is, fully consider the compatibility and flexibility of different media to ensure automatic, accurate and efficient input of text review, and provide a solid foundation for the smooth progress of the entire review process;
[0092] The chart annotations in the historical text file in word format are converted into Python dictionaries through Matplotlib and Pyecharts; the text information of the historical text file in PDF format is extracted through the fitz library of the PYMUPDF software package, and the historical text file is converted into a PNG image; the text information of the historical text file in image format is extracted through PADDLEOCR, and the historical text file is converted into a PNG image; each of the historical text files is annotated based on the Python dictionary or text information to generate corresponding tags to complete the preprocessing of each of the historical text files;
[0093] A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio. That is, autoregressive training is performed using image (historical text file)-text (label) pairs, following the popular LVLM input format, that is, image tokens (tokens corresponding to historical text files) and text tokens (tokens corresponding to labels) are packaged in the form of prefixes; specifically, two special tags "" and "" are used to indicate the input position of the image token.
[0094] In the visual language large model creation module, the VARY-TINY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer;
[0095] The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048;
[0096] The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder, that is, to align with the VARY-BASE module; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training;
[0097] The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-T INY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1. The learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5;
[0098] The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
[0099] The visual language large model training module is specifically used for:
[0100] The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met;
[0101] The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
[0102] The visual language large model deployment module is specifically used for:
[0103] After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism of the API interface is set;
[0104] By setting the authentication mechanism of the API interface, the visual language model can be called for text review only after the authentication is passed, thus preventing the visual language model from being illegally called;
[0105] The text review module is specifically used for:
[0106] The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language model through the API interface, inputs the text file to be reviewed into the visual language model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into a JSON format for feedback. That is, if there is a matching text recognition content, a text review result that passes the review is generated, and if there is no matching text recognition content, a text review result that fails the review is generated.
[0107] Before providing feedback on the text review result, the text review result is encrypted into an encrypted result, specifically:
[0108] The text review result is encrypted by the AES algorithm to obtain first encrypted data, a random character string of a specified length is randomly generated, the random character string is added to a specified position of the first encrypted data to obtain second encrypted data, the second encrypted data is encrypted by the RSA algorithm to obtain third encrypted data, the third encrypted data is divided based on a preset division ratio and the order is swapped to obtain fourth encrypted data, the fourth encrypted data is encrypted by the IDEA algorithm to obtain fifth encrypted data, each character of the fifth encrypted data is mapped by a preset mapping rule to obtain sixth encrypted data, and the sixth encrypted data is encrypted by the El Gama l algorithm to obtain an encryption result.
[0109] The text review result is encrypted by the AES algorithm to obtain the first encrypted data, a random character string of a specified length is randomly generated and added to the specified position of the first encrypted data to obtain the second encrypted data, the second encrypted data is encrypted by the RSA algorithm to obtain the third encrypted data, the third encrypted data is divided based on the division ratio and the front-to-back order is swapped to obtain the fourth encrypted data, the fourth encrypted data is encrypted by the IDEA algorithm to obtain the fifth encrypted data, each character of the fifth encrypted data is mapped by the mapping rule to obtain the sixth encrypted data, and the sixth encrypted data is encrypted by the El Gama l algorithm to obtain the encrypted result. If the corresponding encryption algorithm or data transformation rule is not known, the encryption result cannot be cracked. At least 8 security measures (AES algorithm, specified length, specified position, RSA algorithm, division ratio, IDEA algorithm, mapping rule, El Gama l algorithm) are taken to prevent the text review result from being stolen and tampered with in plain text, which greatly improves the security of the text review result transmission.
[0110] In summary, the advantages of the present invention are:
[0111] By obtaining a large number of historical text files, each historical text file is preprocessed to build a data set, and the data set is divided into a training set and a verification set; then, a visual language model for text recognition is created based on the VARY-TINY module and the VARY-BASE module, and the hyperparameters, loss optimizer, and cosine annealing scheduler of the visual language model are set; then, the visual language model is trained with the training set, and the trained visual language model is verified with the verification set, and the verified visual language model is deployed on the server; finally, the server obtains the text file to be reviewed and the review information, inputs the text file to be reviewed into the visual language model to obtain the text recognition content, matches the text recognition content with the review information, and generates the text review result; that is, based on VARY-T The large visual language model created by the INY module and the VARY-BASE module performs text recognition and has good fine-grained text encoding capabilities. It can effectively learn the context of the entire sentence or paragraph in the text file, which helps to improve recognition accuracy, especially in scenarios with high noise, complex formats or handwritten text. Compared with the traditional CNN OCR recognition algorithm, it avoids error transmission in text detection, text box correction and text recognition. The large visual language model is trained through a large number of historical text files, which makes the large visual language model more robust when facing challenges such as blur, tilt, and occlusion. It can more accurately extract text information from complex visual environments, ultimately greatly improving the accuracy and generalization ability of text review.
[0112] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A text review method based on a large visual language model, Features: The steps include: Step S1, obtaining a large number of historical text files, constructing a data set after preprocessing each of the historical text files, and dividing the data set into a training set and a validation set; Step S2, creating a visual language model for text recognition based on the VARY-TINY module and the VARY-BASE module, and setting hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model; Step S3, training the visual language model using the training set, and verifying the trained visual language model using the verification set; Step S4, deploying the verified visual language large model on a server; Step S5: The server obtains the text file to be reviewed and the review information, inputs the text file to be reviewed into the visual language model to obtain text recognition content, matches the text recognition content with the review information, and generates a text review result.
2. A text review method based on a visual language large model as claimed in claim 1, Features: The step S1 is specifically as follows: Obtain a large number of historical text files in word, PDF or image formats; Convert the chart annotations in the historical text file in word format into Python dictionaries through Matplotlib and Pyecharts; extract the text information of the historical text file in PDF format through the fitz library of the PYMUPDF software package, and convert the historical text file into a PNG image; extract the text information of the historical text file in image format through PADDLEOCR, and convert the historical text file into a PNG image; annotate each of the historical text files based on the Python dictionary or text information, generate corresponding tags, and complete the preprocessing of each of the historical text files; A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio.
3. A text review method based on a visual language large model as claimed in claim 1, Features: In step S2, the VARY-TINY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer; The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048; The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training; The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-TINY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1, the learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5; The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
4. A text review method based on a visual language large model as claimed in claim 1, Features: The step S3 is specifically as follows: The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met; The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
5. A text review method based on a visual language large model as claimed in claim 1, Features: The step S4 is specifically as follows: After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism for the API interface is set; The step S5 is specifically as follows: The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language big model through the API interface, inputs the text file to be reviewed into the visual language big model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into JSON format for feedback.
6. A text review system based on a large visual language model, Features: Includes the following modules: A data set construction module is used to obtain a large number of historical text files, construct a data set after preprocessing each of the historical text files, and divide the data set into a training set and a validation set; A visual language model creation module, used to create a visual language model for text recognition based on the VARY-TINY module and the VARY-BASE module, and set the hyperparameters, loss optimizer and cosine annealing scheduler of the visual language model; A visual language large model training module, used to train the visual language large model through the training set, and verify the trained visual language large model through the verification set; A visual language large model deployment module, used to deploy the verified visual language large model on a server; The text review module is used for the server to obtain the text file to be reviewed and the review information, input the text file to be reviewed into the visual language model to obtain the text recognition content, match the text recognition content with the review information, and generate the text review result.
7. A text review system based on a visual language macro model as claimed in claim 6, Features: The data set construction module is specifically used for: Obtain a large number of historical text files in word, PDF or image formats; Convert the chart annotations in the historical text file in word format into Python dictionaries through Matplotlib and Pyecharts; extract the text information of the historical text file in PDF format through the fitz library of the PYMUPDF software package, and convert the historical text file into a PNG image; extract the text information of the historical text file in image format through PADDLEOCR, and convert the historical text file into a PNG image; annotate each of the historical text files based on the Python dictionary or text information, generate corresponding tags, and complete the preprocessing of each of the historical text files; A data set is constructed based on each of the historical text files and labels, and the data set is divided into a training set and a validation set based on a preset segmentation ratio.
8. A text review system based on a visual language macro model as claimed in claim 6, Features: In the visual language large model creation module, the VARY-TINY module is provided with a first image encoder, a first convolutional layer and a second convolutional layer; the VARY-BASE module is provided with a second image encoder, a third image encoder, a first input embedding layer and a second input embedding layer; The first image encoder is SAM-BASE; the first input embedding layer and the second input embedding layer are both linear layers, with an input dimension of 1024 and an output dimension of 2048; The second image encoder is used to generate a new visual vocabulary based on the data set, which is generated based on fine-tuning of the first image encoder; the third image encoder is used to encode the old visual vocabulary in the data set, and is a native image encoder of CLIP-L; the first convolutional layer and the second convolutional layer are used to align the output of the first image encoder; the first input embedding layer is used for input of the new visual vocabulary; the second input embedding layer is used for input of the old visual vocabulary; the VARY-BASE module is used to merge the old visual vocabulary and the new visual vocabulary for training; The hyperparameters include at least batch size, training cycle, learning rate and loss function; the batch size of the VARY-TINY module is 512, and the training cycle is 3; the batch size of the VARY-BASE module is 256, and the training cycle is 1, the learning rate in the pre-training stage is 5E-5, and the learning rate in the supervised fine-tuning stage is 1E-5; The loss optimizer and the cosine annealing scheduler both use AdamW to adjust the learning rate.
9. A text review system based on a visual language macromodel as claimed in claim 6, Features: The visual language large model training module is specifically used for: The visual language large model is trained using the training set, and the hyperparameters are continuously optimized during the training process until a preset convergence condition is met; The trained visual language model is verified by the verification set to determine whether the recognition accuracy is greater than a preset accuracy threshold. If so, the training is terminated; if not, the training set is expanded to continue the training.
10. A text review system based on a visual language macro model as claimed in claim 6, Features: The visual language large model deployment module is specifically used for: After adjusting the weights of the VARY-TINY module and the VARY-BASE module of the verified visual language model, the visual language model is deployed on a server, an API interface for calling the visual language model is set, and an identity authentication mechanism for the API interface is set; The text review module is specifically used for: The server obtains a text review request that carries at least a text file to be reviewed, review information, and identity information through the HTTP protocol, parses the text review request to obtain the text file to be reviewed, review information, and identity information, verifies the identity information based on the identity authentication mechanism, calls the visual language big model through the API interface, inputs the text file to be reviewed into the visual language big model to obtain text recognition content, performs regularized matching on the text recognition content through the review information, generates a text review result, and converts the text review result into JSON format for feedback.
Citation Information
Cited By
Charging pile non-inductive charging method and system
CN120481743A
Text review method and system based on language large model
CN120853200A
