OCR (Optical Character Recognition) method, device and equipment based on large model and storage medium

Abstract

The invention relates to the technical field of image recognition, in particular to an OCR (Optical Character Recognition) method, device and equipment based on a large model and a storage medium, which can directly parse structured semantic information with clear logic from an original image through the powerful image-text joint modeling capability of a visual language large model in combination with visual and text reasoning, and improve the recognition efficiency. And manual secondary processing on the intermediate result is not needed, so that the automation degree and efficiency of processing are greatly improved. Besides, the handwritten character recognition sub-model is introduced to enhance handwritten recognition and face various signature styles, the sub-model can combine visual features and text semantics to carry out comprehensive analysis, recognition accuracy is remarkably improved, the situation of recognition failure caused by the problems of irregular strokes, continuous strokes and the like is avoided, effective extraction of key handwritten information is guaranteed, and user experience is improved. The problem that end-to-end understanding from an original image to structured semantic information is difficult to realize in a traditional OCR scheme is solved, and the technical problem that the handwriting recognition effect is poor is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to an OCR recognition method and device based on a large model, equipment and a storage medium. BACKGROUND

[0002] In the contract processing scene, the application of the traditional OCR (Optical Character Recognition) system has obvious short boards. The function of this kind of system is only limited to the conversion of image information into text content, and lacks the deep analysis ability of the semantic structure of the contract, and cannot accurately capture the context logical association between clauses. Since the contract document usually contains complex layout and professional terms, the recognition accuracy of the traditional OCR system is low. At the same time, the extraction process of the key information highly depends on the rules or templates set by artificial pre-setting, which leads to the insufficient flexibility and adaptability when facing different formats and different types of contracts.

[0003] Especially in processing handwritten content, the defects of the traditional OCR technology are more prominent. For the common handwritten signature and handwritten annotation in the contract, due to the problems such as irregular stroke form, common pen phenomenon, etc., the traditional OCR system often fails to recognize, and it is difficult to meet the accurate extraction demand of this kind of key information in actual application, which brings great obstacles to the automatic processing and intelligent analysis of the contract.

[0004] It can be seen that the prior art still needs to be improved and improved. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide an OCR recognition method based on a large model, which improves the processing efficiency and accuracy of contract image recognition, solves the problem that the traditional OCR cannot realize the end-to-end understanding from the original image to the structured semantic information, and solves the technical problem of poor handwriting recognition effect.

[0006] The first aspect of the present application provides an OCR recognition method based on a large model, comprising: obtaining a target contract image and a pre-trained visual language large model, the visual language large model comprising a visual encoder, a language decoder, a format constraint module, a handwritten character recognition sub-model, and an output module; using the visual encoder to perform feature extraction on the target contract image to obtain high-order image features; using the language decoder to perform cross-modal reasoning on the high-order image features to obtain unstructured text; using the format constraint module to perform multi-round autoregressive generation on the unstructured text to output structured text; using the handwritten character recognition sub-model to obtain handwritten character region images in the target contract image and output recognized text based on the handwritten character region images; and using the output module to output a visual report based on the structured text and the recognized text.

[0007] Optionally, in the first implementation manner of the first aspect of the present application, the obtaining of the target contract image comprises: obtaining an initial contract image, and performing noise removal processing on the initial contract image to obtain a denoised image; performing preliminary correction on the denoised image based on an edge detection algorithm to obtain a first corrected image, and then performing secondary correction on the first corrected image by using a mesh deformation bridge technology to obtain a second corrected image; and performing binarization processing on the second corrected image to obtain the target contract image.

[0008] Optionally, in the second implementation manner of the first aspect of the present application, the visual encoder comprises an input layer and a plurality of first self-attention layers, and the feature extraction on the target contract image by using the visual encoder to obtain high-order image features comprises: performing feature extraction on the target contract image by using the input layer to obtain image features; introducing a low-rank matrix into the plurality of first self-attention layers to obtain a plurality of low-rank self-attention layers; and performing self-attention calculation on the image features by using the plurality of low-rank self-attention layers to obtain the high-order image features.

[0009] Optionally, in the third implementation manner of the first aspect of the present application, the feature extraction on the target contract image by using the input layer to obtain image features comprises: performing semantic pre-segmentation on the target contract image by using the input layer to distinguish text regions, table regions, seal regions and blank regions in the target contract image; calculating character density in the text regions and the table regions by using a multi-scale sliding window; performing dynamic region division on the target contract image in combination with the calculated character density to obtain a plurality of image blocks; performing pixel feature extraction on each image block, and assigning a dynamic weight to each image block after pixel feature extraction to obtain the image features.

[0010] Optionally, in a fourth implementation form of the first aspect of the present application, the step of performing cross-modal inference on the high-level image feature by using the language decoder to obtain unstructured text comprises: obtaining a preset contract field vocabulary; incorporating the contract field vocabulary into the language decoder to obtain a target language decoder; and performing cross-modal inference on the high-level image feature by using the target language decoder to obtain the unstructured text.

[0011] Optionally, in a fifth implementation form of the first aspect of the present application, the handwriting recognition sub-model comprises a convolutional layer, a long short-term memory network layer and a plurality of second self-attention layers, and the step of obtaining a handwriting region image in the target contract image by using the handwriting recognition sub-model and outputting recognized text based on the handwriting region image comprises: obtaining a handwriting region image in the target contract image; obtaining local stroke features in the handwriting region image by using the convolutional layer to obtain a stroke sequence; obtaining time sequence dependency in the stroke sequence by using the long short-term memory network layer to obtain a hidden state sequence containing the time sequence dependency; calculating attention weights of the hidden state sequence by using the plurality of second self-attention layers to obtain a global feature vector; and performing feature conversion processing and decoding processing on the global feature vector to obtain the recognized text.

[0012] Optionally, in a sixth implementation form of the first aspect of the present application, the output module comprises a preset knowledge graph, a preset risk rule library and a visualization submodule, and the step of outputting a visualization report based on the structured text and the recognized text by using the output module comprises: performing entity standardization processing and relationship verification on the structured text and the recognized text by using the knowledge graph to obtain structured information; performing risk rule matching on the structured information by using the risk rule library to obtain a risk list; and performing risk level evaluation and inference based on the risk list by using the visualization submodule to output the visualization report.

[0013] In a second aspect, the present application provides an OCR recognition device based on a large model, comprising: a data acquisition module configured to acquire a target contract image and a pre-trained visual language large model, the visual language large model comprising a visual encoder, a language decoder, a format constraint module, a handwritten character recognition sub-model, and an output module; a feature extraction module configured to perform feature extraction on the target contract image using the visual encoder to obtain high-order image features; a cross-modal reasoning module configured to perform cross-modal reasoning on the high-order image features using the language decoder to obtain unstructured text; a structured generation module configured to perform multi-round autoregressive generation on the unstructured text using the format constraint module to output structured text; a handwritten character recognition module configured to acquire handwritten character region images in the target contract image using the handwritten character recognition sub-model and output recognized text based on the handwritten character region images; and a report acquisition module configured to output a visualized report based on the structured text and the recognized text using the output module.

[0014] In a third aspect, the present application provides an OCR recognition device based on a large model, comprising a memory and at least one processor, wherein the memory stores instructions; and the at least one processor invokes the instructions in the memory to enable the OCR recognition device based on a large model to perform the steps of any of the above-mentioned OCR recognition methods based on a large model.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions are executed by a processor to implement the steps of any of the above-mentioned OCR recognition methods based on a large model.

[0016] In the technical solution of the present application, the powerful joint modeling capability of the visual language large model, combined with visual and text reasoning, realizes end-to-end understanding from the original image to the structured semantic information, and can directly parse the structured semantic information with clear logic from the original image, without the need for manual secondary processing of the intermediate results, greatly improving the automation degree and efficiency of processing. For complex document structures, whether it is multi-column layout, nested tables or mixed professional terms, the joint reasoning of visual and text can accurately understand the layout and meaning, ensuring the accuracy of structured information extraction and avoiding the poor adaptability problem caused by relying on artificial rules or templates in traditional methods. In addition, the introduction of the handwritten character recognition sub-model can enhance handwritten recognition, face various signature styles such as rough writing, artistic signatures, and various handwritten annotations including modification annotations and supplementary explanations, and the sub-model can combine visual features and text semantics for comprehensive analysis, significantly improving the accuracy of recognition and avoiding recognition failure due to irregular strokes, connected writing, and other problems, ensuring the effective extraction of key handwritten information. Attached Figure Description

[0017] Figure 1 A first flowchart of the OCR recognition method based on a large model provided in an embodiment of the present invention; Figure 2 A second flowchart of the OCR recognition method based on a large model provided in an embodiment of the present invention; Figure 3 A third flowchart of the OCR recognition method based on a large model provided in an embodiment of the present invention; Figure 4 The fourth flowchart of the OCR recognition method based on a large model provided in the embodiments of the present invention; Figure 5 The fifth flowchart of the OCR recognition method based on a large model provided in the embodiments of the present invention; Figure 6 The sixth flowchart of the OCR recognition method based on a large model provided in the embodiments of the present invention; Figure 7 The seventh flowchart of the OCR recognition method based on a large model provided in the embodiments of the present invention; Figure 8 A schematic diagram of a large-model-based OCR recognition device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of an OCR recognition device based on a large model provided in an embodiment of the present invention. Detailed Implementation

[0018] This invention provides a method, apparatus, device, and storage medium for OCR recognition based on a large model. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0019] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the OCR recognition method based on a large model in this invention includes: 101、obtaining a target contract image and a pre-trained visual language large model, the visual language large model comprising a visual encoder, a language decoder, a format constraint module, a handwritten character recognition sub-model and an output module; In the present embodiment, the pre-trained visual language large model can be initialized using the pre-training weights of Qwen2.5 LLM, and the 1D RoPE is modified to a multi-modal rotational position embedding aligned with absolute time. The Qwen2.5-VL model is used as the basic architecture, which can accurately understand information of multiple modalities such as images, videos and texts. When processing complex documents, it can upgrade text recognition to full document parsing, which can not only recognize characters, but also process tables, charts, chemical formulas, music scores and other content, and is widely used in office automation, education, scientific research and other fields, improving the efficiency of document processing and the accuracy of information extraction.

[0020] 102、using the visual encoder to extract features from the target contract image to obtain high-order image features; In the present embodiment, the visual encoder extracts features from the target contract image to obtain high-order image features, which is a process of deep analysis and refinement of various information contained in the target contract image. In this process, the visual encoder automatically identifies and captures representative key information in the contract image, such as font, font size, arrangement of characters, structure and layout of tables, shape and position of seals, and details such as texture, edge, and light and dark distribution of images, and integrates and transforms these scattered low-level information, finally forming high-order image features that can reflect the overall attributes and core content of the contract image. On the one hand, high-order image features retain key information and internal relations in the contract image, which helps to improve the accuracy and efficiency of subsequent tasks such as character recognition, table extraction, seal verification, etc. On the other hand, the refined high-order features reduce the interference of redundant information, reduce the computational complexity in the subsequent processing process, so that the model can quickly and accurately complete the understanding and analysis of the contract content, thereby significantly improving the automation level and intelligent degree of the overall processing flow in contract auditing, information archiving, intelligent retrieval and other practical application scenarios, saving labor costs and improving work efficiency.

[0021] 103、using the language decoder to perform cross-modal reasoning on the high-order image features to obtain unstructured text; In this embodiment, the language decoder is used to perform cross-modal reasoning on high-level image features to obtain unstructured text, which refers to the process of converting the abstracted visual features into natural language text content after obtaining the high-level features of the contract image through a decoder with language understanding and generation capabilities. In this process, the language decoder relies on the semantic understanding ability formed by pre-training to establish the association between visual features and text semantics, and through deep analysis and cross-modal mapping of the structural information and content logic contained in the high-level image features, it converts the information such as text content, layout relationship, and symbol meaning contained in the image into continuous and language expression habit-compliant unstructured text, effectively converting visual information into language information, so that the contract content that can only be presented through the image carrier can be extracted and utilized in the form of editable and understandable text. At the same time, cross-modal reasoning ensures the accuracy and integrity of the conversion of visual features to text, avoids information loss or distortion during the conversion process, reduces the tediousness and errors of manual transcription, and improves the automation level of contract processing while enhancing the flexibility and efficiency of information utilization.

[0022] 104. performing multi-round autoregressive generation on the unstructured text by using the format constraint module to output structured text; In this embodiment, the format constraint module is used to perform multi-round autoregressive generation on the unstructured text to output structured text, i.e., according to its preset structure framework, the text content is analyzed, reorganized and generated in multiple rounds, and finally the structured text conforming to the standard format is formed. In this process, the format constraint module will first identify the key information in the unstructured text, then classify, associate and integrate the information, and gradually optimize the structural integrity and logical consistency of the text through multi-round autoregressive generation, to ensure that the finally output text not only retains the accuracy of the original content, but also conforms to the explicit format specification, such as presenting in the form of table, JSON, etc. to make the hierarchy, attributes and association of information clear and identifiable, achieving accurate conversion from unstructured information to structured data, converting the originally scattered and disordered text content into a standardized form convenient for computer processing and efficient reading by humans, and reducing the problem of information fragmentation and inefficient processing caused by inconsistent formats.

[0023] 105. obtaining a handwritten character region image in the target contract image by using the handwritten character recognition sub-model, and outputting recognized text based on the handwritten character region image; In the embodiment, the handwriting recognition sub-model adopts a multi-modal fusion design. In addition to adopting the generative adversarial network technology, the style transfer technology is introduced in the data enhancement link. By converting printed characters into samples with different handwriting styles, various interference factors that may occur in real scenarios are simulated, such as stroke blurring caused by signature ink bleeding and stroke deformation caused by paper creases, so as to enrich the diversity and authenticity of the training data. In the actual recognition process, the handwriting recognition sub-model first locates and obtains the handwriting region image in the target contract image, and then analyzes and processes the region image to finally output the corresponding recognition text. The multi-modal fusion data enhancement method significantly improves the adaptability of the model to different handwriting styles. By simulating the interference factors in the real scene, the model is exposed to complex situations close to actual application in the training stage, thereby enhancing the robustness of the recognition and effectively reducing the recognition errors caused by handwriting style differences and writing environment interference.

[0024] 106. Output a visual report based on the structured text and the recognized text using the output module.

[0025] In the embodiment, the output module outputs a visual report based on the structured text and the recognized text. The visual report analyzes the risk point position, the basis for violation, and the like of the target contract image, and proposes modification suggestions based on the risk point position, the basis for violation, and the like. The visual report can present complex contract information and analysis results in an intuitive and clear manner, enabling relevant personnel to quickly and accurately grasp the risks in the contract. The explicit risk point position facilitates direct positioning of the problem, the listed basis for violation enhances the authority and persuasiveness of the risk judgment, and the modification suggestions provide specific guidance for contract improvement, greatly improving the efficiency of contract auditing and revision. At the same time, the integration of structured text and handwriting recognition text ensures the comprehensiveness of the report information, avoiding risk misjudgment caused by omission of handwritten content.

[0026] Please refer to Figure 2 In the embodiment, the target contract image is obtained, including: 201. An initial contract image is obtained, and noise removal processing is performed on the initial contract image to obtain a denoised image. In the embodiment, in the image collection stage, not only clear scanned documents, but also images in various complex scenes, such as photos with shadows, reflections, wrinkles, and document images with large resolution differences from low-pixel mobile phone shooting to high-resolution scanner output, are included to ensure the universality of the preprocessing scheme. Therefore, after obtaining the initial contract image, noise removal processing is needed. Noise removal processing refers to the process of identifying and eliminating the interference information (such as speckles generated during scanning, light spots formed by light reflection during shooting, paper stains or texture interference, and pixel abnormalities introduced during transmission) in the initial contract image using image processing algorithms, making the effective information in the image more clear and identifiable.

[0027] In the embodiment, the noise removal processing adopts a hierarchical processing strategy. First, the corresponding processing strategy needs to be selected according to the noise intensity of the initial contract image. The noise intensity can be divided into slight noise, dense salt and pepper noise, and stripe noise generated by scanning. For slight noise, that is, light interference in the image, a filtering algorithm based on a Gaussian function is used to calculate the weighted average of the gray value of each pixel and its surrounding neighborhood pixels, thereby weakening or eliminating these slight noises and making the image overall appear smoother. Gaussian filtering can effectively suppress slight noise while maximizing the retention of edge and detail features of key information in the image, avoiding image blurring or information loss caused by excessive smoothing. For dense salt and pepper noise, that is, a large number of random black and white noise points in the image, a median filtering algorithm that can automatically adjust the window size according to local pixel characteristics is used. For noise-intensive areas, the filtering window is increased to enhance the noise removal capability and ensure the effective filtering of concentrated noise points. In critical areas such as text edges, the window size is automatically reduced to reduce excessive processing of edge pixels. In this process, the algorithm dynamically balances the window size and noise removal needs by continuously judging the noise characteristics and signal features of the pixels in the window, thereby eliminating dense salt and pepper noise while minimizing the blurring effect on text edge details. For stripe noise generated by scanning, the image is converted from spatial domain to frequency domain through Fourier transform. Based on the characteristic that stripe noise usually exhibits as a peak value at a specific frequency in the frequency domain, the frequency components corresponding to the stripe noise are identified and filtered in the frequency domain. Then, the processed frequency domain information is converted back to the spatial domain through inverse Fourier transform to form an image with removed stripe noise. This hierarchical processing strategy can effectively suppress high-intensity and complex noise while preventing excessive processing of low-intensity noise areas that may cause loss or blurring of image details, thereby maximizing the retention of effective information in the image while removing noise. At the same time, hierarchical processing can be optimized according to the regional differences of noise distribution, improving processing efficiency while reducing unnecessary computational consumption, making the noise processing link more suitable for the complexity of actual images.

[0028] 202、based on the edge detection algorithm on the denoising image preliminary correction, get the first correction image, and then use the grid deformation bridge technology to the first correction image secondary correction, get the second correction image; In this embodiment, first, the edge detection algorithm (such as Canny operator) is used to identify the region with obvious contour features (such as the edge of the contract, the table line, the edge of the text line, etc.) in the denoised image. According to the edge information, the inclination angle and the distortion degree of the image are judged, and the basic geometric correction is carried out to make the image basically recover to the horizontal or vertical state, and the first correction image is obtained. On this basis, the grid deformation bridge technology is further used to divide the image into fine grid units. The local slight deformation (such as local distortion caused by paper crease and shooting angle deviation) is accurately captured by analyzing the offset of the grid nodes, and each grid unit is adjusted by stretching, compressing or translating. Through the perspective transformation, the area is adjusted to ensure that the text line is strictly horizontal, realize the fine secondary correction of the image, and get the second correction image. The preliminary correction quickly solves the inclination and deflection of the whole image by means of edge detection, which lays a foundation for subsequent processing. The secondary correction accurately repairs the local slight deformation through the grid deformation bridge technology, which makes up for the defects of the preliminary correction in dealing with complex deformation. The combination of the two ensures the correction efficiency and significantly improves the correction accuracy.

[0029] 203、the second correction image is binarized to obtain the target contract image.

[0030] In this embodiment, the binarization processing is based on the U-Net architecture to train the image segmentation model to accurately distinguish the text and background area in the contract image. On this basis, according to the characteristics of different regions after segmentation, such as dark text on light background or light text on dark background, the threshold algorithm is dynamically selected and adapted: for the region with uniform illumination and stable contrast, the global threshold algorithm is used to ensure the processing efficiency; for the region with local light and dark difference, the local threshold algorithm is enabled to adapt to the local light change by calculating the threshold value of the image block, and finally output the binary image with the maximum contrast between text and background. The segmentation model assisted by deep learning breaks through the dependence on manual threshold setting in traditional binarization method, greatly improves the accuracy of distinguishing text and background in complex scenes, and the dynamic threshold selection mechanism further adapts to the characteristics of different regions, avoiding the problems of text breakage and background residue caused by single threshold. The finally generated binary image not only retains the complete text contour and details, but also eliminates the interference of redundant information by maximizing the contrast.

[0031] Please refer to Figure 3In this embodiment of the invention, the visual encoder includes an input layer and multiple first self-attention layers. The step of using the visual encoder to extract features from the target contract image to obtain high-order image features includes: 301. Use the input layer to extract features from the target contract image to obtain image features; 302. Introduce a low-rank matrix into the first self-attention layer to obtain a multi-layer low-rank self-attention layer; 303. Perform self-attention calculation on the image features using multiple low-rank self-attention layers to obtain the high-order image features.

[0032] In this embodiment, the visual encoder consists of an input layer and multiple first self-attention layers. Its working process is as follows: the input layer performs initial feature extraction on the target image, transforming the image information into image features that can be processed by a computer; a low-rank matrix is ​​introduced into the multiple first self-attention layers, transforming the original self-attention layers into multiple low-rank self-attention layers, and simplifying the complexity of self-attention calculation by leveraging the characteristics of the low-rank matrix; then, these low-rank self-attention layers are used to perform self-attention operations on the image features, and through multi-layer processing, key information in the image is gradually extracted and integrated, ultimately obtaining high-order image features that can reflect the core content of the image.

[0033] In this embodiment, a low-rank matrix is ​​introduced into the multi-layer first self-attention layer. The high-dimensional attention weight matrix is ​​decomposed into the product of two low-rank matrices. Only these newly added low-rank parameters are trained, freezing the original pre-trained parameters of the model. When performing supervised fine-tuning using a large-scale labeled contract dataset, the model is adapted to specific scenarios (such as contract documents) by updating the parameters of the low-rank matrix. This significantly reduces computational resource consumption, avoids overfitting, and retains the general knowledge transfer capability of the pre-trained model. This allows the model to maintain broad adaptability while improving the recognition accuracy for contract-specific formats and terms.

[0034] Please see Figure 4 In this embodiment of the invention, the step of extracting features from the target contract image using the input layer to obtain image features includes: 401. Using the input layer, perform semantic pre-segmentation on the target contract image to distinguish text regions, table regions, seal regions and blank regions in the target contract image; 402. Calculate text density using a multi-scale sliding window within text and table areas; 403. Based on the calculated text density, the target contract image is dynamically divided into regions to obtain multiple image blocks; 404. Extract pixel features from each image block and assign dynamic weights to each image block after pixel feature extraction to obtain the image features.

[0035] In this embodiment, firstly, semantic pre-segmentation is performed on the target contract image using the input layer. The core of this step is to accurately distinguish text regions, table regions, seal regions, and blank regions in the image, laying the foundation for subsequent targeted processing. After completing region segmentation, for text and table regions, a multi-scale sliding window is used to calculate the text density, thereby determining the density of text distribution in different regions. Subsequently, based on the obtained text density data, the target contract image is dynamically divided into multiple image blocks. This segmentation method can flexibly adjust the size and range of image blocks according to the actual text distribution, avoiding information fragmentation or redundancy that may be caused by fixed segmentation. Next, pixel feature extraction is performed on each image block to capture the visual feature information contained in the image block. At the same time, dynamic weights are assigned to each image block after pixel feature extraction, and finally integrated to form image features. Semantic pre-segmentation accurately identifies different types of key regions in contract images, eliminating interference from irrelevant information such as blank areas and improving the targeting of feature extraction. Multi-scale sliding window calculation of text density and dynamic region division make the division of image blocks more closely match the actual characteristics of text distribution, ensuring the integrity and relevance of information within each image block while avoiding information waste or insufficiency caused by fixed division. Furthermore, assigning dynamic weights to image blocks allows for differentiated processing based on the importance of different image blocks in the expression of contract information, enabling the final image features to focus more on key information and effectively improving the accuracy and efficiency of subsequent tasks such as contract content recognition and analysis based on these features.

[0036] Please see Figure 5 In this embodiment of the invention, the step of using the language decoder to perform cross-modal reasoning on the high-order image features to obtain unstructured text includes: 501. Obtain the preset contract domain glossary; 502. The contract domain vocabulary is integrated into the language decoder to obtain the target language decoder; 503. Use the target language decoder to perform cross-modal reasoning on the high-order image features to obtain the unstructured text.

[0037] In this embodiment, a pre-defined contract domain vocabulary is integrated into the language decoder. This vocabulary covers contract terminology, legal clauses, and industry-specific expressions, resulting in a target language decoder. This enhances the language decoder's semantic understanding of professional terms, avoiding issues such as terminology misuse and non-standard expressions that may occur with general language models when using the target language decoder for cross-modal reasoning to generate unstructured text. This makes the generated unstructured text more closely aligned with the professional context of contracts, improving the professionalism and accuracy of the text content. Simultaneously, the integration of professional vocabulary accelerates the decoder's understanding and transformation of contract-specific information, reducing information distortion caused by semantic ambiguity. This ensures the text fully conveys the key content of the contract, providing a high-quality text foundation for subsequent structured processing and risk analysis, thereby enhancing the reliability and efficiency of the entire contract processing flow. This advantage is particularly evident in contract scenarios involving complex legal clauses or industry-specific agreements, where its ability to convey professional information is significantly enhanced.

[0038] Please see Figure 6 In this embodiment of the invention, the handwritten text recognition sub-model includes convolutional layers, long short-term memory network layers, and multiple layers of second self-attention layers. The step of using the handwritten text recognition sub-model to obtain a handwritten text region image in the target contract image and outputting recognized text based on the handwritten text region image includes: 601. Obtain an image of the handwritten text region in the target contract image; 602. Use the convolutional layer to obtain local stroke features in the handwritten text region image to obtain a stroke sequence; 603. Use the Long Short-Term Memory network layer to obtain the temporal dependencies in the stroke sequence, and obtain a hidden state sequence containing the temporal dependencies; 604. Calculate the attention weights of the hidden state sequence using multiple layers of the second self-attention layer to obtain the global feature vector; 605. Perform feature transformation and decoding processing on the global feature vector to obtain the recognized text.

[0039] In this embodiment, the handwritten text recognition sub-model adopts a combination of convolutional layers, long short-term memory network layers, and multiple layers of second self-attention layers. Since the edges of handwritten text are more irregular than those of printed text, feature differences can be extracted first through edge detection (such as the Canny operator) to obtain the handwritten text region image in the target contract image. Then, the convolutional layer is used to obtain the local stroke features in the handwritten text region image. The convolutional layer captures the local stroke shape, edge, and texture features in the handwritten text region through a sliding window, transforming the spatial visual information into an ordered stroke sequence. Then, the long short-term memory network layer is used to obtain the temporal dependencies in the stroke sequence, that is, it uses its ability to remember temporal information to mine the dynamic dependencies such as the order of strokes and the connection between strokes, and obtains the hidden state sequence containing temporal dependencies. Then, the attention weights of the hidden state sequence are calculated using multiple layers of second self-attention layers to focus on key stroke features and integrate global information to obtain a global feature vector. Finally, the global feature vector is processed by feature transformation and decoding to obtain the recognized text. The multi-stage progressive feature extraction and processing not only preserves the local details of handwritten characters but also captures the temporal relationships and global structure between strokes, effectively solving the recognition challenges posed by diverse writing styles and stroke distortions in handwritten characters. The combination of convolutional layers and long short-term memory network layers can accurately analyze the shape and writing order of handwritten strokes, while the self-attention mechanism can highlight important features and weaken interfering information, making the model more adaptable to complex handwriting styles.

[0040] In this embodiment, for the specific task of signature recognition, metric learning is introduced into the handwritten text recognition sub-model. The model is trained by the triple loss function to distinguish subtle differences in signatures, and the consistency of signatures is judged by combining time series analysis. In the end, the accuracy of handwritten content recognition and signature authenticity identification is improved.

[0041] Please see Figure 7 In this embodiment of the invention, the output module includes a preset knowledge graph, a preset risk rule base, and a visualization submodule. The step of using the output module to output a visualization report based on the structured text and recognized text includes: 701. Using the knowledge graph, perform entity standardization and relationship verification on the structured text and the identified text to obtain structured information; 702. Use the risk rule base to match the structured information with risk rules to obtain a risk list; 703. Using the visualization submodule, perform risk level assessment and reasoning based on the risk list, and output the visualization report.

[0042] In this embodiment, the output module includes a preset knowledge graph, a preset risk rule base, and a visualization submodule. The preset knowledge graph contains structured knowledge such as legal clauses, industry standards, and the relationships between contract elements. The preset risk rule base covers the judgment criteria for various contract risks, common violations, and historical risk cases.

[0043] In this embodiment, a knowledge graph is first used to perform entity standardization and relationship verification on the structured text and the identified text to obtain structured information. Entity standardization involves aligning the entities extracted from the text (such as person names and company names) with standard entities in the knowledge graph. Through similarity matching and contextual attribute disambiguation, the identifiers and formats are unified (e.g., converting "100,000" to "100,000 yuan"). Relationship verification uses rules in the knowledge graph (such as relationship legality and attribute consistency) to check whether the relationships between entities (such as "legal representative") are reasonable, correcting errors and filling in missing relationships, ultimately forming structured information. Then, a risk rule base is used to match the structured information with risk rules to identify potential risk points in the contract, resulting in a risk list. Finally, a visualization submodule is used to assess and infer risk levels based on the risk list, and the risk list and related analysis are output in the form of a visual report. Knowledge graphs and risk rule bases provide solid knowledge support and clear judgment criteria for risk identification, ensuring the professionalism and standardization of risk identification. The generated visual reports can help relevant personnel quickly grasp the potential risks in contracts, provide precise guidance for contract review and revision, effectively reduce legal disputes or economic losses caused by risk omissions, improve the efficiency and quality of contract processing, and promote the development of contract management towards intelligence and refinement.

[0044] The above describes the OCR recognition method based on a large model in the embodiments of the present invention. The following describes the OCR recognition device based on a large model in the embodiments of the present invention. Please refer to [link / reference]. Figure 8 One embodiment of the OCR recognition device based on a large model in this invention includes: Data acquisition module 801: used to acquire the target contract image and a pre-trained visual language large model, wherein the visual language large model includes a visual encoder, a language decoder, a format constraint module, a handwritten character recognition sub-model and an output module; Feature extraction module 802: used to extract features from the target contract image using the visual encoder to obtain high-order image features; Cross-modal reasoning module 803: used to perform cross-modal reasoning on the high-order image features using the language decoder to obtain unstructured text; Structured generation module 804: Used to perform multi-round autoregressive generation on the unstructured text using the format constraint module, and output structured text; Handwritten text recognition module 805: used to obtain the handwritten text region image in the target contract image using the handwritten text recognition sub-model, and output the recognized text based on the handwritten text region image; Report acquisition module 806: Used to output a visual report based on the structured text and recognized text using the output module.

[0045] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0046] above Figure 8 The OCR recognition device based on a large model in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The OCR recognition device based on a large model in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0047] Figure 9 This is a schematic diagram of the structure of a large-model-based OCR recognition device 900 provided in an embodiment of the present invention. The large-model-based OCR recognition device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the large-model-based OCR recognition device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the large-model-based OCR recognition device 900 to implement the steps of the large-model-based OCR recognition method provided in the above-described method embodiments.

[0048] The OCR recognition device 900 based on a large model may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 9 The illustrated structure of a large-model-based OCR recognition device does not constitute a limitation on large-model-based OCR recognition devices. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0049] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a large model-based OCR recognition method.

[0050] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0051] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large model-based OCR recognition method, characterized in that, The method comprises the following steps: obtaining a target contract image and a pre-trained visual language large model, the visual language large model comprising a visual encoder, a language decoder, a format constraint module, a handwritten character recognition sub-model and an output module; extracting features of the target contract image by using the visual encoder to obtain high-order image features; performing cross-modal reasoning on the high-order image features by using the language decoder to obtain unstructured text; generating the unstructured text in multiple rounds of self-recursion by using the format constraint module to output structured text; obtaining handwritten character region images in the target contract image by using the handwritten character recognition sub-model, and outputting recognized text based on the handwritten character region images; outputting a visual report based on the structured text and the recognized text by using the output module.

2. The large model-based OCR recognition method of claim 1, wherein, The method comprises the following steps: obtaining an initial contract image, and removing noise from the initial contract image to obtain a denoised image; performing preliminary correction on the denoised image based on an edge detection algorithm to obtain a first corrected image, and performing secondary correction on the first corrected image by using a mesh deformation bridge technology to obtain a second corrected image; performing binaryzation on the second corrected image to obtain the target contract image.

3. The large model-based OCR recognition method of claim 1, wherein, The visual encoder comprises an input layer and a plurality of first self-attention layers, and the method comprises the following steps: extracting features of the target contract image by using the input layer to obtain image features; introducing a low-rank matrix into the plurality of first self-attention layers to obtain a plurality of low-rank self-attention layers; performing self-attention calculation on the image features by using the plurality of low-rank self-attention layers to obtain the high-order image features.

4. The large model-based OCR recognition method according to claim 3, characterized in that, The method comprises the following steps: performing semantic pre-segmentation on the target contract image by using the input layer to distinguish text regions, table regions, seal regions and blank regions in the target contract image; calculating character density by using a multi-scale sliding window in the text regions and the table regions; performing dynamic region division on the target contract image in combination with the calculated character density to obtain a plurality of image blocks; extracting pixel features of each image block, and assigning a dynamic weight to each image block after pixel feature extraction to obtain the image features.

5. The large model-based OCR recognition method according to claim 1, wherein, The method comprises the following steps: obtaining a preset contract field vocabulary table; integrating the contract field vocabulary table into the language decoder to obtain a target language decoder; performing cross-modal reasoning on the high-order image features by using the target language decoder to obtain the unstructured text.

6. The large model-based OCR recognition method according to claim 1, wherein, The handwritten character recognition sub-model comprises a convolution layer, a long short-term memory network layer and a plurality of second self-attention layers, and the method comprises the following steps: obtain a handwritten character region image in the target contract image; obtain local stroke features in the handwritten character region image by using the convolution layer to obtain a stroke sequence; obtain a time sequence dependency in the stroke sequence by using the long short-term memory network layer to obtain a hidden state sequence containing the time sequence dependency; calculate attention weights of the hidden state sequence by using multiple layers of the second self-attention layer to obtain a global feature vector; perform feature conversion processing and decoding processing on the global feature vector to obtain the recognized text.

7. The large model-based OCR recognition method according to claim 1, characterized in that, The output module includes a preset knowledge graph, a preset risk rule library, and a visualization submodule. The output module is used to output a visual report based on the structured text and the recognized text, including: perform entity standardization processing and relationship verification on the structured text and the recognized text by using the knowledge graph to obtain structured information; perform risk rule matching on the structured information by using the risk rule library to obtain a risk list; perform risk level assessment and reasoning based on the risk list by using the visualization submodule to output the visual report.

8. A large model-based OCR recognition device, characterized by, The method includes: a data acquisition module configured to acquire a target contract image and a pre-trained visual language large model, the visual language large model including a visual encoder, a language decoder, a format constraint module, a handwritten character recognition submodel, and an output module; a feature extraction module configured to perform feature extraction on the target contract image by using the visual encoder to obtain high-order image features; a cross-modal reasoning module configured to perform cross-modal reasoning on the high-order image features by using the language decoder to obtain unstructured text; a structured generation module configured to perform multi-round autoregressive generation on the unstructured text by using the format constraint module to output structured text; a handwritten character recognition module configured to acquire a handwritten character region image in the target contract image by using the handwritten character recognition submodel and output recognized text based on the handwritten character region image; a report acquisition module configured to output a visual report based on the structured text and the recognized text by using the output module.

9. A large model-based OCR recognition device, characterized by, The OCR recognition device based on a large model includes a memory and at least one processor, and the memory stores instructions; at least one processor calls the instructions in the memory to enable the OCR recognition device based on a large model to perform each step of the OCR recognition method based on a large model according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon instructions, the instructions comprising, The instructions are executed by the processor to implement each step of the OCR recognition method based on a large model according to any one of claims 1-7.

Images