Multi-layer text representation and semantic reconstruction method and system for chinese scenario text

Abstract

The application discloses a multi-layer text representation and semantic reconstruction method and system for Chinese scene text, and relates to the fields of computer vision and natural language processing. In view of the problems that the prior art cannot distinguish between creative expression and real error, the processing flow is rigid and lacks interpretability, the application first detects and identifies image text to generate original text; then performs spelling correction and grammar correction on the original text to obtain standardized text; then extracts and analyzes the differences between the original text and the standardized text to generate explanation information containing error types and sources; finally, according to the application scene requirements, the original text, the standardized text or the explanation information is flexibly output. The application improves the text processing accuracy, retains the creative expression, and enhances the interpretability and scene adaptability of the system, and is suitable for intelligent translation, advertising creativity and visual impairment assistance scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, and in particular relates to a method and system for multi-layer text representation and semantic reconstruction of Chinese text. Background Technology

[0002] With the widespread adoption of smart devices, users increasingly rely on tools such as smartphones, augmented reality devices, and in-vehicle cameras to capture and recognize Chinese scene text for real-time translation, navigation, and information retrieval. However, currently widely used scene text processing technologies face significant challenges in practical applications. Firstly, in common scenarios such as shop signs, advertising signs, and traffic signs, text often employs creative expressions such as homophones, puns, artistic fonts, or deliberately misspelled words. These expressions are intuitive and intentional to humans, but existing machine processing technologies struggle to accurately recognize and process them. Mainstream technologies typically employ single spelling correction and grammar correction algorithms, often neglecting the creativity and flexibility of language. This results in machines failing to faithfully capture and retain the original text's true intent and stylistic features, potentially even damaging it.

[0003] Secondly, the complexity of text in a given scene and the inherent limitations of optical character recognition (OCR) technology itself lead to numerous recognition errors in practical applications. Under complex lighting conditions, background interference, or low resolution, traditional OCR technology is prone to misidentification, missed identification, and typos. While modern OCR models perform well in recognizing standard text, their accuracy remains insufficient for artistic fonts or text against complex backgrounds. For applications with high reliability requirements, such as traffic sign recognition in autonomous driving and assisted reading for visually impaired users, recognition errors directly impact system safety and user experience.

[0004] Furthermore, different downstream applications have fundamentally different text requirements. In tasks such as machine translation, speech-to-speech, and text retrieval, it is typically necessary to standardize the original text into text conforming to standard grammar and semantics. However, in scenarios such as legal evidence collection, digitization of historical archives, or advertising creative analysis, it is essential to maintain the original appearance of the text to ensure its authenticity and integrity. Most existing technologies provide a one-way "error correction" output, lacking a flexible mechanism that can simultaneously preserve the original information, provide a standardized version, and explain the differences between the two. Therefore, how to perform intelligent standardization and semantic reconstruction while faithfully reproducing the text in the context, and support on-demand output, has become a pressing technical challenge. At its root, the difficulty lies in the need for a systematic solution that can deeply integrate visual information and linguistic context, precisely distinguish between errors and creative ideas, and provide explainable explanations of the processing. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a multi-layered text representation and semantic reconstruction method and system for Chinese-language scenarios. This method improves text processing accuracy, preserves creative expression, and enhances the interpretability and scenario adaptability of the system. It is applicable to scenarios such as intelligent translation, advertising creativity, and visual impairment assistance.

[0006] To achieve the above objectives, this invention provides a multi-layered text representation and semantic reconstruction method for Chinese-language text, including: S1. Perform text region detection and localization on the acquired scene image; S2. Recognize the text content within the text area and generate original text that faithfully reflects the visual content of the image; S3. Perform language standardization processing on the original text to generate standardized text that conforms to standard syntax and semantics; S4. Extract and compare the differences between the original text and the normalized text to obtain text difference information with type annotation; S5. Based on the text difference information, generate explanatory information describing the reasons for and sources of the modification; S6. Based on the requirements of the target application scenario, select at least one from the original text, the normalized text, and the explanation information for output.

[0007] Optionally, in S1, text region detection and localization includes: using a text detection network based on the Transformer architecture to perform global context modeling on the scene image, so as to locate the text region in the image and output its position coordinates.

[0008] Optionally, in S2, generating the original text includes: using a multimodal OCR model to fuse the visual features of the text region with linguistic context information, recognizing and outputting the unmodified original text.

[0009] Optionally, in S3, the language normalization process includes: using a pre-trained language model to perform spelling correction on the original text; and using a large language model to perform grammatical correction and semantic reconstruction on the corrected text to generate the normalized text.

[0010] Optionally, in S4, extracting and comparing differences includes: using a text comparison algorithm to align the original text and the normalized text at the character or sub-word unit level, identifying editing operations such as replacement, insertion, or deletion; and labeling the editing operation as one of the following types based on the contextual semantics: spelling error, grammatical correction, or creative expression normalization.

[0011] Optionally, in S5, generating explanatory information includes: taking the text difference information with type annotation as input, generating structured explanatory information containing error type, error source, and modification suggestions; the error source includes OCR recognition error, original creative writing, or standardized modification.

[0012] Optionally, in S6, the selection based on the needs of the target application scenario includes: in scenarios where the original visual information needs to be preserved, selecting to output the original text; in tasks where standardized text needs to be standardized for subsequent processing, selecting to output the standardized text; in scenarios where the reasons for text modification need to be reviewed or understood, selecting to output the explanatory information or combining it with the original text or the standardized text for output.

[0013] On the other hand, to achieve the above objectives, the present invention also provides a multi-layer text representation and semantic reconstruction system for Chinese-language scenarios, including: The text region detection module is used to detect and locate text regions in the acquired scene images; The original text generation module is used to identify the text content within the text area and generate original text that is faithful to the visual content of the image; The normalized text generation module is used to perform language normalization processing on the original text to generate normalized text that conforms to standard syntax and semantics; The text difference analysis module is used to extract and compare the differences between the original text and the normalized text to obtain text difference information with type annotations; The explanation information generation module is used to generate explanation information describing the reasons and sources of the modification based on the text difference information; The output control module is used to select at least one from the original text, the normalized text, and the explanation information for output according to the needs of the target application scenario.

[0014] An electronic device, the electronic device comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the multi-layer text representation and semantic reconstruction method for Chinese scene text.

[0015] A computer storage medium storing computer program instructions, which, when executed by a processor, implement the multi-layer text representation and semantic reconstruction method for Chinese-language scenarios.

[0016] Technical Effects of this Invention: This invention discloses a multi-layered text representation and semantic reconstruction method and system for Chinese scene text. By generating a multi-layered representation of original text, standardized text, and explanatory information, it effectively distinguishes between real errors and creative expressions in scene text, thereby improving the quality of text standardization while avoiding the destruction of original semantics due to excessive correction. It integrates Transformer-based detection and multimodal recognition technologies, enhancing the robustness and accuracy of text recognition in challenging scenarios such as complex backgrounds and artistic fonts. By providing difference analysis and interpretable modification information, it significantly improves the transparency and credibility of the entire processing. Ultimately, the system can flexibly output the most suitable text form according to the specific needs of different application scenarios such as translation, advertising creativity, visual impairment assistance, or security monitoring, achieving a balance between processing accuracy, creative protection, and application flexibility. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the multi-layer text representation and semantic reconstruction method for Chinese-language text in an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] like Figure 1 As shown, this embodiment provides a multi-layered text representation and semantic reconstruction method for Chinese-language text, including: S1. Perform text region detection and localization on the acquired scene image; S2. Recognize the text content within the text area and generate original text that faithfully reflects the visual content of the image; S3. Perform language standardization processing on the original text to generate standardized text that conforms to standard syntax and semantics; S4. Extract and compare the differences between the original text and the normalized text to obtain text difference information with type annotation; S5. Based on the text difference information, generate explanatory information describing the reasons for and sources of the modification; S6. Based on the requirements of the target application scenario, select at least one from the original text, the normalized text, and the explanation information for output.

[0021] Furthermore, in S1, text region detection and localization includes: using a text detection network based on the Transformer architecture to perform global context modeling on the scene image, so as to locate the text region in the image and output its position coordinates.

[0022] Specifically, the implementation process of this embodiment includes: In the input image, the scene text detection module first accurately locates regions containing Chinese scene text. This detection module employs a text detection network based on the Transformer architecture. Compared to traditional convolutional neural network (CNN) models, the Transformer architecture has stronger global context modeling capabilities. This architecture can more effectively handle images with complex backgrounds, irregular fonts, and varying text shapes, achieving higher detection accuracy. Especially when dealing with images containing complex backgrounds, noise, or deformed fonts, the Transformer-based detection network maintains high accuracy.

[0023] The overall architecture of the text detection network based on the Transformer architecture uses ResNet50 as the backbone network, outputting multi-scale feature maps with stride=4, 8, 16, and 32 to comprehensively capture visual features at all levels, from edges and textures to text outlines. After processing the highest-level feature map (stride=32) output by the backbone network to generate sequential features, it is input into a 6-layer Transformer encoder. The global context relationship is modeled through an 8-head multi-head self-attention mechanism and a feedforward neural network with a hidden layer dimension of 2048, effectively capturing long-distance dependencies between text regions. Subsequently, a Feature Pyramid Network (FPN) is used to weightedly fuse the global features output by the Transformer encoder with the local features of each layer of the backbone network, generating a fused feature map with dimensions of H / 8×W / 8×256, balancing localization accuracy and semantic understanding ability. Finally, a prediction head consisting of 2 layers of 3×3 convolutions outputs the coordinates of the quadrilateral or rectangular bounding boxes of the text regions and their confidence scores. Low-confidence regions are filtered out using a confidence threshold to achieve accurate text region localization.

[0024] This detection model accurately extracts the location information of text regions in images through fine-grained image analysis. Specifically, the model determines the location of each text region by analyzing the visual features of the image (such as font, size, color, shape, etc.) and their relationship with the surrounding background. The output is the bounding box coordinates or polygon coordinates of the text region. This location information provides accurate region references for subsequent text recognition modules, ensuring that text recognition is performed only within the actual regions containing the text.

[0025] This step enables accurate extraction of text regions in complex scenes, avoiding the missed and false detection problems caused by background interference, poor image quality, or irregular text shapes in traditional methods. Precise text region localization improves the reliability and accuracy of scene text detection.

[0026] Furthermore, in S2, generating the original text includes: employing a multimodal OCR model to fuse the visual features of the text region with linguistic context information, recognizing and outputting the unmodified original text.

[0027] Specifically, the implementation process of this embodiment includes: After locating the text regions, the information from these regions is passed to a multimodal OCR model for further processing. This model combines visual information from the image with linguistic information from the text to accurately identify the text content in the image. In this process, an OCR model based on the TrOCR architecture is used, combining visual features such as font, color, and shape in the image with the linguistic structure and contextual information of the text to ensure accurate recognition of each text region. The multimodal OCR model focuses on fusing visual features and linguistic contextual information. First, a visual encoder such as ViT segments the text region image into a patch sequence, embeds positional encoding, and extracts the visual feature sequence. Then, the encoder performs contextual modeling on the dictionary embedding and positional encoded linguistic information to generate a linguistic feature sequence. Subsequently, a multi-head cross-attention mechanism is used to achieve precise alignment and fusion of visual and linguistic features, with visual features serving as queries and linguistic features as keys and values, resulting in cross-modal fused features. Finally, a decoder, combined with an autoregressive decoding strategy, maps the fused features to a dictionary probability distribution through a linear layer. After post-processing to filter invalid characters, the output is the original text faithful to the visual content of the image, while also possessing the ability to handle complex fonts, low-quality images, and semantic association recognition.

[0028] This OCR model can handle complex fonts, artistic fonts, and blurred characters, and is especially suitable for situations with poor image quality or non-standard fonts, greatly improving the accuracy of text recognition. Based on the visual-text alignment results output by the model, the actual text content in the image is extracted and the raw text (RawText) is generated. RawText preserves the original expression form in the image and does not make any grammatical or spelling modifications to the recognized text, ensuring that the text is faithful to the content in the original image.

[0029] Through this step, the present invention can provide a realistic text representation in an image, laying the foundation for subsequent text processing, standardization, and error correction.

[0030] Furthermore, in S3, the language normalization process includes: using a pre-trained language model to perform spelling correction on the original text; and using a large language model to perform grammatical correction and semantic reconstruction on the corrected text to generate the normalized text.

[0031] Specifically, the implementation process of this embodiment includes: After generating RawText, this invention further processes it to generate NormText. This process aims to correct spelling errors, grammatical errors, and unnatural expressions in the original text, ensuring the text conforms to standard grammatical rules and is adaptable to downstream tasks such as translation, text retrieval, and speech recognition. The generation of NormText mainly includes two sub-stages: spell correction and grammatical correction and semantic reconstruction.

[0032] First, the spelling correction stage primarily addresses potential misidentifications, omissions, and misspellings during OCR recognition. Since OCR technology is often affected by factors such as blurred fonts, lighting variations, or poor image quality when processing images, some characters may be inaccurately recognized or spelled incorrectly. Therefore, this invention employs a BERT pre-trained language model for spelling correction. This model takes the token sequence after word-level segmentation as input, fusing word embeddings, positional embeddings, and segment embeddings to construct an input feature matrix of dimension T×768 (T being the sequence length). Its core consists of 12 stacked bidirectional encoder layers, each containing a multi-head self-attention mechanism with 12 attention heads and a feedforward neural network with a hidden layer dimension of 3072, effectively capturing the bidirectional dependencies within the text context. During spelling correction, a masking language model strategy is used to randomly mask tokens in the original text that may contain spelling errors. Global contextual semantic modeling is used to output the feature representation of the masked position. Then, linear layer mapping and Softmax operations are used to predict the correct tokens. Simultaneously, a confidence threshold is set to filter low-confidence correction results, avoiding over-correction. This model corrects misidentified words by leveraging contextual information. Utilizing word vector matching and contextual semantic reasoning techniques, it accurately detects and corrects spelling errors, ensuring text accuracy and consistency. Spelling correction goes beyond simply correcting individual words; it involves semantic understanding based on context, avoiding the limitations of traditional OCR technology that relies solely on literal word spelling, thus significantly improving text quality.

[0033] Secondly, after spelling correction, this invention enters the grammar correction and semantic reconstruction stage. In this stage, the system uses large-scale models such as Qwen and GPT for grammar correction and semantic reconstruction to optimize the sentence structure and language expression of the text. The goal of grammar correction is not only to fix spelling errors but also to address issues such as non-standard vocabulary usage, unnatural syntactic structures, or non-compliance with standard written language norms. The large-scale model first performs full-domain preprocessing on the spell-corrected text, transforming it into structured feature vectors through character-level word segmentation, part-of-speech tagging, and syntactic parsing. Simultaneously, it loads a pre-trained grammar rule base (including core norms such as subject-verb agreement, word collocation, and punctuation usage) and a semantic knowledge base, laying the data foundation for subsequent processing. In the grammar correction stage, the large-scale model relies on context-aware technology to accurately detect issues such as non-standard vocabulary usage and unnatural syntactic structures. At the vocabulary level, it identifies collocation conflicts by calculating the semantic similarity and collocation frequency ratio between the target word and the context words; at the syntactic level, it judges the rationality of word order and the completeness of sentence structure by analyzing the distribution of self-attention weights. After detection, the large model invokes its built-in grammar correction algorithm and combines it with a semantic knowledge base to generate the optimal correction scheme. For specific application scenarios such as advertising copy and news headlines, this invention further reconstructs language to make sentences more fluent, natural, and conform to standard written language. The large model first clarifies the core information and potential expression needs of the text through topic modeling and semantic integrity analysis. Then, based on preset application scenario tags (such as advertising copy, news headlines, legal documents, etc.), it loads the style feature vectors of the corresponding scenarios and completes feature embedding and fusion. Driven by the large model, multiple versions of reconstructed text conforming to the scenario style are generated based on the fused scenario features and core text features. At the same time, the large model ensures the quality of reconstruction through a dual constraint mechanism: on the one hand, it ensures that the core meaning of the reconstructed text is consistent with the original text through cosine semantic similarity calculation, avoiding semantic deviation; on the other hand, it optimizes the fluency of the text through the perplexity score of the pre-trained language model, with lower perplexity indicating more natural expression. Finally, the large model performs full-domain validation on the corrected and reconstructed text, cross-validating grammatical norms, semantic consistency, and scenario adaptability, outputting optimized text that conforms to standard written language norms, retains the core information of the original, and fits the needs of specific application scenarios. It retains the meaning of the original information while making its language more concise and precise.

[0034] Through these two stages of processing, the generated normalized text conforms to standard grammatical rules, effectively improving its performance in downstream tasks such as translation, text retrieval, and speech recognition. Unlike traditional OCR technology's single spell correction, this invention performs comprehensive text optimization by incorporating contextual information, ensuring that the generated text meets high standards in grammar, expression, and logic, thus adapting to the needs of different tasks.

[0035] Furthermore, in S4, extracting and comparing differences includes: using a text comparison algorithm to align the original text and the normalized text at the character or sub-word unit level, identifying editing operations such as replacement, insertion, or deletion; and labeling the editing operation as a type of spelling error, grammatical correction, or creative expression normalization based on the contextual semantics of the editing operation.

[0036] Specifically, the implementation process of this embodiment includes: After obtaining the raw text (RawText) and normalized text (NormText), this invention performs systematic text difference extraction and alignment processing on both. First, RawText and NormText are preprocessed at the character and word levels, including unified encoding formats, word segmentation, and punctuation normalization, to ensure alignment under the same comparison benchmark. Then, a text comparison algorithm is invoked to construct an alignment path from RawText to NormText at the character or sub-word unit level. By analyzing editing operations such as "match," "substitution," "insertion," and "deletion" in the alignment path, the system can accurately locate text segments that changed during normalization and extract these segments as difference units.

[0037] After basic alignment, these discrepancy fragments are further mapped back to specific location intervals in the original and normalized texts, forming a set of discrepancy fragments with location annotations. For each discrepancy fragment, the system combines the spelling correction results, grammar correction results, and language model output from the previous stage to determine the error type and analyze semantic attributes. For example, when a discrepancy fragment involves only the replacement of a single word or a small number of characters and the contextual semantics remain consistent, the system tends to mark it as a spelling error or OCR misrecognition; when a discrepancy fragment involves word order adjustment, syntactic restructuring, or the addition or deletion of function words, it is marked as grammar correction; if the normalized text excessively rewrites the original expression with advertising, brand, or slogan characteristics, causing a change in semantic style, the fragment can be marked as excessive correction; for parts that inherently contain deliberate homophones, puns, or non-standard characters in RawText but are "normalized" in NormText, they can be marked as normalization modifications corresponding to creative usage.

[0038] Through the aforementioned difference extraction and type labeling process, this invention can distinguish at a fine-grained level which modifications are normal adjustments necessary for error correction and standardization, which modifications may constitute excessive intervention at the semantic or stylistic level, and which texts are inherently creative expressions and should not be simply regarded as "errors." The key to this step lies in the accurate extraction and reasonable classification of the differences. On the one hand, this ensures that the system can accurately distinguish the sources of different types of text modifications; on the other hand, it provides structured difference input for the subsequent explanatory information generation module. Based on these difference entries, traceable and interpretable MetaInfo is generated, thereby providing high-quality support for text selection and output strategies in different application scenarios.

[0039] Furthermore, in S5, generating explanatory information includes: taking the text difference information with type annotation as input, generating structured explanatory information containing error type, error source, and modification suggestions; the error source includes OCR recognition error, original creative writing, or standardized modification.

[0040] Specifically, the implementation process of this embodiment includes: After text difference extraction and alignment, this invention generates an explanatory information layer (MetaInfo) by combining text differences, contextual information, and visual features through an explanatory information generation module. MetaInfo provides a detailed description of the text modification process, aiming to provide interpretability and decision-making basis for subsequent system processing. Through MetaInfo, this invention can clearly annotate the reasons for and sources of text modifications, as well as how to conduct subsequent processing, ensuring greater transparency and operability in the system when processing text.

[0041] First, the system identifies the error types in the text through text difference comparison. Error types include spelling errors, grammatical errors, OCR misrecognition, and creative expression. In the case of spelling errors, the system will label it as a misspelling in OCR recognition; grammatical errors will be labeled as non-standard syntax or vocabulary usage; OCR misrecognition indicates misunderstanding or omission of text in the image; and creative expression indicates that although the text is grammatically correct, it belongs to non-standard or creative expression.

[0042] Secondly, the system labels errors based on their source, clearly indicating whether they stem from creative styling of the original text, OCR recognition errors, or modifications introduced during grammar correction. For creative styling, the system identifies it as the unique style of the original text, meaning this part itself does not require modification. For OCR misrecognition, the system records errors caused by image blurring or font issues during the recognition process, while grammar correction points to changes introduced by standardization processing. Through this labeling, the system provides a clear background and logical basis for subsequent modifications.

[0043] During the information generation process, modification suggestions are provided based on different application scenarios. These suggestions include whether to recommend using normalized text (NormText) or retaining the raw text (RawText) in subsequent tasks. If normalized text is recommended, the system will attach relevant modification explanations, clarifying why this modification is necessary for the task. For scenarios such as creative writing or advertising copywriting, the system suggests retaining the raw text or combining it with normalized text to ensure that creative expression is preserved while complying with legal, platform, and other regulatory requirements.

[0044] Finally, the generated MetaInfo not only provides interpretability for text modifications but also offers flexible control strategies for subsequent task processing. Depending on different application requirements, the system can choose to output the original text, normalized text, or a combination of both to ensure that the output accurately meets the task objectives. In scenarios requiring transparency (such as advertising design and legal evidence collection), the detailed explanations provided by MetaInfo assist the system in decision-making, ensuring the traceability and rationality of text modifications.

[0045] Furthermore, in S6, the selection based on the needs of the target application scenario includes: in scenarios where the original visual information needs to be preserved, selecting to output the original text; in tasks where standardized text needs to be standardized for subsequent processing, selecting to output the standardized text; in scenarios where the reasons for text modification need to be reviewed or understood, selecting to output the explanatory information or combining it with the original text or the standardized text for output.

[0046] Specifically, the implementation process of this embodiment includes: The output control module employs a flexible, multi-layered output strategy to dynamically select the output result based on different application scenarios and task requirements. Its core function is to determine whether to output raw text, normalized text, or explanatory information (MetaInfo) according to the specific requirements of the task, and to support combining these three types of output as needed, thereby meeting the practical needs of different scenarios.

[0047] The output control strategy is implemented based on different task requirements. Depending on the application scenario, the system makes a selection: when it needs to retain the original information, avoid modifying the original content, or ensure the authenticity of the information, the system will prioritize outputting RawText; in tasks that require ensuring standardized expression and grammatical correctness, the system will prioritize outputting NormText; and in tasks that require detailed explanations, especially in advertising creative design and content review, the system will prioritize outputting MetaInfo. Through this flexible, multi-layered output control strategy, the system can select the appropriate text version for output according to the needs of different application scenarios, ensuring that tasks can be flexibly processed according to actual requirements. The core workflow of this strategy begins with intelligent scene recognition. The system has a built-in scene type detector based on a text classification model, which can deeply analyze the core requirements and application background of the input task. By capturing the key features of the task text and semantically matching them with preset scene features, it accurately determines the scene category to which the task belongs, and finally outputs three clear scene labels: original information retention scenario, standardized processing scenario, and review and understanding scenario. Among them, the original information preservation scenario is applicable to scenarios with extremely high requirements for the integrity and authenticity of the original data, such as legal evidence collection and historical archive management. It is necessary to completely preserve the original text after OCR recognition without any additional modification or alteration. The standardization processing scenario is for scenarios that require unified text format and expression standards, such as intelligent translation, speech reading, and automated processing of office documents. It is necessary to output standardized text after spell correction, grammatical correction, and semantic reconstruction. The review and understanding scenario is for scenarios that require clear text optimization basis, such as advertising review, content quality inspection, and compliance review. It is necessary to provide optimized text and related explanatory information simultaneously.

[0048] After scene recognition, the system matches the corresponding output content and format based on the determined scene labels. For scenes that retain original information, the system directly outputs the unoptimized raw text, ensuring the purity and traceability of the original data. For standardized processing scenes, the system only outputs standardized text that has undergone full-process optimization, ensuring the consistency and usability of the text. For review and understanding scenes, the system supports two output combination modes: one is a combination of raw text and explanatory information, and the other is a combination of standardized text and explanatory information. The explanatory information covers key content such as error detection results, explanations of correction logic, semantic reconstruction basis, and reasons for scene adaptation, providing users with a clear optimization path reference. To meet users' personalized needs, this strategy also sets up a flexible dynamic adjustment mechanism, allowing users to customize the output content combination according to their actual application needs. At the same time, the system will perform format verification on the final output content to ensure that it meets the input requirements of downstream tasks. It supports multiple mainstream output formats such as plain text, JSON, and XML, ultimately achieving full-process adaptation from scene recognition to output delivery, comprehensively covering diverse usage needs in different scenarios.

[0049] Raw text is the text content obtained through OCR recognition, without any grammatical or spelling modifications, preserving the original expression form of the image. In some tasks, raw text needs to be output first, especially in scenarios with high security requirements, or when the system needs to retain the authenticity of the original image information. Outputting raw text ensures a faithful reproduction of the textual information in the image.

[0050] NormText is text obtained by correcting spelling errors and grammar in the original text. This text conforms to standard grammatical rules and is suitable for downstream tasks such as translation, text retrieval, and speech recognition. For these tasks, the system prioritizes outputting NormText to ensure accuracy and consistency.

[0051] MetaInfo provides a detailed description of the text differences, covering the type of text modification, the source of the modification, and the suggested modifications. MetaInfo not only provides explainability of text modifications but also offers control strategies for subsequent tasks, ensuring the system can flexibly adjust text processing methods according to actual needs. This information layer helps in making reasonable decisions when a clear distinction needs to be made between creative expression and standardized expression.

[0052] In some complex scenarios, a single text format may not meet all the requirements of the task. In such cases, the system supports outputting RawText, NormText, and MetaInfo simultaneously, or a combination of two of them depending on the task requirements. This allows the system to simultaneously provide the original text to preserve the true content, normalize the text to ensure high-quality language expression, and provide explanatory information to help understand the reasons and background for text modifications.

[0053] This embodiment provides a multi-layered text representation and semantic reconstruction system for Chinese-language scenarios, including: The text region detection module is used to detect and locate text regions in the acquired scene images; The original text generation module is used to identify the text content within the text area and generate original text that is faithful to the visual content of the image; The normalized text generation module is used to perform language normalization processing on the original text to generate normalized text that conforms to standard syntax and semantics; The text difference analysis module is used to extract and compare the differences between the original text and the normalized text to obtain text difference information with type annotations; The explanation information generation module is used to generate explanation information describing the reasons and sources of the modification based on the text difference information; The output control module is used to select at least one from the original text, the normalized text, and the explanation information for output according to the needs of the target application scenario.

[0054] Specifically, the implementation process of this embodiment includes: Scene Text Detection Module: This module detects text regions in an image and outputs their location information. Using a Transformer-based text detection network, it accurately locates text regions within the image. This module effectively handles complex backgrounds and irregularly shaped fonts, outputting the location information of each text region, providing accurate reference for subsequent text recognition.

[0055] Raw Text Recognition Module: This module is used to recognize the text content in images and generate RawText. Through visual-text multimodal alignment, it combines the visual features of the image (such as font, color, shape, etc.) and the linguistic information of the text (such as contextual structure) to ensure that the recognized text retains the expression in the original image without spelling or grammatical modification, and outputs the true raw text (RawText).

[0056] Normalized Text Generation Module: This module performs spell correction and grammar correction on RawText to generate NormText. Spell correction is performed using a BERT pre-trained language model, while grammar correction optimizes sentence structure using a large model (such as Qwen3-Max) to generate text that conforms to standard grammar rules (NormText). This module effectively improves the accuracy and fluency of the text, adapting it to the needs of downstream tasks.

[0057] Text Difference Extraction and Alignment Module: This module aligns RawText and NormText, extracts differences, and annotates them. Using text comparison algorithms (such as Levenshtein distance and dynamic time warping), the system analyzes the differences between the two and labels them as spelling errors, grammatical errors, creative expressions, etc. This step lays the foundation for subsequent explanatory information generation.

[0058] Explanation Information Generation Module: This module generates MetaInfo based on text differences and contextual information, annotating the reasons for modifications and the sources of errors. By analyzing text differences, it generates detailed explanation information, describing the type, source, and suggested modifications for each change, helping the system determine whether to use normalized text (NormText) or retain the original text (RawText) in subsequent tasks. This module provides a basis for system decision-making and enhances the interpretability of the processing.

[0059] The multi-layered output and control strategy module dynamically selects to output raw text, normalized text, or explanatory information (MetaInfo) based on task requirements. This module can flexibly combine outputs according to different application scenarios to ensure that specific task needs are met. For example, in advertising creative design, the raw text is retained; in translation tasks, normalized text is output; and in legal evidence collection tasks, the raw text is output along with MetaInfo to ensure the authenticity and completeness of the information.

[0060] An electronic device, the electronic device comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the multi-layer text representation and semantic reconstruction method for Chinese scene text.

[0061] A computer storage medium storing computer program instructions, which, when executed by a processor, implement the multi-layer text representation and semantic reconstruction method for Chinese-language scenarios.

[0062] Example 1: Creative Protection and Standardization of Shop Sign Text: Shop signs, as part of a business's image, often use creative fonts, homophones, puns, or misspellings. While these texts are visually appealing, traditional OCR technology is prone to recognition errors in these complex scenarios. To address this issue, this invention provides a novel method for scene text processing that can standardize the text while preserving its creativity, and provide different versions of output according to application requirements.

[0063] First, the system locates text regions in the input image using a text detection network based on the Transformer architecture. This network has powerful global context modeling capabilities, effectively handling complex backgrounds, artistic fonts, and irregular fonts, enabling accurate extraction of text regions even in complex scenarios like shop signs. Next, the system performs text recognition (RawText) on the identified text regions using a multimodal OCR model (such as the TrOCR architecture), combining visual features of the image with linguistic information to accurately identify the text content while preserving creative expression. Then, the system performs spell correction and grammar correction on the RawText, generating standardized text (NormText) that conforms to standard written language. These two processing stages ensure the linguistic standardization of the advertising copy while avoiding excessive modification of creative expression. Finally, the system extracts differences between the original text and the standardized text using text comparison algorithms such as Levenshtein distance, generating MetaInfo to explain the source and reasons for text modifications. Ultimately, the system flexibly chooses to output either the original text (RawText) or the standardized text (NormText) according to the needs of different platforms, achieving a balance between creativity and standardization.

[0064] This invention effectively protects the creative text of shop signs and adapts copywriting standards across different platforms, ensuring the consistency between the creative expression of advertising content and platform requirements.

[0065] Example 2: Recognition and Optimization of Traffic Sign Text: In autonomous driving systems, the accuracy of traffic sign recognition is crucial. Traditional OCR technology suffers from errors in recognizing text in traffic signs, especially in complex backgrounds, low light conditions, or when the sign is dirty, which may lead to the loss or misrecognition of traffic sign information. To address this issue, this invention provides a complete traffic sign text processing solution that combines OCR technology with semantic reconstruction methods, enabling efficient and accurate recognition and optimization of text content in traffic signs.

[0066] The system first uses a Transformer-based text detection network to accurately locate the text regions in the input traffic sign image. This network can handle complex situations such as background interference and reflections, ensuring accurate text extraction. Then, the system uses a multimodal OCR model to perform raw text recognition (RawText), effectively identifying the text content on the traffic sign. Next, the system performs spelling correction and grammar correction on the recognized text, generating standardized text (NormText) that conforms to a standard format, for example, changing "speed limit 60 km / h" to "speed limit 60 km / h". After obtaining the RawText and NormText, the system uses text comparison algorithms (such as Levenshtein distance) to extract the differences between them, marking spelling errors, formatting issues, etc., and using MetaInfo to explain the source and necessity of each modification. Finally, depending on the application requirements, the system can flexibly choose to output RawText or NormText, ensuring the accuracy of traffic sign information in the autonomous driving system.

[0067] This invention significantly improves the accuracy of traffic sign text recognition, especially under complex backgrounds and low-light conditions. The system can accurately identify and optimize traffic sign information, thus enhancing the reliability of autonomous driving systems.

[0068] Example 3: Assisted reading for visually impaired users: For visually impaired users, speech synthesis or real-time translation is relied upon to recognize and understand textual information in the environment. In practical applications, text in many environments has creative expressions (such as restaurant menus, shop signs, etc.), which often leads to semantic and grammatical inaccuracies when traditional OCR technology is used, thus affecting the user's understanding and experience.

[0069] The system accurately identifies text regions in the environment surrounding visually impaired users using a text detection model based on the Transformer architecture. After locating the text regions, the system uses a multimodal OCR model for raw text recognition (RawText), combining visual and contextual information to ensure accurate text extraction even in low light and complex backgrounds. Next, the system performs spell correction and grammar adjustment on the raw text, generating standardized text (NormText) that conforms to standard written language, making the text more suitable for speech synthesis. After extracting the differences between RawText and NormText, MetaInfo is generated, providing detailed explanations for each modification and offering modification suggestions based on different scenarios. The system selects to output either raw text or standardized text according to the scenario requirements, ensuring that visually impaired users can understand the surrounding text information.

[0070] This invention enables visually impaired users to obtain more accurate and standardized text information in real time, while preserving the creativity and contextual meaning of the text. The system improves recognition accuracy while also optimizing the user experience, helping visually impaired users better understand and use text information in their environment.

[0071] This invention discloses a multi-layered text representation and semantic reconstruction method and system for Chinese scene text. By generating multi-layered representations of original text, normalized text, and explanatory information, it effectively distinguishes between real errors and creative expressions in scene text, thereby improving the quality of text normalization while avoiding the destruction of original semantics due to excessive correction. It integrates Transformer-based detection and multimodal recognition technologies, enhancing the robustness and accuracy of text recognition in challenging scenarios such as complex backgrounds and artistic fonts. By providing difference analysis and interpretable modification information, it significantly improves the transparency and credibility of the entire processing. Ultimately, the system can flexibly output the most suitable text form according to the specific needs of different application scenarios such as translation, advertising creativity, visual impairment assistance, or security monitoring, achieving a balance between processing accuracy, creative protection, and application flexibility.

[0072] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-layered text representation and semantic reconstruction method for Chinese-language scenarios, characterized in that, include: S1. Perform text region detection and localization on the acquired scene image; S2. Recognize the text content within the text area and generate original text that faithfully reflects the visual content of the image; S3. Perform language standardization processing on the original text to generate standardized text that conforms to standard syntax and semantics; S4. Extract and compare the differences between the original text and the normalized text to obtain text difference information with type labels; the extraction and comparison of differences includes: using a text comparison algorithm to align the original text and the normalized text at the character or sub-word unit level, and identifying editing operations such as replacement, insertion, or deletion; based on the contextual semantics of the editing operation, label it as one of the following types: spelling error, grammar correction, or creative expression normalization; S5. Based on the text difference information, generate explanatory information describing the reasons and sources of the modification; generating explanatory information includes: taking the text difference information with type labels as input, generating structured explanatory information containing error type, error source and modification suggestions; the error source includes OCR recognition error, original creative writing or standardized modification; S6. Based on the requirements of the target application scenario, select at least one from the original text, the normalized text, and the explanation information for output.

2. The multi-layer text representation and semantic reconstruction method for Chinese-language scenarios as described in claim 1, characterized in that, In S1, text region detection and localization includes: using a text detection network based on the Transformer architecture to perform global context modeling on the scene image, so as to locate the text region in the image and output its position coordinates.

3. The multi-layer text representation and semantic reconstruction method for Chinese-language scenarios as described in claim 1, characterized in that, In S2, generating the original text includes: using a multimodal OCR model to fuse the visual features of the text region with linguistic context information, recognizing and outputting the unmodified original text.

4. The multi-layer text representation and semantic reconstruction method for Chinese-language scenarios as described in claim 1, characterized in that, In S3, language normalization processing includes: using a pre-trained language model to correct spelling errors in the original text; and using a large language model to perform grammatical correction and semantic reconstruction on the corrected text to generate the normalized text.

5. The multi-layer text representation and semantic reconstruction method for Chinese-language scenarios as described in claim 1, characterized in that, In S6, the selection based on the needs of the target application scenario includes: in scenarios where the original visual information needs to be preserved, the original text is selected to be output; in tasks where standardized text needs to be processed later, the standardized text is selected to be output; in scenarios where the reasons for text modification need to be reviewed or understood, the explanatory information is selected to be output or combined with the original text or the standardized text for output.

6. A multi-layered text representation and semantic reconstruction system for Chinese-language scenarios, characterized in that, The system is used to implement the multi-layer text representation and semantic reconstruction method for Chinese scene text as described in any one of claims 1-5, the system comprising: The text region detection module is used to detect and locate text regions in the acquired scene images; The original text generation module is used to identify the text content within the text area and generate original text that is faithful to the visual content of the image; The normalized text generation module is used to perform language normalization processing on the original text to generate normalized text that conforms to standard syntax and semantics; The text difference analysis module is used to extract and compare the differences between the original text and the normalized text to obtain text difference information with type annotations; The explanation information generation module is used to generate explanation information describing the reasons and sources of the modification based on the text difference information; The output control module is used to select at least one from the original text, the normalized text, and the explanation information for output according to the needs of the target application scenario.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the multi-layer text representation and semantic reconstruction method for Chinese scene text as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the multi-layer text representation and semantic reconstruction method for Chinese scene text as described in any one of claims 1-5.

Images