A multi-language OCR recognition enhancement method, apparatus, device and medium

Abstract

This invention provides a multilingual OCR recognition enhancement method, apparatus, device, and medium. The method includes: performing text recognition on an input image using the EasyOCR engine to obtain the original text, location coordinates, and recognition confidence; inputting the recognition result into a large language model, driving the model to perform language recognition, text correction, multilingual translation, and key information extraction through a preset prompt word template; matching and fusing the location coordinates and confidence output by EasyOCR with the enhanced information output by the large model line by line to generate a structured output containing the original text, corrected text, language type, translated text, confidence, location coordinates, and key product information; this invention combines the speed of traditional OCR recognition with the semantic understanding capabilities of a large model to achieve high-precision multilingual text recognition and information extraction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multilingual OCR recognition enhancement method, apparatus, device, and medium. Background Technology

[0002] In cross-border e-commerce, international brand operation, and multilingual product retrieval, product images often contain mixed text in multiple languages, including Simplified Chinese, English, Japanese, Korean, French, and less common languages ​​such as Thai and Vietnamese. This text carries crucial information such as product names, prices, specifications, and brands, and is the core basis for product information entry, cross-language retrieval, and international market compliance verification. However, current methods for multilingual text recognition in product images suffer from numerous technical bottlenecks, making it difficult to meet the accuracy and efficiency requirements of actual business operations. Specifically: I. Traditional OCR tools are mostly optimized for a single or a few languages, and are not well adapted to minority languages ​​and multilingual mixed scenarios. They are prone to character misrecognition and omission, especially for languages ​​with special writing forms such as Arabic and Thai, where the recognition accuracy will drop significantly.

[0003] Second, product images often have complex backgrounds and diverse text layouts, such as text with superimposed decorative elements, interference from light and shadow, and font distortion. Traditional OCR recognition is easily affected by these factors, which not only cause text errors and low confidence, but also cannot effectively correct the recognition results.

[0004] Third, existing recognition methods can only output raw recognized text and lack core functions such as language type judgment, cross-language translation and key information extraction. They require manual secondary processing, resulting in low efficiency in product information entry and cross-language verification, and cannot meet the automated processing needs of massive cross-border e-commerce products.

[0005] Fourth, the multilingual recognition process is fragmented, with recognition, correction, and translation steps operating independently. This makes it difficult to achieve structured integration of results from each step and fails to provide unified and standardized data support for subsequent business operations. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a multilingual OCR recognition enhancement method, device, equipment, and medium. By integrating the multilingual rapid recognition capabilities of the EasyOCR tool with the semantic enhancement capabilities of a large model, an integrated process of "basic recognition - semantic optimization - result fusion" is constructed. This enables accurate recognition, error correction, language classification, cross-language translation, and key information extraction of multilingual text in product images. At the same time, it outputs structured fusion results, providing reliable data support for scenarios such as cross-border e-commerce product information management, multilingual retrieval, and international compliance verification, and significantly improving the automation and intelligence level of business processing.

[0007] In a first aspect, the present invention provides a multilingual OCR recognition enhancement method, comprising the following steps: Step 1: Use the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; Step 2: Determine the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassemble the discrete text segments into text blocks with logical coherence; Step 3: Input the text block and its corresponding recognition confidence into the pre-configured large language model, and sequentially call the preset four-stage prompt word templates to drive the large language model to perform enhancement processing, wherein: Step 3.1: Call the first prompt word template. The first prompt word template contains language type definition and language recognition instructions, which are used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and generate a comprehensive confidence score for language recognition based on the recognition confidence score. Step 3.2: Input the language type and overall confidence score together with the original text block, and call the second prompt word template. The second prompt word template contains text correction instructions, which are used to guide the large language model to correct the segments in the original text with a recognition confidence score lower than a preset threshold, based on the language type and the contextual semantics of the text block. At the same time, the original text of segments with a recognition confidence score higher than or equal to the preset threshold is retained. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence score. Step 3.3: Input the complete corrected text block of the output and call the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate a unified target language text. Step 3.4: Input the target language text and call the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text and form a structured information summary. Step 4: Combine the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence to generate the final structured output containing spatial location, original text, corrected text, language type, translated text, confidence, and key product information.

[0008] Secondly, the present invention provides a multilingual OCR recognition enhancement device, comprising: The OCR recognition module uses the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; The text preprocessing module determines the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassembles the discrete text segments into text blocks with logical coherence. The semantic enhancement module inputs the text block and its corresponding recognition confidence into a pre-configured large language model, and sequentially calls four preset stage prompt word templates to drive the large language model to perform enhancement processing, wherein: The first template invocation unit invokes the first prompt word template, which contains a language type definition and a language recognition instruction. This template is used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and to generate a comprehensive confidence score for language recognition by combining the recognition confidence score. The second template invocation unit takes the language type and overall confidence level as input along with the original text block, and invokes the second prompt word template. The second prompt word template contains text correction instructions, which guide the large language model to correct the segments in the original text with a recognition confidence level lower than a preset threshold, based on the language type and combined with the contextual semantics of the text block. At the same time, it retains the original text of segments with a recognition confidence level higher than or equal to the preset threshold. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence level. The third template invocation unit inputs the complete corrected text block of the output and invokes the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate unified target language text. The fourth template invocation unit inputs the target language text and invokes the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text to form a structured information summary. The result fusion module associates and fuses the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence level to generate a final structured output containing spatial location, original text, corrected text, language type, translated text, confidence level, and key product information.

[0009] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0010] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0011] One or more technical solutions provided by this invention have at least the following technical effects or advantages: 1. This invention significantly improves the accuracy of multilingual recognition through a combination of "EasyOCR fast recognition + large model enhancement": EasyOCR supports multilingual configuration and adapts to the recognition needs of less commonly spoken languages; the Doubao large model can accurately correct OCR misidentified text (especially spelling and grammatical errors in less commonly spoken languages), with a language classification accuracy of over 95%, and the translation results are consistent with the semantics of the product scenario, effectively solving the problems of poor multilingual adaptation and large recognition errors in traditional OCR.

[0012] 2. This invention constructs a fully automated process that integrates multiple steps such as recognition, correction, translation, extraction, and fusion. It requires no manual intervention and can complete the multilingual recognition and processing of a single 1024×1024 resolution product image within 5 seconds. It supports GPU acceleration and batch processing, adapts to the automated processing needs of massive product images in cross-border e-commerce, and improves efficiency by more than 60% compared to traditional fragmented processes.

[0013] 3. The present invention outputs structured fusion results, which include both original identification data (location, confidence level) and enhanced data (corrected text, translation, key information), and can be directly connected to business systems such as cross-border e-commerce product management, multilingual retrieval, and compliance verification; the language list and model parameters can be flexibly adjusted to adapt to the needs of different countries / regions for minority languages, and the environment variable configuration method takes into account both security and deployment convenience, making it highly versatile.

[0014] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0015] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0016] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0017] The overall concept of the technical solution in this application is as follows: This invention achieves end-to-end recognition and optimization of multilingual text in product images through a four-step core process: "multilingual basic recognition - EasyOCR result acquisition - Doubao large model semantic enhancement - structured result fusion." It combines the synergistic effects of the EasyOCR tool, the Doubao large model API, image processing technology, and data fusion algorithms. Each step is closely integrated, ensuring both recognition efficiency and improved accuracy and practicality. Specific steps and terminology are explained below: (I) Terminology Explanation 1. EasyOCR: An open-source optical character recognition tool that supports multilingual text detection and recognition, is adapted to GPU-accelerated computing, allows flexible configuration of the list of recognized languages, is suitable for preliminary extraction of multilingual text in complex backgrounds, and has the advantages of fast recognition speed and wide language coverage.

[0018] 2. Doubao Large Model: A large language model with powerful semantic understanding, text correction, language classification and translation capabilities. It can intelligently process OCR raw text through API interface and output structured results. Low temperature parameter configuration can ensure output stability and format standardization.

[0019] 3. Structured Fusion Results: The text location and confidence score recognized by EasyOCR are integrated with the corrected text, language type, translation results, and key product information output by the Doubao big model to form a unified data set, which is convenient for subsequent business systems to call.

[0020] 4. Language Mapping Table: A pre-defined table mapping OCR language identifiers to natural language names, used to convert EasyOCR language abbreviations (such as ch_sim, ja) into standard language names (such as Simplified Chinese, Japanese) that can be recognized by the Doubao model, ensuring the accuracy of language classification.

[0021] (II) Initial configuration of multilingual OCR engine Construct the MultiLangOCREngine class to initialize the EasyOCR reader, the Doubao large model API, and the language mapping table, laying the foundation for the subsequent recognition process. The specific configuration is as follows: 1. Language list configuration: By default, it loads a list of common and less common languages ​​such as Simplified Chinese, English, Japanese, Korean, French, German, Spanish, Russian, and Arabic. It supports flexible adjustment of language parameters according to actual business scenarios (such as adding Thai and Vietnamese for Southeast Asian market adaptation) to meet different cross-border business needs.

[0022] 2. EasyOCR Initialization: Enable GPU acceleration mode to build the EasyOCR reader, improve the speed of multilingual text recognition, and adapt to batch processing scenarios with massive product images; if there is no GPU environment, you can switch to CPU mode by adjusting parameters to ensure tool compatibility.

[0023] 3. Doubao large model configuration: Pass the API address and key from environment variables or directly to avoid security risks caused by hard-coded keys; the default model is Doubao Professional Edition, and the temperature parameter is configured to 0.1 to ensure that the output results are stable and strictly follow the JSON format for easy program parsing.

[0024] 4. Language Mapping Table Construction: Establish the correspondence between OCR language abbreviations and standard language names to achieve standardized display of language types in the recognition results, providing support for subsequent cross-language translation and classification statistics.

[0025] (III) EasyOCR Basic Recognition and Result Collection The input product images are preprocessed and multilingual character recognition is performed to extract text content, confidence scores, and location coordinates, forming the original recognition result set. The specific steps are as follows: 1. Product Image Reading: Reads product image files using the OpenCV library and converts the images into a format that can be processed by EasyOCR, adapting to product image inputs of different resolutions and formats (JPG, PNG, etc.).

[0026] 2. Multilingual Text Recognition: Call the readtext method of EasyOCR reader to perform text detection and recognition on the image, and output the raw results including text content, confidence score (recognition accuracy evaluation value) and bounding box coordinates (coordinates of the four corner points).

[0027] 3. Results Processing and Formatting: Traverse the original recognition results, convert the bounding box coordinates to integer format, retain three decimal places for the confidence score, and organize them into a structured list of "text-confidence score-location coordinates"; at the same time, concatenate all the original text into a string to provide input data for subsequent processing of the Doubao large model.

[0028] 4. Error Handling: For abnormal scenarios such as image reading failure and OCR recognition error, return error status and detailed information to ensure process stability and facilitate troubleshooting.

[0029] (iv) Semantic enhancement processing of the Doubao large model The raw text is input into the EasyOCR API, and guided by precise prompts, language recognition, text correction, multilingual translation, and key information extraction are completed, outputting structured augmented results. The specific steps are as follows: 1. API Request Validation: First, verify whether the API address and key of the Doubao Big Model are configured completely. If they are missing, return a configuration error message to avoid invalid requests.

[0030] 2. Prompt word construction: Clearly define task requirements and output format. The prompt words include four core tasks: language type determination (limiting the range of supported languages), text correction (correcting OCR misidentification and spelling errors in minor languages), multilingual translation (converting to Simplified Chinese), and structured output (specifying JSON fields, including corrected text, language type, translation results, and key product information). At the same time, supplementary image scene descriptions (such as product images in Southeast Asian minor languages) are provided to help the model understand the context and improve processing accuracy.

[0031] 3. API Call and Result Parsing: Construct the request header (including content type and authentication information) and request body (including model name, prompt words, temperature parameters, and output format requirements), and send a POST request to the Doubao Big Model API; set a 30-second timeout to avoid request blocking; after receiving the response, parse the JSON format result and extract the enhanced data.

[0032] 4. Exception handling: For exceptions such as API request failure, incorrect response format, and timeout, return corresponding error information to ensure process controllability.

[0033] (v) Structured Result Fusion and Output The EasyOCR original recognition results are combined with the Doubao large model enhancement results to fill in missing data and form a final integrated structured result. The specific steps are as follows: 1. Result Matching and Fusion: Match EasyOCR results and Doubao enhancement results according to the number of lines in the text. Supplement the corrected original text, language type, and simplified Chinese translation for each text, while retaining the original text, confidence level, and location coordinates. If the number of lines in the enhancement result is inconsistent with the original result, the missing part is retained from the original text, and the language type is marked as "unknown" to ensure data integrity.

[0034] 2. Extraction of key product information: Extract key product information (such as name, price, specifications, and brand) from the Doubao big model, categorize and display it separately, so that it can be directly called by subsequent product information entry, retrieval and other business operations.

[0035] 3. Final Result Encapsulation: Integrate and merge the text details, key product information, and process summary, and output a complete response including status indicators, raw data, enhanced data, and the final fusion result. Supports JSON formatted printing and adapts to the data receiving requirements of business systems.

[0036] (vi) Full process execution and example verification The main execution function `run()` enables fully automated calls throughout the entire process. Users only need to pass in the product image path and scene description to complete the one-stop processing of "recognition-enhancement-fusion". Sample code is also provided, and API information can be configured through environment variables or by directly passing parameters to adapt to different deployment scenarios. After execution, the status indicator can be used to determine whether it is successful. If successful, the formatted final result will be output; if it fails, the error reason will be displayed, which is convenient for users to use and debug quickly.

[0037] Example 1 like Figure 1 As shown, this embodiment provides a multilingual OCR recognition enhancement method, including the following steps: Step 1: Use the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; Step 2: Determine the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassemble the discrete text segments into text blocks with logical coherence; Step 3: Input the text block and its corresponding recognition confidence into the pre-configured large language model, and sequentially call the preset four-stage prompt word templates to drive the large language model to perform enhancement processing, wherein: Step 3.1: Call the first prompt word template. The first prompt word template contains language type definition and language recognition instructions, which are used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and generate a comprehensive confidence score for language recognition based on the recognition confidence score. Step 3.2: Input the language type and overall confidence score together with the original text block, and call the second prompt word template. The second prompt word template contains text correction instructions, which are used to guide the large language model to correct the segments in the original text with a recognition confidence score lower than a preset threshold, based on the language type and the contextual semantics of the text block. At the same time, the original text of segments with a recognition confidence score higher than or equal to the preset threshold is retained. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence score. Step 3.3: Input the complete corrected text block of the output and call the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate a unified target language text. Step 3.4: Input the target language text and call the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text and form a structured information summary. Step 4: Combine the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence to generate the final structured output containing spatial location, original text, corrected text, language type, translated text, confidence, and key product information.

[0038] In this embodiment, preferably, step 2 further includes: Step 2.1: Based on the text position coordinates of each text segment, calculate the spatial distance and arrangement direction between adjacent text segments, and cluster the text segments that are spatially adjacent and have the same arrangement direction into candidate text groups; Step 2.2: Based on the spatial relationship of the candidate text groups, determine the reading order between the text groups according to the preset reading direction rules, and sort the text segments within the group according to the arrangement direction; Step 2.3: Segment the sorted text fragments according to the reading order, identify paragraph boundaries based on the blank areas or separators in the spatial layout, insert paragraph separators into the splicing result, and form text blocks with logical structure.

[0039] In this embodiment, preferably, The first prompt word template is configured to include a predefined list of language types and a description of the linguistic features of each language, as well as instructions to guide the large language model to perform language recognition based on character features and contextual semantics, for executing step 3.1; The second prompt word template is configured to include text correction instructions and dynamically fill in the language type of each text fragment, the original text block, and the recognition confidence of each fragment when it is called. It is used to guide the large language model to perform selective text correction based on language type guidance and confidence weight, and to execute step 3.2. The third prompt word template is configured to contain cross-language translation instructions and dynamically fill in the complete corrected text block when called, which is used to guide the large language model to maintain the semantic integrity and terminology consistency of the original text during translation, and to execute step 3.3; The fourth prompt word template is configured to include predefined product information categories and their semantic features, as well as key information extraction instructions. When called, the target language text is dynamically filled in to guide the large language model to identify and extract key product information from the target language text, and to execute step 3.4.

[0040] In this embodiment, preferably, step 3.2 specifically includes: Step 3.2.1: Based on the comparison result between the recognition confidence and the preset threshold, text segments with recognition confidence lower than the preset threshold are marked as regions to be corrected, and text segments with recognition confidence higher than or equal to the preset threshold are marked as regions to be retained. Step 3.2.2: For each region to be corrected, based on its language type and the contextual semantics of the text block it belongs to, call the large language model to generate multiple candidate correction texts, and obtain the generation probability of each candidate correction text from the output of the large language model. Step 3.2.3: Replace the corresponding region to be corrected with each candidate text and combine it with the surrounding preserved region to form multiple candidate complete text blocks. Re-input each candidate complete text block into the large language model to obtain the overall semantic coherence score of each candidate complete text block from the large language model. Step 3.2.4: Based on the generation probability of each candidate correction text and the semantic coherence score of the corresponding candidate complete text block, calculate the weighted comprehensive score, and select the candidate correction text with the highest score as the correction result. Step 3.2.5: The product of the generation probability of the selected candidate corrected text and the semantic coherence score of the corresponding candidate complete text block is used as the correction confidence of the text segment. The numerical range of the correction confidence is between 0 and 1.

[0041] In this embodiment, preferably, step 3.4 specifically includes: Step 3.4.1: Predefine the product information categories to be extracted and their semantic features. The product information categories include at least one of the following: product name, brand, price, specifications, production date, shelf life, and country of origin. Step 3.4.2: Based on the target language text, named entity recognition technology is used to identify text fragments that conform to the semantic features of each information category as candidate entities, where: Text fragments that match the semantic features of product names are identified as product name entities; Text fragments that match the semantic features of the brand identity are identified as brand entities; Text fragments that conform to the semantic features of price values ​​are identified as price entities, which include combinations of currency symbols and numbers; Text fragments that conform to the semantic features of specification parameters are identified as specification entities; Text fragments that match the semantic features of production date are identified as production date entities; Text fragments that match the semantic features of shelf life are identified as shelf life entities; Text fragments that match the semantic features of country of origin are identified as country of origin entities; And obtain the recognition confidence of each candidate entity from the output of the large language model; Step 3.4.3: Record the start and end character positions of each candidate entity in the target language translated text, and generate an entity character position index; Step 3.4.4: Organize each candidate entity and its recognition confidence score, as well as the entity character position index corresponding to each candidate entity, according to a predefined JSON data structure to generate a structured information digest containing entity type, entity text, recognition confidence score, and entity character position index.

[0042] In this embodiment, preferably, step 4 further includes: Step 4.1: Based on the reading order in Step 2, create a line index for the original text and record the original text content, text position coordinates, and recognition confidence of all text fragments contained in each line; Step 4.2: Restore the corrected text blocks to line-by-line text according to the reading order in Step 2, and assign the same line index as in Step 4.1 to each line of corrected text; Step 4.3: Assign the language types generated in Step 3.1 to the corresponding rows according to the reading order determined in Step 2, and assign the same row index as in Step 4.1 to each row of language types; Step 4.4: Reconstruct the target language text into line-by-line translation text according to the reading order in Step 2, and assign the same line index as in Step 4.1 to each line of translation text; Step 4.5: Based on the same line index, associate the text position coordinates with the corrected text with the same line index in Step 4.2, the language type with the same line index in Step 4.3, and the translated text with the same line index in Step 4.4 to generate a line-level mapping table containing the line index, original text, corrected text, language type, translated text, and text position coordinates. Step 4.6: For each text segment in the original text in Step 4.1, the corresponding recognition confidence and the correction confidence generated in Step 3.2.5 are weighted and fused. The weights are adjusted according to preset rules. The correction confidence weight of the region to be corrected is greater than the original recognition confidence, and the original recognition confidence weight of the region to be retained is greater than the correction confidence. A comprehensive confidence score for each text segment is generated. Step 4.7: Traverse each candidate entity in the structured information digest generated in step 3.4.4, and according to its corresponding entity character position index, find the text line to which the character position index belongs in the line-level mapping table generated in step 4.5, and assign the text position coordinates of the text line to the candidate entity. Step 4.8: For candidate entities whose character position index spans multiple text lines, merge the text position coordinates of the text line where the starting character position is located with the text position coordinates of the text line where the ending character position is located to generate the complete spatial location region of the candidate entity. Step 4.9: Organize the entity type, entity text, recognition confidence and corresponding text location coordinates of each candidate entity obtained after processing in Steps 4.7 and 4.8 according to the predefined JSON data structure to generate an enhanced structured information digest containing spatial location information, which serves as the key product information field in the final structured output.

[0043] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0044] Example 2 like Figure 2 As shown, this embodiment provides a multilingual OCR recognition enhancement device, including: The OCR recognition module uses the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; The text preprocessing module determines the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassembles the discrete text segments into text blocks with logical coherence. The semantic enhancement module inputs the text block and its corresponding recognition confidence into a pre-configured large language model, and sequentially calls four preset stage prompt word templates to drive the large language model to perform enhancement processing, wherein: The first template bar unit calls the first prompt word template, which contains language type definition and language recognition instructions. It is used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and to generate a comprehensive confidence score for language recognition based on the recognition confidence score. The second template bar unit takes the language type and overall confidence level as input along with the original text block, and calls the second prompt word template. The second prompt word template contains text correction instructions, which are used to guide the large language model to correct the segments in the original text with a recognition confidence level lower than a preset threshold, based on the language type and combined with the contextual semantics of the text block. At the same time, it retains the original text of segments with a recognition confidence level higher than or equal to the preset threshold. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence level. The third template bar unit inputs the complete corrected text block of the output and calls the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate a unified target language text. The fourth template unit inputs the target language text and calls the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text to form a structured information summary. The result fusion module associates and fuses the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence level to generate a final structured output containing spatial location, original text, corrected text, language type, translated text, confidence level, and key product information.

[0045] In this embodiment, preferably, the text preprocessing module further includes: The spatial clustering unit calculates the spatial distance and arrangement direction between adjacent text segments based on the text position coordinates of each text segment, and clusters text segments that are spatially adjacent and have the same arrangement direction into candidate text groups; The reading order determination unit determines the reading order between the text groups according to the spatial relationship of the candidate text groups and according to the preset reading direction rules, and sorts the text segments within the group according to the arrangement direction. The logical reorganization unit splices the sorted text fragments according to the reading order, identifies paragraph boundaries based on the blank areas or separator lines in the spatial layout, inserts paragraph separator marks into the splicing result, and forms text blocks with logical structure.

[0046] In this embodiment, preferably, The first prompt word template is configured to include a predefined list of language types and a description of the linguistic features of each language, as well as instructions to guide the large language model to perform language recognition based on character features and contextual semantics, and is used to execute the first template invocation unit; The second prompt word template is configured to include text correction instructions and dynamically fill in the language type of each text fragment, the original text block, and the recognition confidence of each fragment when it is called. It is used to guide the large language model to perform selective text correction based on language type guidance and confidence weight, and to execute the second template calling unit. The third prompt word template is configured to contain cross-language translation instructions and dynamically fill in the complete corrected text block when called. It is used to guide the large language model to maintain the semantic integrity and terminology consistency of the original text during translation and to execute the third template calling unit. The fourth prompt word template is configured to include predefined product information categories and their semantic features, as well as key information extraction instructions. When called, the target language text is dynamically filled in to guide the large language model to identify and extract key product information from the target language text, and to execute the fourth template calling unit.

[0047] In this embodiment, preferably, the second template calling unit specifically comprises: Based on the comparison result between the recognition confidence level and the preset threshold, text segments with recognition confidence levels lower than the preset threshold are marked as regions to be corrected, and text segments with recognition confidence levels higher than or equal to the preset threshold are marked as regions to be retained. For each region to be corrected, based on its language type and the contextual semantics of the text block it belongs to, a large language model is invoked to generate multiple candidate correction texts, and the generation probability of each candidate correction text is obtained from the output of the large language model. Replace the corresponding region to be corrected with each candidate text and combine it with the surrounding preserved region to form multiple candidate complete text blocks. Re-input each candidate complete text block into the large language model to obtain the overall semantic coherence score of each candidate complete text block from the large language model. Based on the generation probability of each candidate correction text and the semantic coherence score of the corresponding candidate complete text block, a weighted comprehensive score is calculated, and the candidate correction text with the highest score is selected as the correction result. The product of the generation probability of the selected candidate corrected text and the semantic coherence score of its corresponding candidate complete text block is used as the correction confidence of the text segment, and the value of the correction confidence ranges from 0 to 1.

[0048] In this embodiment, preferably, the fourth template calling unit specifically comprises: The categories of product information to be extracted and their semantic features are predefined. The product information categories include at least one of the following: product name, brand, price, specifications, production date, shelf life, and country of origin. Based on the target language text, named entity recognition technology is used to identify text fragments that conform to the semantic features of each information category as candidate entities, where: Text fragments that match the semantic features of product names are identified as product name entities; Text fragments that match the semantic features of the brand identity are identified as brand entities; Text fragments that conform to the semantic features of price values ​​are identified as price entities, which include combinations of currency symbols and numbers; Text fragments that conform to the semantic features of specification parameters are identified as specification entities; Text fragments that match the semantic features of production date are identified as production date entities; Text fragments that match the semantic features of shelf life are identified as shelf life entities; Text fragments that match the semantic features of country of origin are identified as country of origin entities; And obtain the recognition confidence of each candidate entity from the output of the large language model; Record the start and end character positions of each candidate entity in the target language translated text, and generate an entity character position index; Each candidate entity, its recognition confidence score, and the corresponding entity character position index are organized according to a predefined JSON data structure to generate a structured information digest containing entity type, entity text, recognition confidence score, and entity character position index.

[0049] In this embodiment, preferably, the result fusion module further includes: The raw text line indexing unit establishes a line index for the raw text according to the reading order of the text preprocessing module, and records the raw text content, text position coordinates and recognition confidence of all text segments contained in each line; The text line restoration unit corrects the text blocks and restores them into line-by-line text according to the reading order of the text preprocessing module, and assigns the same line index as the original text line index unit to each line of corrected text. The language type allocation unit assigns language types to corresponding lines according to the reading order determined by the text preprocessing module, and assigns the same line index as the original text line index unit to each line of language types. The translation text line restoration unit restores the target language text into line-by-line translation text according to the reading order of the text preprocessing module, and assigns the same line index as the original text line index unit to each line of translation text; The line-level mapping table generation unit, based on the same line index, associates the text position coordinates with the corrected text with the same line index in the corrected text line restoration unit, the language type with the same line index in the language type allocation unit, and the translated text with the same line index in the translated text line restoration unit, to generate a line-level mapping table containing the line index, the original text, the corrected text, the language type, the translated text, and the text position coordinates. The comprehensive confidence calculation unit performs a weighted fusion of the recognition confidence and the correction confidence for each text segment within the original text in the original text line index unit. The weights are adjusted according to preset rules. The correction confidence weight of the region to be corrected is greater than the original recognition confidence weight, and the original recognition confidence weight of the region to be retained is greater than the correction confidence weight, thereby generating a comprehensive confidence score for each text segment. The line attribution determination unit traverses each candidate entity in the structured information digest, and according to its corresponding entity character position index, searches for the text line to which the character position index belongs in the line-level mapping table generated by the line-level mapping table generation unit, and assigns the text position coordinates of the text line to the candidate entity. The cross-line merging unit merges the text position coordinates of the text line where the starting character position is located and the text position coordinates of the text line where the ending character position is located to generate the complete spatial location region of the candidate entity for a candidate entity whose character position index spans multiple text lines. The entity spatial information output unit organizes the entity type, entity text, recognition confidence score, and corresponding text location coordinates of each candidate entity obtained after processing by the row attribution determination unit and the cross-row merging unit according to a predefined JSON data structure, and generates an enhanced structured information summary containing spatial location information, which serves as the key product information field in the final structured output.

[0050] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0051] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3.

[0052] Example 3 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the implementation methods in Embodiment 1.

[0053] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0054] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 4.

[0055] Example 4 This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it can implement any of the implementation methods in Embodiment 1.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multilingual OCR recognition enhancement method, characterized in that, Includes the following steps: Step 1: Use the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; Step 2: Determine the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassemble the discrete text segments into text blocks with logical coherence; Step 3: Input the text block and its corresponding recognition confidence into the pre-configured large language model, and sequentially call the preset four-stage prompt word templates to drive the large language model to perform enhancement processing, wherein: Step 3.1: Call the first prompt word template. The first prompt word template contains language type definition and language recognition instructions, which are used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and generate a comprehensive confidence score for language recognition based on the recognition confidence score. Step 3.2: Input the language type and overall confidence score together with the original text block, and call the second prompt word template. The second prompt word template contains text correction instructions, which are used to guide the large language model to correct the segments in the original text with a recognition confidence score lower than a preset threshold, based on the language type and the contextual semantics of the text block. At the same time, the original text of segments with a recognition confidence score higher than or equal to the preset threshold is retained. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence score. Step 3.3: Input the complete corrected text block of the output and call the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate a unified target language text. Step 3.4: Input the target language text and call the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text and form a structured information summary. Step 4: Combine the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence to generate the final structured output containing spatial location, original text, corrected text, language type, translated text, confidence, and key product information.

2. The multilingual OCR recognition enhancement method according to claim 1, characterized in that, Step 2 further includes: Step 2.1: Based on the text position coordinates of each text segment, calculate the spatial distance and arrangement direction between adjacent text segments, and cluster the text segments that are spatially adjacent and have the same arrangement direction into candidate text groups; Step 2.2: Based on the spatial relationship of the candidate text groups, determine the reading order between the text groups according to the preset reading direction rules, and sort the text segments within the group according to the arrangement direction; Step 2.3: Segment the sorted text fragments according to the reading order, identify paragraph boundaries based on the blank areas or separators in the spatial layout, insert paragraph separators into the splicing result, and form text blocks with logical structure.

3. The multilingual OCR recognition enhancement method according to claim 1, characterized in that, The first prompt word template is configured to include a predefined list of language types and a description of the linguistic features of each language, as well as instructions to guide the large language model to perform language recognition based on character features and contextual semantics, for executing step 3.1; The second prompt word template is configured to include text correction instructions and dynamically fill in the language type of each text fragment, the original text block, and the recognition confidence of each fragment when it is called. It is used to guide the large language model to perform selective text correction based on language type guidance and confidence weight, and to execute step 3.

2. The third prompt word template is configured to contain cross-language translation instructions and dynamically fill in the complete corrected text block when called, which is used to guide the large language model to maintain the semantic integrity and terminology consistency of the original text during translation, and to execute step 3.3; The fourth prompt word template is configured to include predefined product information categories and their semantic features, as well as key information extraction instructions. When called, the target language text is dynamically filled in to guide the large language model to identify and extract key product information from the target language text, and to execute step 3.

4.

4. The multilingual OCR recognition enhancement method according to claim 1, characterized in that, Step 3.2 specifically involves: Step 3.2.1: Based on the comparison result between the recognition confidence and the preset threshold, text segments with recognition confidence lower than the preset threshold are marked as regions to be corrected, and text segments with recognition confidence higher than or equal to the preset threshold are marked as regions to be retained. Step 3.2.2: For each region to be corrected, based on its language type and the contextual semantics of the text block it belongs to, call the large language model to generate multiple candidate correction texts, and obtain the generation probability of each candidate correction text from the output of the large language model. Step 3.2.3: Replace the corresponding region to be corrected with each candidate text and combine it with the surrounding preserved region to form multiple candidate complete text blocks. Re-input each candidate complete text block into the large language model to obtain the overall semantic coherence score of each candidate complete text block from the large language model. Step 3.2.4: Based on the generation probability of each candidate correction text and the semantic coherence score of the corresponding candidate complete text block, calculate the weighted comprehensive score, and select the candidate correction text with the highest score as the correction result. Step 3.2.5: The product of the generation probability of the selected candidate corrected text and the semantic coherence score of the corresponding candidate complete text block is used as the correction confidence of the text segment. The numerical range of the correction confidence is between 0 and 1.

5. The multilingual OCR recognition enhancement method according to claim 4, characterized in that, Step 3.4 specifically involves: Step 3.4.1: Predefine the product information categories to be extracted and their semantic features. The product information categories include at least one of the following: product name, brand, price, specifications, production date, shelf life, and country of origin. Step 3.4.2: Based on the target language text, named entity recognition technology is used to identify text fragments that conform to the semantic features of each information category as candidate entities, where: Text fragments that match the semantic features of product names are identified as product name entities; Text fragments that match the semantic features of the brand identity are identified as brand entities; Text fragments that conform to the semantic features of price values ​​are identified as price entities, which include combinations of currency symbols and numbers; Text fragments that conform to the semantic features of specification parameters are identified as specification entities; Text fragments that match the semantic features of production date are identified as production date entities; Text fragments that match the semantic features of shelf life are identified as shelf life entities; Text fragments that match the semantic features of country of origin are identified as country of origin entities; And obtain the recognition confidence of each candidate entity from the output of the large language model; Step 3.4.3: Record the start and end character positions of each candidate entity in the target language translated text, and generate an entity character position index; Step 3.4.4: Organize each candidate entity and its recognition confidence score, as well as the entity character position index corresponding to each candidate entity, according to a predefined JSON data structure to generate a structured information digest containing entity type, entity text, recognition confidence score, and entity character position index.

6. The multilingual OCR recognition enhancement method according to claim 5, characterized in that, Step 4 further includes: Step 4.1: Based on the reading order in Step 2, create a line index for the original text and record the original text content, text position coordinates, and recognition confidence of all text fragments contained in each line; Step 4.2: Restore the corrected text blocks to line-by-line text according to the reading order in Step 2, and assign the same line index as in Step 4.1 to each line of corrected text; Step 4.3: Assign the language types generated in Step 3.1 to the corresponding rows according to the reading order determined in Step 2, and assign the same row index as in Step 4.1 to each row of language types; Step 4.4: Reconstruct the target language text into line-by-line translation text according to the reading order in Step 2, and assign the same line index as in Step 4.1 to each line of translation text; Step 4.5: Based on the same line index, associate the text position coordinates with the corrected text with the same line index in Step 4.2, the language type with the same line index in Step 4.3, and the translated text with the same line index in Step 4.4 to generate a line-level mapping table containing the line index, original text, corrected text, language type, translated text, and text position coordinates. Step 4.6: For each text segment in the original text in Step 4.1, the corresponding recognition confidence and the correction confidence generated in Step 3.2.5 are weighted and fused. The weights are adjusted according to preset rules. The correction confidence weight of the region to be corrected is greater than the original recognition confidence, and the original recognition confidence weight of the region to be retained is greater than the correction confidence. A comprehensive confidence score for each text segment is generated. Step 4.7: Traverse each candidate entity in the structured information digest generated in step 3.4.4, and according to its corresponding entity character position index, find the text line to which the character position index belongs in the line-level mapping table generated in step 4.5, and assign the text position coordinates of the text line to the candidate entity. Step 4.8: For candidate entities whose character position index spans multiple text lines, merge the text position coordinates of the text line where the starting character position is located with the text position coordinates of the text line where the ending character position is located to generate the complete spatial location region of the candidate entity. Step 4.9: Organize the entity type, entity text, recognition confidence and corresponding text location coordinates of each candidate entity obtained after processing in Steps 4.7 and 4.8 according to the predefined JSON data structure to generate an enhanced structured information digest containing spatial location information, which serves as the key product information field in the final structured output.

7. A multilingual OCR recognition enhancement device, characterized in that, include: The OCR recognition module uses the EasyOCR engine to perform multilingual text recognition on the input image, obtain the original text, and generate corresponding text location coordinates and recognition confidence for each text segment; The text preprocessing module determines the spatial distribution relationship and reading order of each text segment based on the text position coordinates, and reassembles the discrete text segments into text blocks with logical coherence. The semantic enhancement module inputs the text block and its corresponding recognition confidence into a pre-configured large language model, and sequentially calls four preset stage prompt word templates to drive the large language model to perform enhancement processing, wherein: The first template bar unit calls the first prompt word template, which contains language type definition and language recognition instructions. It is used to guide the large language model to identify the language type of each text segment based on the character features and contextual semantics of the text segment, and to generate a comprehensive confidence score for language recognition based on the recognition confidence score. The second template bar unit takes the language type and overall confidence level as input along with the original text block, and calls the second prompt word template. The second prompt word template contains text correction instructions, which are used to guide the large language model to correct the segments in the original text with a recognition confidence level lower than a preset threshold, based on the language type and combined with the contextual semantics of the text block. At the same time, it retains the original text of segments with a recognition confidence level higher than or equal to the preset threshold. The corrected segments and the retained segments are combined in the original order to generate a complete corrected text block and its corresponding corrected confidence level. The third template bar unit inputs the complete corrected text block of the output and calls the third prompt word template. The third prompt word template contains cross-language translation instructions, which are used to guide the large language model to perform cross-language translation based on the semantic coherence of the text block and generate a unified target language text. The fourth template unit inputs the target language text and calls the fourth prompt word template. The fourth prompt word template contains predefined product information category features and key information extraction instructions, which are used to guide the large language model to identify and extract key product information from the target language text to form a structured information summary. The result fusion module associates and fuses the corrected text block, language type, target language text, and structured information summary with the text location coordinates and recognition confidence level to generate a final structured output containing spatial location, original text, corrected text, language type, translated text, confidence level, and key product information.

8. The multilingual OCR recognition enhancement device according to claim 5, characterized in that, The text preprocessing module further includes: The spatial clustering unit calculates the spatial distance and arrangement direction between adjacent text segments based on the text position coordinates of each text segment, and clusters text segments that are spatially adjacent and have the same arrangement direction into candidate text groups; The reading order determination unit determines the reading order between the text groups according to the spatial relationship of the candidate text groups and according to the preset reading direction rules, and sorts the text segments within the group according to the arrangement direction. The logical reorganization unit splices the sorted text fragments according to the reading order, identifies paragraph boundaries based on the blank areas or separator lines in the spatial layout, inserts paragraph separator marks into the splicing result, and forms text blocks with logical structure.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images