A method and system for Chinese character image recognition based on a multi-modal large model

Abstract

The application discloses a Chinese character image recognition method and system based on a multimodal large model, relates to the technical field of image processing, and comprises the following steps: acquiring a first prediction result set; counting the character label frequency in the first prediction result set, determining the character label with the highest frequency as a candidate recognition result, and generating a recognition confidence according to the appearance frequency of the candidate recognition result in the first prediction result set; if the recognition confidence is greater than or equal to a confidence threshold, the candidate recognition result is determined as a Chinese character image recognition result; otherwise, in an enhanced inference process, the enhanced inference object is subjected to multiple image transformation processing, and the processed images are respectively input into a multimodal visual language model for inference to obtain a second prediction result set; the character labels in the second prediction result set are fused, and the fused character labels are determined as the Chinese character image recognition result; and the application is favorable for reducing long-tail Chinese character recognition fluctuation and improving recognition reliability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for Chinese character image recognition based on a multimodal large model. Background Technology

[0002] Chinese character image recognition technology is widely used in scenarios such as digitizing paper archives, entering handwritten forms, organizing historical documents, and interacting with smart terminals. Existing recognition methods mostly rely on training visual recognition models using labeled Chinese character image samples, extracting stroke outlines, character structure, and local textures to determine character categories. Under conditions of relatively balanced standard data distribution and stable image quality, these methods can meet the needs of routine Chinese character recognition.

[0003] However, the actual Chinese character images collected are often affected by writing habits, scanning quality, and character sample distribution. Insufficient sample numbers for some categories, and the adhesion, breakage, or deformation of strokes in complex characters can weaken the separability of visual features, causing fluctuations in model output between similar characters. Existing methods typically focus on single-shot recognition results, lacking a discrimination mechanism for output stability, and struggle to fully utilize multi-view image representation in challenging images. Therefore, in scenarios involving long-tailed distribution of Chinese character images, the reliability of recognition remains insufficient. Summary of the Invention

[0004] The purpose of this invention is to provide a Chinese character image recognition method and system based on a multimodal large model to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a Chinese character image recognition method based on a multimodal large model, characterized in that it includes:

[0007] An image of a Chinese character to be identified is acquired and input into a multimodal visual language model finely tuned with a training set of Chinese character images for multiple random sampling and inferences to obtain a first set of prediction results; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the training set of Chinese character images;

[0008] The frequency of character labels in the first prediction result set is counted, and the character label with the highest frequency is determined as the candidate recognition result. The recognition confidence is generated based on the frequency of the candidate recognition result in the first prediction result set.

[0009] If the recognition confidence level is greater than or equal to the confidence level threshold, the candidate recognition result is determined as the Chinese character image recognition result; otherwise, the enhanced inference process is triggered, and the Chinese character image to be recognized is used as the enhanced inference object.

[0010] In the augmented inference process, the augmented inference object is subjected to various image transformations, and the processed images are input into the multimodal visual language model for inference to obtain the second prediction result set.

[0011] The character labels in the second prediction result set are merged to obtain the merged character labels, and the merged character labels are determined as the Chinese character image recognition results.

[0012] Secondly, the present invention provides a Chinese character image recognition system based on a multimodal large model, implemented based on the method described above, including:

[0013] The sampling module is used to acquire the image of the Chinese character to be recognized, input it into the multimodal visual language model finely tuned by the Chinese character image training set for multiple random sampling and inference, and obtain the first prediction result set; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the Chinese character image training set;

[0014] The evaluation module is used to count the frequency of character labels in the first prediction result set, determine the character label with the highest frequency as the candidate recognition result, and generate the recognition confidence based on the frequency of the candidate recognition result in the first prediction result set.

[0015] The determination module is used to determine the candidate recognition result as the Chinese character image recognition result if the recognition confidence is greater than or equal to the confidence threshold; otherwise, it triggers the enhanced inference process and uses the Chinese character image to be recognized as the enhanced inference object.

[0016] The enhancement module is used to perform various image transformations on the enhanced inference object in the enhanced inference process, and input the processed images into the multimodal visual language model for inference to obtain a second set of prediction results.

[0017] The fusion module is used to fuse the character labels in the second prediction result set to obtain fused character labels, and to determine the fused character labels as the Chinese character image recognition results.

[0018] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0019] This invention limits a multimodal visual language model finely tuned with a Chinese character image training set to multiple random sampling inferences within the output space of the Chinese character set. It then generates recognition confidence based on the frequency of occurrence of candidate recognition results in the first prediction result set, allowing for a quantifiable assessment of the model's output stability for the same Chinese character image to be recognized. When the recognition confidence is insufficient, the Chinese character image to be recognized undergoes scaling, stroke edge enhancement, and binarization processing. Finally, it fuses these processes based on the second prediction result set to obtain the Chinese character image recognition result. This reduces the reliance of single inference on low-frequency characters, complex glyphs, and low-quality images, thereby improving the reliability of long-tailed Chinese character image recognition. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0021] Figure 1 This is a flowchart illustrating a Chinese character image recognition method based on a multimodal large model according to the present invention.

[0022] Figure 2 This is a schematic diagram of the framework of a Chinese character image recognition system based on a multimodal large model according to the present invention. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more complete and comprehensive, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative illustrations of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of the exemplary embodiments disclosed in this application. However, those skilled in the art will recognize that the technical solutions disclosed in this application can be practiced with one or more specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the disclosure of this application.

[0025] Example 1

[0026] like Figure 1 As shown, this embodiment discloses a Chinese character image recognition method based on a multimodal large model. This method uses the Chinese character image to be recognized as the processing object and a multimodal visual language model fine-tuned from a Chinese character image training set as the recognition model. During recognition, a first prediction result set is first generated based on the same Chinese character image to be recognized. Then, candidate recognition results and recognition confidence are determined based on the frequency of character labels in the first prediction result set. When the recognition confidence reaches the direct output condition, the candidate recognition result is output; when the recognition confidence does not reach the direct output condition, the enhanced inference process is entered, and the Chinese character image recognition result is determined based on the output result of the enhanced inference process. The method specifically includes:

[0027] S101: Obtain the image of the Chinese character to be recognized, input it into the multimodal visual language model finely tuned by the Chinese character image training set for multiple random sampling and inference, and obtain the first prediction result set; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the Chinese character image training set;

[0028] The image of the Chinese character to be recognized is image data containing a single Chinese character. This image data comes from images of handwritten Chinese characters or from single-character images cropped from character regions of document images. Before inputting into the multimodal visual language model, the image of the Chinese character to be recognized is converted into an image format that the visual encoder can receive. The image format includes an image pixel matrix and normalized image data corresponding to the input size of the model image.

[0029] After receiving an image of a Chinese character to be recognized, the multimodal visual language model performs multiple random sampling inferences on the same image. Each random sampling inference uses the image of the Chinese character to be recognized as the image condition input and generates a character label in the output space corresponding to the Chinese character set. Multiple character labels are recorded in the sampling order to form the first prediction result set. The first prediction result set is used to characterize the multiple outputs of the multimodal visual language model for the same Chinese character image and provides a data basis for subsequent frequency statistics.

[0030] The output space is determined by the set of Chinese characters. This set of characters originates from the character labels in the Chinese character image training set. During the recognition phase, the character labels output by the multimodal visual language model belong to this set of Chinese characters, thus limiting the output results to the character range corresponding to the Chinese character image recognition task. This process reduces text output irrelevant to the Chinese character recognition task and ensures that the elements in the first prediction result set have a unified data type.

[0031] The Chinese character image training set includes Chinese character image samples, character labels corresponding to the Chinese character image samples, and a Chinese character set formed by the character labels. The number of samples of the Chinese character image samples after being grouped by the character labels follows a long-tail distribution, and the Chinese character set corresponds to the output space of the multimodal visual language model.

[0032] The Chinese character image training set is a Chinese character image dataset with completed character annotation. Each training sample in the Chinese character image training set includes a Chinese character image sample and a character label corresponding to the Chinese character image sample. The Chinese character image sample is used as the image condition input in the model fine-tuning stage, and the character label is used as the target output in the model fine-tuning stage. The character labels in the training set are deduplicated to form a Chinese character set.

[0033] After the Chinese character image samples are grouped according to the character labels, each character label corresponds to a sample quantity. The sample quantities corresponding to each character label form a category quantity distribution. This category quantity distribution is a long-tail distribution, which indicates that the sample quantities corresponding to different character labels are unbalanced, and there are character labels with sample quantities located at the tail of the distribution. After the multimodal visual language model is fine-tuned by the Chinese character image training set, it can generate character labels within the output space corresponding to the Chinese character set. This setting incorporates the character labels with sample quantities at the tail of the distribution into the model output space, so that the relevant characters can be used as candidate outputs in the recognition stage.

[0034] Taking the to-be-recognized Chinese character image with the real character "Xi" as an example, this character contains complex strokes and multi-layer structures, and there may be connected strokes and local structure compression in the handwritten image. After the to-be-recognized Chinese character image is input into the multimodal visual language model, the model performs multiple random sampling inferences on the same image. If the multiple sampling results are "Xi", "Yi", and "Xi" in sequence, they are recorded in the sampling order to form a first prediction result set. The first prediction result set is not directly used as the final recognition result, but is used for the subsequent generation of candidate recognition results and recognition confidence. This processing converts multiple model outputs of the complex Chinese character image into statistically processable data.

[0035] The multimodal visual language model is a structurally structured multimodal visual language model, and the multimodal visual language model includes a visual encoder , a multi-layer perception fusion module and a language decoder , the visual encoder adopts a based , and contains visual modules, denoted as , for the input image , the visual feature extraction relationship of the visual encoder is:

[0036] ;

[0037] wherein, is the patch embedding function, is the patch embedding vector, and are respectively the visual number and the visual dimension, is the output of the -th visual module.

[0038] The multimodal vision-language model adopts a structure. The multimodal vision-language model includes a visual encoder , a multi-layer perceptron fusion module and a language decoder . The visual encoder is configured to extract visual features from a Chinese character image to be recognized. The multi-layer perceptron fusion module is configured to map the visual features to the dimension of the language decoder. The language decoder is configured to generate a character label according to the injected visual features.

[0039] The visual encoder adopts a -based . For an input image , the patch embedding function divides the input image into patches, and maps the patches to patch embedding vectors . The visual modules sequentially process the output of the previous layer in a hierarchical order. The -th visual module receives and outputs . After processing by visual modules, the stroke edges, local structures, component positions and overall glyph in the Chinese character image to be recognized are converted into a visual feature sequence.

[0040] For a Chinese character image to be recognized whose real character is "Xi", the patch embedding function divides the image into a plurality of patches. Each patch covers a local stroke area or a background area. Patch embedding is performed to form . Subsequently, the visual modules hierarchically extract stroke edges, stroke connection relationships and overall glyph information to obtain a visual feature sequence. The visual feature sequence is used as the input of the multi-layer perceptron fusion module, so that both the local stroke information and the overall structure information of complex Chinese characters participate in character label generation.

[0041] The multi-layer perceptron fusion module adopts a structure, and selects the Intermediate layer The output features, through the corresponding Projection layer These are respectively mapped to the language decoder dimension and injected into the language decoder's first... To the The visual feature injection relationship of the multi-layer perception fusion module is as follows:

[0042] ;

[0043] in, The number of layers to inject visual features into the language decoder, For the injection of the first Visual features of layer language decoders For the visual encoder The output features of the intermediate layer For the first indivual Projection layer; the language decoder Include indivual Module, denoted as It generates text sequences using an autoregressive approach; for those containing category labels Prompt text The hidden state calculation relationship of the language decoder is as follows:

[0044]

[0045]

[0046] ;

[0047] in, For text embedding functions, For text embedding vectors, and These are the text sequence length and the language decoder hidden dimension, respectively. For the first Injecting visual features into the layer language decoder, For the first The textual hidden state of the layer language decoder It is a zero vector.

[0048] The multi-layer sensing fusion module adopts Structure. This structure is selected from the visual encoder. The visual encoder outputs multiple intermediate layer features, and then injects these visual features at different levels into the language decoder. The intermediate layer output features of the visual encoder contain image information at different levels. The intermediate layer output features closer to the input include stroke edges, grayscale variations, and local textures. The intermediate layer output features closer to the output include component structure, spatial relationships, and overall glyphs. Injecting multiple intermediate layer output features into the language decoder allows the language decoder to utilize both local stroke information and overall structural information during character tag generation.

[0049] Specifically, no. The intermediate layer output features Enter the first indivual Projection layer . Projection layer Will Mapped to the language decoder dimension, the injected first... Visual features of layer language decoder This mapping ensures that the output features of the visual encoder correspond dimensionally to the hidden space of the language decoder, guaranteeing that visual information can enter the corresponding layer of the language decoder.

[0050] Language decoder Include indivual The module generates text sequences using an autoregressive approach. Text embedding function. The prompt text Convert to text embedding vector The prompt text Used for fine-tuning the training phase and associated with the target character labels. It should be noted that during the recognition phase, the language decoder receives cue text to instruct on the Chinese character recognition task. The prompt text It does not contain the actual characters in the image of the Chinese character to be recognized. The text embedding vector in the recognition stage is... The subsequent hidden state calculation relationship is the same as the above relationship.

[0051] Language decoder in When calculating the hidden state of the text, the layer is determined based on whether it belongs to the visual feature injection layer. .when hour, Pick .when hour, Take the zero vector. Then, the language decoder is based on... and the hidden state of the previous layer of text Calculate the hidden state of the current layer text This computational relationship allows visual features to participate in the language decoder. Calculation of text hidden states of the layer, so that character label generation is constrained by the visual features of the Chinese character image to be recognized.

[0052] For a Chinese character image to be recognized whose real character is "Xi", The structure injects visual features of different levels into the language decoder. Local stroke features are used to characterize stroke edges and stroke connection relationships, and overall structural features are used to characterize the component layout and glyph proportion of the character. This processing reduces the loss of local stroke information caused by single-layer visual features, and improves the stability of character label generation for Chinese characters with complex structures.

[0053] The multimodal visual language model fine-tuned via the Chinese character image training set takes the Chinese character image as conditional input, takes the corresponding character label sequence as target output, and takes minimizing the following cross-entropy loss function as the fine-tuning objective:

[0054] ;

[0055] wherein, is the Chinese character image, is the target sequence length, is the i-th character in the target sequence, is the character sequence before the i-th character.

[0056] In the fine-tuning stage, the multimodal visual language model takes the Chinese character image as conditional input, and takes the character label sequence corresponding to the Chinese character image as target output. The model is based on the Chinese character image and the character sequence before the -th character , to predict the -th character in the target sequence . By minimizing the cross-entropy loss function , the character labels output by the model converge to the real character labels corresponding to the training samples. After fine-tuning, the multimodal visual language model obtains the image-to-character label generation capability oriented to the Chinese character set. This fine-tuning process makes the model output correspond to the character label system in the Chinese character image training set, and provides a model basis for the generation of the first prediction result set.

[0057] In the fine-tuning training stage, this embodiment also adopts a hierarchical parameter freezing strategy, so as to retain the general representation capability of the multimodal large model while improving the generalization fitting speed for the features of Chinese characters with long-tail distribution. Specifically, during the fine-tuning process, the bottom parameters of the multimodal visual language model are frozen, and the full fine-tuning is performed on the top parameters. The bottom parameters specifically include the first ​​parameters of the vision Transformer module; the high-level parameters include the network parameters of the remaining vision Transformer modules, the multi-layer perception fusion module, and the language decoder.

[0058] Specifically, the performing multiple random sampling inferences to obtain a first prediction result set includes:

[0059] Within the output space corresponding to the Chinese character set, autoregressive decoding is performed on the same Chinese character image to be recognized using a temperature sampling method, to obtain a sampled character label;

[0060] Multiple random sampling inferences are all performed for the same Chinese character image to be recognized. Each random sampling inference is conditioned on the visual features of the Chinese character image to be recognized, and autoregressive decoding is performed in the output space corresponding to the Chinese character set using the temperature sampling method to obtain one sampled character label. The temperature sampling method enables the model to perform random sampling in the probability distribution of character labels, thereby reflecting the output stability of the model for the same Chinese character image to be recognized. If the model has a stable recognition judgment for the Chinese character image to be recognized, the sampled character labels obtained from multiple samplings tend to be the same; if there is a discrepancy in the recognition judgment, the sampled character labels obtained from multiple samplings will differ.

[0061] Matching the sampled character labels with the Chinese character set, and recording the matched sampled character labels according to the sampling order, to obtain a first prediction result set.

[0062] After obtaining the sampled character labels, the sampled character labels are matched with the Chinese character set. Sampled character labels that belong to the Chinese character set are written into the first prediction result set according to the sampling order. Outputs that do not belong to the Chinese character set are not written into the first prediction result set. Through this processing, all elements in the first prediction result set are character labels within the Chinese character set, and subsequent frequency statistics have definite data objects. This processing converts the uncertainty of a single output of the model into statistical data of multiple outputs, providing a reproducible data source for recognition confidence generation.

[0063] For a Chinese character image to be recognized whose real character is "羲", if the sampled character labels obtained through multiple random sampling inferences are sequentially "羲", "義", "羲", and all the above characters belong to the Chinese character set, the first prediction result set is obtained after recording in the sampling order. This set is used for subsequent candidate recognition result determination and recognition confidence generation.

[0064] S102: Count the frequency of character labels in the first prediction result set, determine the character label with the highest occurrence frequency as a candidate recognition result, and generate a recognition confidence according to the occurrence frequency of the candidate recognition result in the first prediction result set;

[0065] After obtaining the first prediction result set, frequency statistics are performed on the character labels in the first prediction result set. For each type of character label appearing in the first prediction result set, its occurrence frequency is counted. The character label with the highest occurrence frequency represents the most concentrated output character label of the multimodal visual language model in multiple random sampling inferences, and is determined as the candidate recognition result.

[0066] After the candidate recognition result is determined, the recognition confidence is generated according to the occurrence frequency of the candidate recognition result in the first prediction result set. The recognition confidence is used to characterize the concentration degree of the output results of multiple random sampling inferences around the candidate recognition result. The higher the proportion of the occurrence frequency of the candidate recognition result in the first prediction result set, the more concentrated the character output of the model for the to-be-recognized Chinese character image. This processing characterizes the recognition reliability by the concentration degree of the sampling output, and provides an objective basis for subsequent direct output or selection of the enhanced inference process.

[0067] For a to-be-recognized Chinese character image whose real character is "Xi", if the first prediction result set is "Xi", "Yi", "Xi", then "Xi" has the highest occurrence frequency, and "Xi" is determined as the candidate recognition result. Subsequently, the recognition confidence is generated based on the ratio of the occurrence frequency of "Xi" in the first prediction result set to the number of character labels in the first prediction result set. This recognition confidence is used to subsequently determine whether to directly determine "Xi" as the Chinese character image recognition result.

[0068] Specifically, the generating the recognition confidence includes:

[0069] Frequency statistics are performed on the character labels in the first prediction result set, and the candidate recognition result is determined according to the following formula:

[0070] ;

[0071] wherein, is the i-th character label in the first prediction result set, is the Chinese character set, is the number of character labels in the first prediction result set, is an indicator function;

[0072] wherein, represents the i-th character label in the first prediction result set. represents the Chinese character set. represents the number of character labels in the first prediction result set. represents the indicator function. When , ; when , . For to The values ​​of the indicator function are summed to obtain the character label. Frequency of occurrence in the first prediction result set. (For the set of Chinese characters) After iterating through the character labels, the character label that appears most frequently is identified as the candidate recognition result. .

[0073] The frequency of occurrence of candidate recognition results in the first prediction result set is statistically analyzed, and the recognition confidence is obtained by taking the proportion of the frequency of occurrence in the number of character tags in the first prediction result set.

[0074] Determine candidate identification results Then, calculate the identification confidence level according to the following formula. :

[0075] ;

[0076] in, For candidate identification results Frequency of occurrence in the first set of prediction results This represents the number of character labels within the first prediction result set. This recognition confidence score is derived from multiple random sampling inference results of the same Chinese character image to be recognized, and is used to characterize the stability of the model's character output for that image.

[0077] S103: If the recognition confidence is greater than or equal to the confidence threshold, the candidate recognition result is determined as the Chinese character image recognition result; otherwise, the enhanced inference process is triggered, and the Chinese character image to be recognized is used as the enhanced inference object.

[0078] After the recognition confidence score is generated, it is compared with a confidence threshold. The confidence threshold is an inference parameter corresponding to the multimodal visual language model. In one implementation, the confidence threshold is determined based on the correspondence between recognition confidence scores and real character labels in validation samples before model deployment and is fixedly invoked during the inference phase. This threshold is used to distinguish whether candidate recognition results have the output stability required for direct output.

[0079] If the recognition confidence score is greater than or equal to the confidence threshold, it indicates that the character labels in the first prediction result set are concentrated in the candidate recognition result. In this case, the candidate recognition result is determined as the Chinese character image recognition result, and this Chinese character image recognition result is output. This direct output path does not execute subsequent augmentation inference processes, thereby reducing repeated processing of stable recognition samples.

[0080] If the recognition confidence is less than the confidence threshold, it indicates that the character label distribution in the first prediction result set has a dispersed output, and the candidate recognition result does not meet the direct output condition. At this time, the enhanced inference process is triggered, and the Chinese character image to be recognized is taken as the enhanced inference object. The enhanced inference object still corresponds to the same recognition task, and its output space is still determined by the Chinese character set. This processing selects a recognition path according to the output stability of multiple samplings, so that stable samples are directly output, and unstable samples enter enhanced inference.

[0081] For a Chinese character image to be recognized whose real character is "Xi", if the candidate recognition result is "Xi" and the recognition confidence is greater than or equal to the confidence threshold, "Xi" is determined as the recognition result of the Chinese character image. If the recognition confidence is less than the confidence threshold, the image does not directly output the candidate recognition result, but enters the enhanced inference process as an enhanced inference object.

[0082] S104: In the enhanced inference process, perform various image transformation processes on the enhanced inference object, and respectively input the processed images into the multimodal visual language model for inference to obtain a second prediction result set;

[0083] The enhanced inference process is executed when the recognition confidence is less than the confidence threshold. The enhanced inference object is the aforementioned Chinese character image to be recognized. Various image transformation processes are performed on the enhanced inference object to obtain images of different visual expression forms. Subsequently, the processed images are respectively input into the same multimodal visual language model for inference to obtain a second prediction result set. All character labels in the second prediction result set belong to the Chinese character set.

[0084] The data sources of the first prediction result set and the second prediction result set are different. The first prediction result set is derived from multiple random sampling inferences of the same original image. The second prediction result set is derived from the inference results of different image expression forms in the enhanced inference process. This processing introduces multiple visual expression forms when the output stability of the original image is insufficient, so as to provide supplementary recognition basis for final character fusion.

[0085] Specifically, the obtaining a second prediction result set in the enhanced inference process includes:

[0086] Performing scale transformation on the enhanced inference object to obtain a scale-transformed image;

[0087] Performing scale transformation on the enhanced inference object to obtain a scale-transformed image. The scale transformation is specifically a scale amplification process. The scale-transformed image retains the main Chinese character body in the enhanced inference object, and changes the spatial scale of the main Chinese character body in the model input image. The scale transformation enables Chinese character strokes to form a new local expression in the image input, which is beneficial for the visual encoder to extract stroke connection relationships and overall structural features at different scales.

[0088] For a to-be-recognized Chinese character image whose real character is "羲", the lower strokes in the original image may have joined writing and compression. After scaling transformation, the relevant stroke regions form different spatial distributions in the model input, which helps the model re-extract local structures.

[0089] Perform stroke edge enhancement on the enhanced inference object and the scaled transformed image as processing objects to obtain an edge-enhanced image;

[0090] Perform stroke edge enhancement on the enhanced inference object and the scaled transformed image as processing objects to obtain edge-enhanced images. The edge-enhanced images include the edge-enhanced image obtained from the enhanced inference object and the edge-enhanced image obtained from the scaled transformed image. Stroke edge enhancement is used to highlight the boundary between Chinese character strokes and the background, so that the visual encoder can extract stroke contours more focusedly. The edge-enhanced images do not change the category of the to-be-recognized character, and only modify the visual representation of the stroke boundaries in the image.

[0091] For a to-be-recognized Chinese character image whose real character is "羲", the edge-enhanced image can highlight the short strokes in the upper structure and the joined strokes in the lower structure, and reduce the interference of background grayscale variation on visual feature extraction.

[0092] Perform binarization processing on the enhanced inference object and the scaled transformed image to obtain a binarized image;

[0093] Perform binarization processing on the enhanced inference object and the scaled transformed image to obtain binarized images. The binarized images include the binarized image obtained from the enhanced inference object and the binarized image obtained from the scaled transformed image. The binarization processing is used to separate the stroke region and the background region, so that the Chinese character subject in the image has a clear foreground representation. This processing can weaken the influence of paper texture, scanning shadows and background grayscale variation on model recognition.

[0094] For a to-be-recognized Chinese character image whose real character is "羲", the binarized image can separate the complex stroke region from the background, so that the visual encoder can obtain clearer stroke subject information.

[0095] Input the enhanced inference object, the scaled transformed image, the edge-enhanced image and the binarized image into the multimodal visual-language model respectively for inference, so as to obtain a second prediction result set.

[0096] Convert the enhanced inference object, the scaled transformed image, the edge-enhanced image and the binarized image into image input formats that the multimodal visual-language model can receive, and input them into the multimodal visual-language model respectively for inference. Each image input correspondingly generates a character label. The character labels corresponding to each image input are recorded in the processing order of the input images to obtain the second prediction result set.

[0097] wherein the enhanced inference object corresponds to the original visual representation, and the scaled image corresponds to the visual representation after scale change; the edge-enhanced images include the edge-enhanced image obtained from the enhanced inference object and the edge-enhanced image obtained from the scaled image; the binarized images include the binarized image obtained from the enhanced inference object and the binarized image obtained from the scaled image. All the aforementioned images are inferred using the same multimodal visual-language model fine-tuned on a Chinese character image training set, and the output space is a set of Chinese characters. Therefore, the second prediction result set has a unified character label range.

[0098] For a to-be-recognized Chinese character image whose real character is "羲", if the inference results corresponding to the enhanced inference object, the scaled image, the edge-enhanced image obtained from the enhanced inference object, the edge-enhanced image obtained from the scaled image, the binarized image obtained from the enhanced inference object, and the binarized image obtained from the scaled image are sequentially "羲", "羲", "羲", "羲", "羲", "義", the second prediction result set is obtained after recording in the processing order of the input images. This set is used for determining the subsequent fused character label.

[0099] S105: In the enhanced inference process, the character labels in the second prediction result set are fused to obtain a fused character label, and the fused character label is determined as the Chinese character image recognition result.

[0100] After the second prediction result set is obtained, the character labels in the second prediction result set are fused. The fusion process is based on the frequency statistics of the character labels. For each type of character label appearing in the second prediction result set, its occurrence frequency is counted. The character label with the highest occurrence frequency is determined as the fused character label. Subsequently, the fused character label is determined as the Chinese character image recognition result under the enhanced inference process.

[0101] The character labels in the second prediction result set are respectively derived from the enhanced inference object, the scaled image, the edge-enhanced image and the binarized image. The fusion process does not rely on the inference result of any single image input, but determines the final character based on the statistical results of character outputs under multiple image representation forms. This processing can reduce the recognition deviation caused by a single image representation form, and improve the final output reliability of low-stability samples.

[0102] Specifically, the step of fusing the character labels in the second prediction result set in the enhanced inference process and determining the fused character label as the Chinese character image recognition result comprises:

[0103] Performing frequency statistics on the character labels in the second prediction result set, and determining the fused character label according to the following formula:

[0104] ;

[0105] wherein, is a Chinese character set, is the th character label in the second prediction result set, is the number of character labels in the second prediction result set, is an indicator function;

[0106] wherein, represents the th character label in the second prediction result set. represents the Chinese character set. represents the number of character labels in the second prediction result set. represents the indicator function. When , . When , . Sum the values of the indicator function from to to obtain the occurrence frequency of the character label in the second prediction result set. After traversing the character labels in the Chinese character set , the character label with the maximum occurrence frequency is determined as the fused character label .

[0107] For a to-be-recognized Chinese character image with the real character being "Xi", if the second prediction result set is "Xi", "Xi", "Xi", "Xi", "Xi", "Yi", then "Xi" has the highest occurrence frequency, and the fused character label is determined as "Xi".

[0108] The fused character label is determined as the Chinese character image recognition result under the enhanced inference process.

[0109] The fused character label is taken as the Chinese character image recognition result under the enhanced inference process, and the Chinese character image recognition result is output. If the to-be-recognized Chinese character image has met the direct output condition in the recognition confidence judgment, the Chinese character image recognition result is determined by the candidate recognition result. If the to-be-recognized Chinese character image enters the enhanced inference process, the Chinese character image recognition result is determined by the fused character label. Both paths output character labels belonging to the Chinese character set, ensuring that the output result corresponds to the Chinese character image recognition task.

[0110] Embodiment 2

[0111] As Figure 2 shown, the parts not detailed in this embodiment are as described in Embodiment 1. This embodiment discloses a Chinese character image recognition system based on a multimodal large model, comprising:

[0112] The sampling module 201 is used to acquire the image of the Chinese character to be recognized, input it into the multimodal visual language model finely tuned by the Chinese character image training set for multiple random sampling and inference, and obtain the first prediction result set; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the Chinese character image training set;

[0113] Evaluation module 202 is used to count the frequency of character labels in the first prediction result set, determine the character label with the highest frequency as the candidate recognition result, and generate recognition confidence based on the frequency of the candidate recognition result in the first prediction result set.

[0114] The determination module 203 is used to determine the candidate recognition result as the Chinese character image recognition result if the recognition confidence is greater than or equal to the confidence threshold; otherwise, it triggers the enhanced inference process and uses the Chinese character image to be recognized as the enhanced inference object.

[0115] Enhancement module 204 is used to perform various image transformation processes on the enhanced inference object in the enhanced inference process, and input the processed images into the multimodal visual language model for inference to obtain a second prediction result set;

[0116] The fusion module 205 is used to fuse the character labels in the second prediction result set to obtain fused character labels, and to determine the fused character labels as Chinese character image recognition results.

[0117] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A Chinese character image recognition method based on a multimodal large model, characterized in that, include: An image of a Chinese character to be identified is acquired and input into a multimodal visual language model finely tuned with a training set of Chinese character images for multiple random sampling and inferences to obtain a first set of prediction results; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the training set of Chinese character images; The frequency of character labels in the first prediction result set is counted, and the character label with the highest frequency is determined as the candidate recognition result. The recognition confidence is generated based on the frequency of the candidate recognition result in the first prediction result set. If the recognition confidence level is greater than or equal to the confidence level threshold, the candidate recognition result is determined as the Chinese character image recognition result; otherwise, the enhanced inference process is triggered, and the Chinese character image to be recognized is used as the enhanced inference object. In the augmented inference process, the augmented inference object is subjected to various image transformations, and the processed images are input into the multimodal visual language model for inference to obtain the second prediction result set. The character labels in the second prediction result set are merged to obtain the merged character labels, and the merged character labels are determined as the Chinese character image recognition results.

2. The method according to claim 1, characterized in that, The Chinese character image training set includes Chinese character image samples, character labels corresponding to the Chinese character image samples, and a set of Chinese characters formed by the character labels. The number of Chinese character image samples after being grouped according to the character labels is distributed in a long tail. The set of Chinese characters corresponds to the output space of the multimodal visual language model.

3. The method according to claim 1, characterized in that, The multimodal visual language model is a Qwen3-VL-2B structured multimodal visual language model, which includes a visual encoder. Multi-layer perception fusion module and language decoder The visual encoder employs a SigLIP-2-based VisionTransformer and includes... A visual Transformer module, denoted as... For the input image The visual feature extraction relationship of the visual encoder is as follows: ； in, For image patch embedding functions, Embedded vectors for image patches, and These are the number of visual tokens and the dimensions of the visual tokens, respectively. For the first The output of a visual Transformer module.

4. The method according to claim 3, characterized in that, The multi-layer perception fusion module adopts a DeepStack structure, selecting from the visual encoder. Intermediate layer The output features are processed through the corresponding MLP projection layer. These are respectively mapped to the language decoder dimension and injected into the language decoder's first... To the The visual feature injection relationship of the multi-layer perception fusion module is as follows: ； in, The number of layers to inject visual features into the language decoder, For the injection of the first Visual features of layer language decoders For the visual encoder The output features of the intermediate layer For the first One MLP projection layer; The language decoder Include There are one Transformer module, denoted as... The text sequence is generated using an autoregressive method; the prompt text in the recognition stage is a prompt text used to instruct on the Chinese character recognition task and does not contain the actual characters in the image of the Chinese character to be recognized. The hidden state calculation relationship of the language decoder is as follows: ； ； ； in, For text embedding functions, For text embedding vectors, and These are the text sequence length and the language decoder hidden dimension, respectively. For the first Injecting visual features into the layer language decoder, For the first The textual hidden state of the layer language decoder It is a zero vector.

5. The method according to claim 1, characterized in that, The multimodal visual language model, fine-tuned using a Chinese character image training set, takes Chinese character images as input, corresponding character label sequences as output, and minimizes the following cross-entropy loss function as its fine-tuning objective: ； in, Images of Chinese characters. The length of the target sequence. For the first in the target sequence One character, For the first The character sequence preceding the given character.

6. The method according to claim 1, characterized in that, The first prediction result set obtained by performing multiple random sampling inferences includes: Within the output space corresponding to the Chinese character set, temperature sampling is used to perform autoregressive decoding on the same Chinese character image to be recognized, and the sampled character label is obtained. The sampled character labels are matched with the set of Chinese characters, and the matched sampled character labels are recorded according to the sampling order to obtain the first prediction result set.

7. The method according to claim 1, characterized in that, The generation of recognition confidence includes: Frequency statistics are performed on the character labels in the first prediction result set, and candidate recognition results are determined according to the following formula: ； in, The first prediction result set Character label, A set of Chinese characters. The number of character labels in the first prediction result set. For indicator functions; The frequency of candidate identification results in the first prediction result set is counted, and the identification confidence is obtained according to the following formula: ； in, To identify confidence levels, For candidate identification results Frequency of occurrence in the first set of prediction results This represents the number of character labels within the first prediction result set.

8. The method according to claim 1, characterized in that, The second set of prediction results obtained in the enhanced inference process includes: A scaling transformation is performed on the enhanced inference object to obtain a scaled image; Stroke edge enhancement is performed on the enhanced inference object and the scale-transformed image to obtain an edge-enhanced image; Binarization is performed on the enhanced inference object and the scale-transformed image to obtain a binary image; The enhanced inference object, the scale-transformed image, the edge-enhanced image obtained from the enhanced inference object, the edge-enhanced image obtained from the scale-transformed image, the binarized image obtained from the enhanced inference object, and the binarized image obtained from the scale-transformed image are respectively input into the multimodal visual language model for inference to obtain the second prediction result set.

9. The method according to claim 1, characterized in that, The step of fusing character labels from the second prediction result set in the enhanced inference process and determining the fused character labels as Chinese character image recognition results includes: Frequency statistics are performed on the character labels in the second prediction result set, and the fused character labels are determined according to the following formula: ； in, A set of Chinese characters. For the second set of prediction results, the first Character label, The number of character labels in the second prediction result set. For indicator functions; The fused character labels are identified as the Chinese character image recognition results under the enhanced inference process.

10. A Chinese character image recognition system based on a multimodal large model, implemented according to the method of any one of claims 1-9, characterized in that, include: The sampling module is used to acquire the image of the Chinese character to be recognized, input it into the multimodal visual language model finely tuned by the Chinese character image training set for multiple random sampling and inference, and obtain the first prediction result set; wherein, the output space of the multimodal visual language model is determined by the set of Chinese characters corresponding to the Chinese character image training set; The evaluation module is used to count the frequency of character labels in the first prediction result set, determine the character label with the highest frequency as the candidate recognition result, and generate the recognition confidence based on the frequency of the candidate recognition result in the first prediction result set. The determination module is used to determine the candidate recognition result as the Chinese character image recognition result if the recognition confidence is greater than or equal to the confidence threshold; otherwise, it triggers the enhanced inference process and uses the Chinese character image to be recognized as the enhanced inference object. The enhancement module is used to perform various image transformations on the enhanced inference object in the enhanced inference process, and input the processed images into the multimodal visual language model for inference to obtain a second set of prediction results. The fusion module is used to fuse the character labels in the second prediction result set to obtain fused character labels, and to determine the fused character labels as the Chinese character image recognition results.

Images