A Multi-Engine Game Theory-Based Intelligent Recognition and Annotation Method and System for Handwritten Remittances from Overseas Chinese

Abstract

This invention discloses a multi-engine game-based intelligent recognition and annotation method and system for handwritten characters in overseas Chinese remittance letters. The method includes: extracting and fusing the original image features, ink saliency features, and edge features of the area to be annotated, and inputting them into a deep learning segmentation model to generate a binary mask; using the mask for visual purification OCR recognition to obtain a first result and confidence level, while simultaneously extracting an elastic visual fingerprint and matching it with a memory database to obtain a second result and similarity score; performing game-based decision-making based on the confidence level and similarity score, and outputting the final recognition suggestion; correcting the mask through user interaction, updating the confirmed characters and fingerprint to the memory database, and incrementally training the segmentation model online using the corrected samples. This invention combines multi-dimensional features, dual-path recognition game theory, and online evolution to effectively improve the accuracy and efficiency of handwritten character recognition and annotation in overseas Chinese remittance letters under complex backgrounds.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a method and system for automatic recognition, pixel-level segmentation, and intelligent auxiliary annotation of handwritten characters in ancient documents (especially overseas remittance archives) with complex backgrounds and illegible handwriting. Background Technology

[0002] Overseas Chinese remittance letters (letters and remittance receipts sent back to China by overseas Chinese) are listed in the "Memory of the World Register" and have extremely high historical and cultural value. However, these remittance archives generally suffer from problems such as aging paper, severe creases, interference from seals, ink bleeding, and extremely illegible handwriting. Traditional OCR (Optical Character Recognition) technology has extremely low accuracy when processing such complex background images.

[0003] Currently, the digital processing of overseas Chinese remittances mainly relies on manual annotation, which has the following drawbacks: 1. Low efficiency, as manually recognizing illegible handwriting is extremely time-consuming; 2. Annotation accuracy is difficult to achieve at the pixel level, which is detrimental to the training of subsequent deep learning models; 3. Lack of self-evolution capability, as the annotated character features cannot be effectively fed back to the recognition engine. Therefore, developing an intelligent annotation system that can integrate multiple recognition engines, possess visual cleansing capabilities, and learn online is of significant practical importance. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a multi-engine game theory-based intelligent recognition and annotation method for handwritten letters from overseas Chinese, which achieves high-precision handwritten character recognition through multi-dimensional feature extraction and pixel-level segmentation, visual purification OCR recognition, flexible visual fingerprint memory matching and game decision-making mechanism; at the same time, through manual interactive correction and online incremental training, the segmentation model and memory bank are continuously evolved, and the system performance is gradually improved.

[0005] To address the aforementioned technical problems, this invention provides a method for intelligent recognition and annotation of handwritten remittance documents based on multi-engine game theory, comprising the following steps: Step S1: Multidimensional feature extraction and segmentation. Obtain the image region to be labeled, extract the original image features, ink saliency features, and edge features, input them into a preset deep learning segmentation model, and generate a pixel-level binary mask. The deep learning segmentation model is a lightweight pixel-level convolutional segmentation model for the handwritten handwriting region of overseas Chinese remittances. Its input is a five-channel fusion feature, and its output is a single-channel probability map, which is thresholded to generate a binary mask. The five-channel fusion feature is formed by splicing three-channel original image features, single-channel ink saliency features, and single-channel edge features in the channel dimension. Step S2: Multi-path parallel recognition processing, including a visual purification recognition path and an elastic memory retrieval path; in the visual purification recognition path, the background of the original image is removed using the binarized mask to generate a clean handwriting image and input it into the OCR engine to obtain the first recognition result and its confidence level; in the elastic memory retrieval path, the elastic visual fingerprint of the binarized mask is extracted, and similarity matching is performed in a preset memory database to obtain the second recognition result and its similarity score; Step S3: Game decision determination, combining the confidence of the first identification result and the similarity score of the second identification result, automatically filter and output the final identification suggestion according to the preset game weight rules; Step S4: Interactive correction and online evolution. In response to the user's confirmation or correction of the final recognition suggestion, obtain the real character label and generate the corrected final binarized mask; generate a flexible visual fingerprint based on the final binarized mask and update the memory index; write the corrected sample into the training sample library and incrementally train the deep learning segmentation model online.

[0006] Furthermore, the multidimensional feature extraction in step S1 specifically involves: The original image is converted to HSV space and ink saliency maps are extracted within a specific threshold range; Edge feature maps of the original image are extracted using the Canny operator; The original image, ink saliency map, and edge feature map are concatenated to form a five-dimensional tensor, which is then input into the deep learning segmentation model.

[0007] Furthermore, the visual purification recognition path in step S2 specifically includes: Using the binarized mask as a mask, extract the handwriting pixels from the original image; Create a pure white background image with the same size as the original image, and overlay the extracted handwriting pixels onto the pure white background image to generate a clean image that eliminates background noise, creases, and stamp interference. The purified image is then input into the OCR engine.

[0008] Furthermore, the process of extracting the elastic visual fingerprint in step S2 includes: The binarized mask is subjected to morphological closing operations to connect broken strokes; Calculate the minimum bounding rectangle of the connected region and align its centroids, then normalize and scale the handwriting region to the center of the canvas of a preset size. The normalized image is subjected to Gaussian blurring to enhance translation tolerance, and the corresponding hash feature value is generated as a visual fingerprint.

[0009] Furthermore, the game weighting rules in step S3 are specifically as follows: If the similarity score of the second identification result is higher than the first preset threshold, and the confidence level of the first identification result is lower than the second preset threshold, then the second identification result is determined to be the final identification suggestion. If the confidence level of the first identification result is higher than the third preset threshold, then the first identification result is determined to be the final identification suggestion.

[0010] Furthermore, the online incremental training in step S4 specifically includes: Use the user-corrected binarized mask as the ground truth label and the corresponding original image slice as the training sample. In the background process of annotation interaction, the weight parameters of the deep learning segmentation model are updated in real time using the stochastic gradient descent algorithm.

[0011] Furthermore, the similarity score of the second identification result is calculated according to the normalized correlation matching method: the similarity score S is equal to the sum of the pixel products of the current elastic visual fingerprint image A and the historical elastic visual fingerprint image B, divided by the square root of the product of the sum of squares of the pixels of A and the sum of squares of the pixels of B.

[0012] Furthermore, the synchronous update process of the visual fingerprint index in step S4 includes: The final character label is obtained after user confirmation or correction; Generate a flexible visual fingerprint image and fingerprint identification value based on the corresponding final binarized mask; The fingerprint identifier value, visual fingerprint image, and final character label are written into the memory; if the same fingerprint identifier value does not exist in the memory, a new record is added; if it already exists, the corresponding record is updated.

[0013] Accordingly, embodiments of the present invention also provide an intelligent recognition and annotation system for handwritten remittances based on multi-engine game theory, comprising: The image processing module is used to acquire image slices and extract multidimensional features; The pixel segmentation module is used to run deep learning models and generate binarized masks; The multi-channel recognition module includes a visual purification OCR unit and a flexible fingerprint retrieval unit; The game-theoretic decision-making module is used to select the best outcome based on confidence and similarity. The online learning module is used for memory updates and real-time evolution of model weights.

[0014] Accordingly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the above-described method.

[0015] Implementing the embodiments of the present invention has the following beneficial effects: 1. Strong anti-interference ability: Through the visual purification technology of "segmentation first, recognition later", the interference of seals and creases on OCR is effectively solved.

[0016] 2. High recognition accuracy: The introduction of flexible fingerprint matching solves the problem of recognizing extremely messy and non-standard fonts.

[0017] 3. Increased annotation efficiency: Automatically generated pixel-level masks reduce the workload of manual painting, and multi-engine suggestions reduce the cost of manual data entry.

[0018] 4. System self-evolution: As the number of annotations increases, the system's recognition rate for specific handwriting styles will improve accordingly. Attached Figure Description

[0019] Figure 1 This is the overall logic flowchart of the present invention; Figure 2 This is a system structure block diagram of the present invention; Figure 3 This is a flowchart of multidimensional feature extraction and pixel segmentation; Figure 4 This is a flowchart of dual-path recognition and game-theoretic decision-making; Figure 5 It is a flowchart of interactive correction and online evolution. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0021] In implementing the multi-engine game theory-based intelligent recognition and annotation method for handwritten remittances from overseas Chinese, this invention combines... Figure 1-5 As shown, follow steps S1 to S4.

[0022] Step S1: Multidimensional Feature Extraction and Segmentation The system acquires the image region to be labeled, which can be a phrase or single character region selected by the user. Based on the selected coordinates, the system crops an image block from the original remittance document image and scales it to a preset size, such as 128×128 pixels. Three types of features are extracted: the first type is the original color image features, where pixel values ​​are normalized to between 0 and 1, preserving the paper background, ink color, and texture information; the second type is ink saliency features, where the image is converted to the HSV color space, dark pixels with a brightness (V) channel in the range of 0 to 100 and a full range of H and S are selected to generate a single-channel saliency map and normalize it; the third type is edge features, where the image is converted to grayscale, and edge detection operators are used to extract stroke edges and contours to obtain a single-channel edge map. These three types of features are concatenated into a five-channel input and fed into a deep learning segmentation model.

[0023] The preset deep learning segmentation model is a lightweight pixel-level convolutional segmentation model for handwritten remittance regions, used to determine whether each pixel in the image region belongs to handwritten ink. This model is not a text recognition model, but a handwriting segmentation model before text recognition and manual annotation. Its output is a single-channel probability map, which is then thresholded to generate a binary mask.

[0024] The model is pre-designed as a multi-input feature fusion model. The input includes three types of features: the first type is the original color image features of the image region to be labeled; the second type is the ink saliency features extracted based on color and brightness information; and the third type is the edge features extracted based on grayscale changes. The three types of features are concatenated along the channel dimension to form a five-channel input, where the original color image occupies three channels, the ink saliency map occupies one channel, and the edge map occupies one channel.

[0025] In its implementation, the deep learning segmentation model can be pre-defined as a fully convolutional neural network structure: First, a first convolutional layer receives five-channel input and extracts low-level handwriting texture features; then, a second convolutional layer further extracts stroke shape, local connectivity structure, and edge response features; finally, a one-channel convolutional layer outputs the probability that each pixel belongs to handwritten ink, and the output is limited to between zero and one by a sigmoid activation function. Pixels with an output probability greater than a preset threshold are identified as ink pixels; pixels with an output probability less than or equal to the preset threshold are identified as background pixels, thereby generating a binarized mask.

[0026] Specifically, first, the target region in the image to be labeled is obtained. This target region can be a phrase region selected by the user, or a single character region obtained through further selection. The system then crops the corresponding image block from the original remittance letter image based on the selected coordinates and records the original width and height of the image block.

[0027] Then, the cropped image patch is scaled to a preset size, such as 128×128 pixels, as the uniform size for subsequent model input.

[0028] Next, the original image features are extracted. The system retains the color pixel information of the image patch and converts it into a three-channel tensor, normalized to between zero and one, to represent the paper background, ink color, stains, wrinkles, and overall texture information.

[0029] Subsequently, ink saliency features are extracted. The system converts image patches to the HSV color space, filters out darker ink candidate regions based on brightness and color range, and generates a single-channel ink saliency map. This saliency map is used to highlight pixels that may belong to handwritten strokes and suppress paper background, light-colored noise, and non-ink areas.

[0030] Next, edge features are extracted. The system converts image patches into grayscale images and uses edge detection operators to extract stroke edges, break contours, and local structural changes, generating a single-channel edge map. This edge map is used to supplement stroke boundary information that the ink saliency map cannot fully express.

[0031] Then, the original color image features, ink saliency features, and edge features are fused according to the channel dimension to form a five-channel feature input, which is then fed into a pre-defined deep learning segmentation model. The model performs convolution operations on the fused features and outputs the probability value of each pixel belonging to handwritten handwriting.

[0032] Finally, the probability map output by the model is thresholded to generate a pixel-level binary mask. In the mask, the ink pixels are assigned a value of 255, and the background pixels are assigned a value of 0. The system then restores the mask to the original size of the image region to be labeled, for subsequent handwriting cleanup, OCR recognition, manual correction, sample saving, and online learning.

[0033] In summary, the deep learning segmentation model is a pixel-level convolutional segmentation model with five-channel input and one-channel output. Its input is formed by concatenating the three-channel original image of the image region to be labeled, a single-channel ink saliency map, and a single-channel edge map. The model extracts handwriting texture features and edge structure features through convolutional layers and outputs a probability map of each pixel belonging to handwritten ink. The probability map is binarized according to a preset threshold to obtain a binarized mask used to represent the handwritten ink region and the background region.

[0034] The steps for extracting the original image features mentioned above include: The original image features refer to the color pixel features directly retained from the image region to be labeled. The system first crops the image region to be labeled from the original image of the overseas remittance letter based on the coordinates obtained by the user selection or automatic positioning; then it scales the image region to a preset input size; and then extracts its three-channel color pixel values ​​as the original image features.

[0035] To facilitate processing by deep learning models, the system normalizes the color value of each pixel to between 0 and 1, and organizes it into a three-channel feature tensor according to the channel dimension. This original image feature is used to preserve basic visual information such as paper background, ink color, stains, wrinkles, scanning noise, stroke thickness, and local texture, enabling the segmentation model to determine whether a pixel belongs to handwritten handwriting by combining the original image context.

[0036] Summary: Image blocks are cropped from the original overseas remittance images based on the coordinates of the areas to be labeled; the image blocks are scaled to a preset size; the color channel pixel values ​​of the image blocks are read and normalized to obtain three-channel original image features.

[0037] The steps for extracting the salient features of the ink marks mentioned above include: The ink saliency feature is a single-channel feature used to highlight the handwritten ink area and weaken the paper background and non-text interference. The system first converts the image area to be labeled from the original color space to the HSV color space; then, based on the characteristics of ink marks, which are usually low in brightness and dark in color, the HSV image is thresholded, and pixels with brightness values ​​lower than a preset threshold are identified as ink mark candidate pixels.

[0038] Ink saliency maps can be generated as follows: After converting the image to an HSV image, set the color range to H channel 0 to 180, S channel 0 to 255, and V channel 0 to 100 to filter dark pixels; pixels that meet this range are assigned a value of 255, and pixels that do not are assigned a value of 0, thus forming a single-channel ink saliency map. This saliency map is then scaled to the same size as the model input and normalized to a feature map between 0 and 1.

[0039] In summary, the extraction of ink saliency features includes: converting the image region to be labeled to a color representation space; thresholding image pixels based on the characteristic that handwritten ink has lower brightness and darker color compared to the paper background to generate ink candidate regions; assigning the first pixel value to pixels belonging to the ink candidate regions and assigning the second pixel value to other pixels to obtain a single-channel ink saliency map; and performing size normalization and numerical normalization on the ink saliency map as one of the input features of the deep learning segmentation model.

[0040] The binarized mask is generated by thresholding a pixel-level probability map output by a preset deep learning segmentation model, specifically including the following steps: First, the system acquires the image region to be labeled and records its original width and height. The image region to be labeled is then scaled to a preset input size to obtain a standardized image region.

[0041] Secondly, the system extracts the original image features, ink saliency features, and edge features of the standardized image region, and then concatenates these three features according to the channel dimension to form a multi-channel fused feature. Specifically, the original image features are three-channel color image features, the ink saliency features are single-channel ink candidate region features, and the edge features are single-channel stroke contour features.

[0042] Then, the multi-channel fused features are input into a preset deep learning segmentation model. The deep learning segmentation model performs pixel-level classification prediction on the input features and outputs a single-channel probability map. The value of each pixel in the probability map represents the probability that the pixel belongs to the handwritten ink area.

[0043] Next, the system binarizes the probability map according to a preset threshold. Pixels with a probability value greater than the preset threshold are identified as ink blot pixels and assigned a first pixel value in the binarized mask; pixels with a probability value less than or equal to the preset threshold are identified as background pixels and assigned a second pixel value in the binarized mask. In this embodiment, the preset threshold is 0.5, the first pixel value is 255, and the second pixel value is 0.

[0044] Finally, the system resizes the generated binarized mask according to the original width and height of the image region to be labeled, obtaining a pixel-level binarized mask corresponding to the spatial position of the image region to be labeled. In the binarized mask, areas with a pixel value of 255 represent handwritten ink areas, and areas with a pixel value of 0 represent background areas. This binarized mask is used for subsequent handwriting extraction, manual correction, sample saving, and online model learning.

[0045] Step S2: Multi-channel parallel recognition processing Visual purification recognition path: Using a binarized mask as a mask, the corresponding pixels of the ink in the original image are retained and superimposed on a pure white background image to generate a purified image that eliminates background, creases and stamp interference. The purified image is then sent to the OCR engine to obtain the first recognition result and its confidence level C.

[0046] Elastic Memory Retrieval Path: Generating an elastic visual fingerprint based on a binary mask. First, morphological closing operations are performed on the mask using a 3×3 rectangular structuring element, followed by dilation and erosion to connect broken strokes and fill small holes. Then, the minimum bounding rectangle of all ink pixels is extracted, and the local handwriting area is cropped and scaled proportionally to a normalized size (e.g., 100 pixels on the long side) based on the long side. This normalized handwriting image is then centered in a 128×128 standard canvas. A 5×5 Gaussian blur is applied to this image to enhance translation tolerance. Subsequently, all pixel values ​​are read in row-major order to generate hash feature values ​​as the visual fingerprint, and the hash values, character labels, and blurred image are stored in memory.

[0047] The process of extracting flexible visual fingerprints includes: S21: Perform morphological closing operations on the binarized mask to connect broken strokes.

[0048] Morphological closing operations specifically include the following steps: First, the system obtains the binarized mask generated in step S1. In the binarized mask, the ink pixels and background pixels are represented by different pixel values; in this embodiment, the ink pixel value is 255, and the background pixel value is 0.

[0049] Secondly, the system presets a structural element as a computational template for morphological processing. This structural element is a two-dimensional neighborhood matrix of preset size, used to determine the neighborhood range around each pixel to be processed that participates in the computation. In this embodiment, the structural element is a 3×3 rectangular structural element.

[0050] Then, the system performs a dilation operation on the binarized mask. Specifically, it aligns the center of the structuring element with each pixel position in the binarized mask sequentially, and detects whether there are ink pixels within the coverage area of ​​the structuring element. If at least one ink pixel exists within the coverage area, the output pixel at the current center position is set as the ink pixel; if no ink pixel exists within the coverage area, the output pixel at the current center position is set as the background pixel. Through this dilation operation, adjacent or closely spaced broken stroke regions are expanded outwards and connected.

[0051] Next, the system performs an erosion operation on the mask after the dilation operation. Specifically, it aligns the center of the same structuring element with each pixel position in the dilated mask and checks whether all pixels within the coverage area of ​​that structuring element are ink pixels. If all pixels within the coverage area are ink pixels, the output pixel at the current center position is set as an ink pixel; if background pixels exist within the coverage area, the output pixel at the current center position is set as a background pixel. Through this erosion operation, the basic size of the stroke boundaries after dilation is restored, and the outward expansion effect caused by dilation is removed.

[0052] S22: Calculate the minimum bounding rectangle of the connected region and perform centroid alignment, then normalize and scale the handwriting region to the center of the canvas of the preset size.

[0053] First, the system obtains a binarized mask after morphological closing operations, and then retrieves all ink pixels from the binarized mask. The ink pixels refer to pixels with a pixel value of a first pixel value; in this embodiment, the first pixel value is 255.

[0054] Secondly, the system calculates the minimum bounding rectangle of the handwriting region based on the coordinates of all ink pixels. Specifically, it calculates the minimum and maximum values ​​of the horizontal and vertical coordinates of all ink pixels; using the minimum horizontal coordinate as the left boundary of the bounding rectangle, the maximum horizontal coordinate as the right boundary, the minimum vertical coordinate as the upper boundary, and the maximum vertical coordinate as the lower boundary, thus determining the minimum bounding rectangle that can contain all ink pixels.

[0055] Then, the system crops the handwriting region from the binary mask according to the minimum bounding rectangle, obtaining a local handwriting mask. The local handwriting mask only contains the valid handwriting of the current character and its minimum bounding region, thereby eliminating the difference in blank edges caused by different sizes of the user's selection area.

[0056] Next, the system calculates the width and height of the local handwriting mask and determines a proportional scaling factor based on its longer side. Specifically, the system divides the preset normalized handwriting size by the larger of the width and height of the local handwriting mask to obtain the scaling factor; then, the system simultaneously scales the width and height of the local handwriting mask according to this scaling factor, so that its longer side is scaled to the preset normalized handwriting size, while the shorter side is scaled proportionally. This maintains the original aspect ratio of the character handwriting.

[0057] Subsequently, the system establishes a standard canvas of a preset size. The standard canvas is a binary or grayscale image, and its initial pixel values ​​are all set to the background pixel values. In this embodiment, the standard canvas size is 128×128 pixels, the background pixel value is 0, and the preset normalized handwriting size is 100 pixels.

[0058] Finally, the system places the scaled local handwriting mask at the center of the standard canvas. Specifically, it calculates a horizontal offset based on the difference between the width of the standard canvas and the width of the scaled handwriting area, and a vertical offset based on the difference between the height of the standard canvas and the height of the scaled handwriting area. The scaled handwriting area is then copied to the position determined by the horizontal and vertical offsets on the standard canvas, ensuring that the scaled handwriting area is centrally distributed within the standard canvas. This results in a normalized handwriting image with uniform size, position, and proportions, which is used to generate a flexible visual fingerprint.

[0059] S23: Perform Gaussian blur processing on the normalized image to enhance translation tolerance and generate corresponding hash feature values ​​as visual fingerprints.

[0060] First, the system acquires a standard handwriting image that has been normalized, scaled, and centered. The standard handwriting image is a binary or grayscale image of a preset size, wherein the ink area and the background area are represented by different pixel values; in this embodiment, the standard handwriting image size is 128×128 pixels.

[0061] Secondly, the Gaussian convolution kernel is a two-dimensional weight matrix, with the largest weight at the center and smaller weights further away from the center. It is used to perform weighted smoothing on each pixel and its neighboring pixels in the standard handwriting image. In this embodiment, the Gaussian convolution kernel size is 5×5. Then, the system sequentially traverses each pixel position in the standard handwriting image using the Gaussian convolution kernel. For any current pixel position, the system obtains the pixel values ​​within the pixel's neighborhood and performs a weighted sum according to the weights of the corresponding positions in the Gaussian convolution kernel. The weighted sum is used as the smoothed pixel value for that pixel position. This yields the Gaussian-blurred standard handwriting image. Through the above Gaussian blurring process, the local response of the handwriting pixels is diffused to their neighboring regions, ensuring that the current handwriting and historical handwriting maintain high pixel distribution consistency even with slight positional shifts, minor stroke misalignments, or edge jitter, thereby enhancing the translational tolerance of subsequent similarity matching. The system generates hash feature values ​​based on the normalized standard handwriting image. Specifically, the system reads all pixel values ​​of a standard handwriting image according to a preset pixel scanning order. In this embodiment, a 128×128 pixel matrix is ​​read in row-priority order from top to bottom and left to right. The read pixel values ​​are then arranged sequentially to form a fixed-length pixel byte sequence. Subsequently, the system performs a hash operation on the pixel byte sequence to generate a fixed-length hash feature value. The hash feature value corresponds one-to-one with the normalized handwriting pattern of the current character and is used as a visual fingerprint index. If the normalized handwriting images of two character samples are completely identical, the same hash feature value is generated; if the normalized handwriting images are different, different hash feature values ​​are generated. Finally, the system synchronously writes the hash feature value, the corresponding character label, and the Gaussian-blurred standard handwriting image into the memory. The hash feature value is used to establish the visual fingerprint index, and the Gaussian-blurred standard handwriting image is used for subsequent similarity matching with the character to be identified.

[0062] The memory bank meets the following conditions: the memory bank is used to store confirmed historical handwritten character handwriting features. The memory bank includes at least character tags, elastic visual fingerprint images, and fingerprint identification values ​​corresponding to the elastic visual fingerprint images.

[0063] The memory bank can be used for matching when it possesses the following characteristics: First, there must be at least one valid historical record in the memory bank; Second, each valid historical record has a clear character label, which is a character result after manual or system confirmation; Third, each valid historical record has a standardized fingerprint image obtained using the same flexible visual fingerprint generation method; Fourth, the standardized fingerprint images in the memory bank have the same size, the same pixel representation, and the same preprocessing process as the current fingerprint image to be matched; Fifth, the memory bank stores the correspondence between fingerprint identification values ​​and character tags, and can read the corresponding elastic visual fingerprint image based on the fingerprint identification value.

[0064] Only when the memory meets the above conditions will the system perform similarity matching between the elastic visual fingerprint of the current character to be identified and the historical elastic visual fingerprints in the memory.

[0065] Similarity matching criteria and calculation methods The system first generates a flexible visual fingerprint image of the current character to be identified using the same method as for historical samples. Then, it sequentially reads the flexible visual fingerprint images corresponding to each historical character from the memory and calculates the template similarity between the current flexible visual fingerprint image and each historical flexible visual fingerprint image.

[0066] The similarity score is calculated using a normalized correlation matching method. For the current elastic visual fingerprint image and historical elastic visual fingerprint images, under the same size condition, the consistency of their pixel distribution is calculated to obtain a similarity score. The similarity score is a normalized value; the larger the value, the closer the current character handwriting pattern is to the historical character handwriting pattern.

[0067] The system iterates through all historical records in the memory, obtains the similarity score corresponding to each historical elastic visual fingerprint image, and selects the historical record with the highest similarity score as the best matching result. If the highest similarity score is greater than a preset matching threshold, the character tag corresponding to that historical record is used as the matching result in the memory; if the highest similarity score does not reach the preset matching threshold, it is determined that there is no directly usable matching result in the memory.

[0068] In this embodiment, the preset matching threshold is 0.50. That is, when the highest similarity score between the current elastic visual fingerprint and a certain historical elastic visual fingerprint in the memory is greater than 0.50, the system considers that the historical record to be superior in the memory matching and outputs the character label corresponding to the historical record.

[0069] Step S3: Game Decision Determination By combining the confidence level of the first identification result and the similarity score of the second identification result, the system automatically filters and outputs the final identification suggestion according to the preset game weight rules.

[0070] The similarity score is calculated as follows: The similarity score of the second recognition result is calculated based on the pixel correlation between the elastic visual fingerprint of the current character to be recognized and the historical elastic visual fingerprints in the memory bank.

[0071] Specifically, the system first generates the current elastic visual fingerprint image based on the binarized mask of the character to be identified; then, it sequentially reads the elastic visual fingerprint images corresponding to each historical character from the memory. The current elastic visual fingerprint image and the historical elastic visual fingerprint images are both grayscale images of the same size, for example, 128×128 pixels.

[0072] For the current elastic visual fingerprint image A and the historical elastic visual fingerprint image B, the system calculates the similarity score S between the two according to the normalized correlation matching method. The calculation method is: S is equal to the sum of the products of the corresponding pixels of A and B, divided by the square root of the product of the sum of squares of the pixels of A and the sum of squares of the pixels of B.

[0073] In other words, the similarity score reflects the degree of consistency between two elastic visual fingerprints in terms of stroke position, stroke shape, overall structure, and grayscale distribution. The similarity score is a normalized value; the higher the value, the more similar the handwriting shape of the current character is to that of a historical character.

[0074] The system iterates through all historical elastic visual fingerprints in the memory, calculates the similarity score between the current elastic visual fingerprint and each historical elastic visual fingerprint, and selects the highest similarity score as the similarity score of the second recognition result; the historical character label corresponding to the highest similarity score is used as the second recognition result.

[0075] The steps involved in determining the criteria for game decision-making include: After obtaining the first and second identification results, the system performs a game-theoretic decision-making process.

[0076] The first recognition result is the character result output by the OCR recognition engine, and its confidence level is denoted as C. The second recognition result is the character result output by the memory matching engine, and its similarity score is denoted as S.

[0077] The system determines the validity of C and S based on preset thresholds: when S is greater than the first preset threshold, the second recognition result is determined to be valid, indicating that the handwriting pattern of the current character has a sufficiently high consistency with the historical handwriting in the memory bank; when C is greater than the second preset threshold, the first recognition result is determined to be valid, indicating that the OCR recognition engine has a sufficiently high recognition credibility for the current character.

[0078] In this embodiment, the first preset threshold is 0.50 and the second preset threshold is 0.40.

[0079] When the similarity score S of the second identification result is greater than 0.50, the system determines that the memory matching engine has the advantage in this game and outputs the second identification result as the final identification suggestion.

[0080] When the second recognition result does not exist, or the similarity score S is less than or equal to 0.50, while the confidence score C of the first recognition result is greater than 0.40, the system determines that the OCR recognition engine has the advantage in this game and outputs the first recognition result as the final recognition suggestion.

[0081] When the similarity score S of the second recognition result does not reach 0.50 and the confidence score C of the first recognition result does not reach 0.40, the system determines that the current character cannot be reliably confirmed by the existing engine, does not directly adopt any recognition result, but outputs a status pending manual confirmation, and uses the result with higher confidence or higher similarity as a candidate prompt.

[0082] The final character results, after manual verification, along with the corresponding binarized mask, flexible visual fingerprint, and related feature data, are written into the sample library or memory for subsequent similarity matching and online model learning.

[0083] The steps involved in the synchronous update process of the visual fingerprint index include: The synchronous update of the visual fingerprint index refers to establishing an index relationship between the finally confirmed character label and the elastic visual fingerprint corresponding to the character after the user confirms or corrects the recognition suggestions output by the system, and writing it into the memory.

[0084] The specific process is as follows: First, the system outputs a recognition suggestion for the current character. This suggestion can be derived from OCR recognition results, similarity matching results from a memory database, or the final suggestion after a game-theoretic decision-making process.

[0085] Next, the user confirms or corrects the recognition suggestion. If the user confirms the recognition suggestion is correct, the system uses the suggestion as the final character label; if the user modifies the suggestion, the system uses the corrected character as the final character label. The final character label serves as the sole valid label for subsequent memory updates.

[0086] The system then obtains the binarized mask corresponding to the final character label. This binarized mask is the final mask generated by a deep learning segmentation model and corrected by the user.

[0087] Next, the system generates a flexible visual fingerprint based on the final binarized mask. Specifically, this involves: performing a morphological closing operation on the binarized mask; extracting all ink pixels; calculating the minimum bounding rectangle based on the ink pixels; cropping a local mask containing the complete handwriting; scaling the local mask proportionally according to a preset ratio; centering the scaled handwriting area on a standard canvas of a preset size; and generating a visual fingerprint image and corresponding fingerprint identifier value from the standard canvas.

[0088] Finally, the system writes the fingerprint identifier value, visual fingerprint image, and final character label into the memory. If the same fingerprint identifier value does not exist in the memory, a new index record is added; if the same fingerprint identifier value already exists in the memory, the index record is updated with the final character label confirmed or corrected by the user. After the update is complete, the system saves the visual fingerprint image to the memory and writes the correspondence between the fingerprint identifier value and the character label into the index file, thereby completing the synchronous update of the visual fingerprint index.

[0089] The steps for instructing users to update the memory after confirmation or correction include: The character labels in the fingerprinting memory directly determine the character results confirmed or corrected by the user. When a user confirms the system's recognition result, the system uses that result as the character label for the current sample and updates the visual fingerprinting index accordingly. When a user corrects the system's recognition result, the system no longer uses the original recognition suggestion but instead uses the user-inputted corrected character as the character label for the current sample and updates the visual fingerprinting index accordingly. Therefore, every addition or update in the fingerprinting memory is based on the character label finally confirmed by the user. This mechanism ensures that the visual fingerprinting index in the fingerprinting memory continuously absorbs reliable samples confirmed by humans for subsequent similarity matching.

[0090] The steps of the binarized mask correction method include: The correction of the binarized mask refers to the user manually correcting the ink and background areas in the mask after the system generates the initial binarized mask, resulting in the final binarized mask. Specifically, the system displays the initial binarized mask as a semi-transparent overlay on the image area to be annotated, allowing the user to simultaneously observe the original handwriting and the mask coverage result. When the initial binarized mask omits some actual strokes, the user performs an addition operation in the omitted area. The system determines the user's relative coordinates in the current character image area based on the user's operation position, and uses these coordinates as the center and a preset brush radius to determine the range of pixels to be modified, assigning the mask pixels within this range the ink pixel value. When the initial binarized mask incorrectly contains background, stains, noise, or non-stroke areas, the user performs an erasure operation in the erroneous area. The system also determines the relative coordinates based on the user's operation position, and uses these coordinates as the center and a preset brush radius to determine the range of pixels to be modified, assigning the mask pixels within this range the background pixel value. In this embodiment, the ink pixel value is 255, and the background pixel value is 0. After each addition or erasure operation, the system updates the mask overlay display in real time. Once the user completes the correction and confirms the save, the current binarized mask becomes the final binarized mask, used for sample saving, flexible visual fingerprint generation, visual fingerprint index updating, and subsequent online model learning.

[0091] The specific implementation steps of the elastic memory retrieval engine include: The elastic memory retrieval engine is used to retrieve the most similar historical character sample from the memory bank based on the handwriting pattern of the current character to be identified, and outputs a second recognition result and its similarity score. The specific implementation process is as follows: First, the system loads the visual fingerprint index from the memory. The visual fingerprint index includes a fingerprint identifier value, historical character tags, and a flexible visual fingerprint image corresponding to the fingerprint identifier value. If the memory is empty, or if no valid visual fingerprint index exists, the flexible memory retrieval engine will not output a valid match result.

[0092] Secondly, the system acquires a binary mask of the character to be identified and generates a current flexible visual fingerprint based on the binary mask. Specifically, the system performs a morphological closing operation on the binary mask; extracts the ink pixels; calculates the minimum bounding rectangle based on the ink pixels; crops to obtain a local mask containing the complete handwriting; scales the local mask proportionally along its long side; centers the scaled handwriting area in a standard canvas of a preset size; and then smooths the standard canvas to obtain the current flexible visual fingerprint image.

[0093] Then, the system sequentially reads historical elastic visual fingerprint images from the memory. The historical elastic visual fingerprint images have the same size, the same pixel representation, and the same preprocessing procedure as the current elastic visual fingerprint image.

[0094] Next, the system calculates the similarity score between the current elastic visual fingerprint image and each historical elastic visual fingerprint image. The similarity score is calculated using a normalized correlation matching method, which involves multiplying the pixel values ​​at corresponding positions in the two elastic visual fingerprint images, summing the results, and then dividing by the square root of the product of the squared pixel values ​​of the two images to obtain the normalized similarity score. This similarity score characterizes the degree of consistency between the current handwriting pattern and the historical handwriting patterns.

[0095] Subsequently, the system iterates through all historical elastic visual fingerprint images and records the historical sample with the highest similarity score. This highest similarity score is used as the similarity score for the second recognition result, and the historical character label corresponding to this historical sample is used as the second recognition result.

[0096] Finally, the system determines whether the second recognition result is valid based on a preset similarity threshold. When the highest similarity score is greater than the preset similarity threshold, the elastic memory retrieval engine is deemed to have successfully retrieved the data, and the second recognition result and its corresponding similarity score are output. When the highest similarity score is less than or equal to the preset similarity threshold, it is determined that there are no reliable matching samples in the memory, and the elastic memory retrieval engine does not directly output an usable recognition result.

[0097] In this embodiment, the standard canvas size is 128×128 pixels, the normalized handwriting long side size is 100 pixels, and the preset similarity threshold is 0.50. If the highest similarity score between the current elastic visual fingerprint and a certain historical elastic visual fingerprint in the memory database is greater than 0.50, then the character label corresponding to that historical sample is used as the second recognition result output by the elastic memory retrieval engine; otherwise, the retrieval result is only used as a candidate reference and is not used as a directly adopted result.

[0098] Step S4: Interactive Correction and Online Evolution The user-corrected binarized mask is used as the ground truth label, and the corresponding original image slice is used as the training sample.

[0099] In the background process of annotation interaction, the weight parameters of the deep learning segmentation model are updated in real time using the stochastic gradient descent algorithm.

[0100] In response to the user's confirmation or correction of the final recognition suggestion, the system uses the user-confirmed character as the true character label for the current character sample, and guides the user to select the corresponding character region character by character based on this true character label. The system crops the character region to obtain single-character image slices and calls a deep learning segmentation model to generate an initial binarized mask. The initial binarized mask is displayed as a semi-transparent overlay on the original character image for the user to interactively correct.

[0101] During the interactive correction process, when a stroke is missed, the system responds to the user's foreground smearing operation by updating the mask pixel value within the user's operating range to 255; when a background is falsely detected, the system responds to the user's background erasure operation by updating the mask pixel value within the user's operating range to 0. The operating range is determined by the user's operating coordinates and the preset brush radius. After each correction, the system refreshes the mask overlay layer in real time until the final binarized mask is obtained.

[0102] The system takes the final binarized mask as input and calls the elastic memory retrieval engine to generate an elastic visual fingerprint. Specifically, the system performs a morphological closing operation on the final binarized mask, extracts the minimum bounding rectangle of the non-zero ink pixels, crops the main character area, and centers the main character area in a standard canvas of a preset size after scaling it proportionally. Then, the system performs Gaussian blur processing on the standard canvas to obtain an elastic glyph image with translation tolerance, and generates a hash value based on the standard canvas as the visual fingerprint index key.

[0103] The system establishes a mapping relationship between the visual fingerprint index key and the real character label confirmed by the user, and writes it into the visual fingerprint index file. Simultaneously, it saves the flexible glyph image, named after the visual fingerprint index key, to the flexible memory directory. Thus, the flexible memory retrieval engine forms a structured memory relationship of "visual fingerprint index key—real character label—flexible glyph image". When processing new characters to be identified, the system can perform similarity matching between the flexible visual fingerprint generated by the new character and the flexible glyph images already saved in the memory, and return candidate characters based on the index relationship corresponding to the highest similarity.

[0104] Furthermore, the system writes the single-character image slice, the corrected final binarized mask, the ink saliency map, the edge contour map, and the skeleton auxiliary map into the training sample library, and records the character label and image coordinate information in the metadata file. Subsequently, the system uses the corrected final binarized mask as the supervision label to perform online incremental training on the deep learning segmentation model. During training, the system extracts the original image features, ink saliency features, and edge contour features of the current character image slice, inputs them into the segmentation model, and uses the corrected final binarized mask as the target output. The error between the predicted mask and the target mask is calculated through a loss function, and then the model parameters are updated through backpropagation and the updated model weights are saved. Through the above process, the system simultaneously completes the expansion of the elastic memory library and the online evolution of the segmentation model, so that the subsequent similar character recognition suggestions and pixel-level segmentation results are continuously optimized with the accumulation of manually corrected samples.

[0105] This invention also provides an intelligent recognition and annotation system for handwritten remittances from overseas Chinese based on multi-engine game theory, including an image processing module, a pixel segmentation module, a multi-path recognition module, a game decision-making module, and an online learning module, each capable of executing the corresponding steps in the above methods. Additionally, a medium storing a computer program is provided, which implements the above methods when executed.

[0106] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for recognizing and labeling a batch of handwritten characters based on multi-engine game, characterized in that, Includes the following steps: Step S1: Multidimensional feature extraction and segmentation. Obtain the image region to be labeled, extract the original image features, ink saliency features, and edge features, input them into a preset deep learning segmentation model, and generate a pixel-level binary mask. The deep learning segmentation model is a lightweight pixel-level convolutional segmentation model for the handwritten handwriting region of overseas Chinese remittances. Its input is a five-channel fusion feature, and its output is a single-channel probability map, which is thresholded to generate a binary mask. The five-channel fusion feature is formed by splicing three-channel original image features, single-channel ink saliency features, and single-channel edge features in the channel dimension. Step S2: Multi-path parallel recognition processing, including a visual purification recognition path and an elastic memory retrieval path; in the visual purification recognition path, the original image is removed using the binarized mask to generate a clean handwriting image and input it into the OCR engine to obtain the first recognition result and its confidence level. In the elastic memory retrieval path, the elastic visual fingerprint of the binarized mask is extracted, and similarity matching is performed in the preset memory to obtain the second recognition result and its similarity score. Step S3: Game decision determination, combining the confidence of the first identification result and the similarity score of the second identification result, automatically filter and output the final identification suggestion according to the preset game weight rules; Step S4: Interactive correction and online evolution, in response to the user's confirmation or correction operation of the final recognition suggestion, obtain the real character label and generate the corrected final binarized mask; Based on the final binarized mask, a flexible visual fingerprint is generated and the memory index is updated; The corrected samples are written into the training sample library, and the deep learning segmentation model is trained incrementally online.

2. The method according to claim 1, wherein, The multidimensional feature extraction in step S1 specifically involves: The original image is converted to HSV space and ink saliency maps are extracted within a specific threshold range; Edge feature maps of the original image are extracted using the Canny operator; The original image, ink saliency map, and edge feature map are concatenated to form a five-dimensional tensor, which is then input into the deep learning segmentation model.

3. The method for intelligent recognition and annotation of handwritten remittances based on multi-engine game theory according to claim 1, characterized in that, The visual purification recognition path in step S2 specifically includes: Using the binarized mask as a mask, extract the handwriting pixels from the original image; Create a pure white background image with the same size as the original image, and overlay the extracted handwriting pixels onto the pure white background image to generate a clean image that eliminates background noise, creases, and stamp interference. The purified image is then input into the OCR engine.

4. The method for intelligent recognition and annotation of handwritten remittances based on multi-engine game theory according to claim 1, characterized in that, The process of extracting the elastic visual fingerprint in step S2 includes: The binarized mask is subjected to morphological closing operations to connect broken strokes; Calculate the minimum bounding rectangle of the connected region and align its centroids, then normalize and scale the handwriting region to the center of the canvas of a preset size. The normalized image is subjected to Gaussian blurring to enhance translation tolerance, and the corresponding hash feature value is generated as a visual fingerprint.

5. The method for intelligent recognition and annotation of handwritten overseas remittance documents based on multi-engine game theory as described in claim 1, characterized in that, The game weighting rules in step S3 are as follows: If the similarity score of the second identification result is higher than the first preset threshold, and the confidence level of the first identification result is lower than the second preset threshold, then the second identification result is determined to be the final identification suggestion. If the confidence level of the first identification result is higher than the third preset threshold, then the first identification result is determined to be the final identification suggestion.

6. The method for intelligent recognition and annotation of handwritten overseas remittance documents based on multi-engine game theory according to claim 1, characterized in that, The online incremental training in step S4 specifically includes: Use the user-corrected binarized mask as the ground truth label and the corresponding original image slice as the training sample. In the background process of annotation interaction, the weight parameters of the deep learning segmentation model are updated in real time using the stochastic gradient descent algorithm.

7. The method for intelligent recognition and annotation of handwritten remittances based on multi-engine game theory according to claim 1, characterized in that, The similarity score of the second identification result is calculated according to the normalized correlation matching method: the similarity score S is equal to the sum of the pixel products of the current elastic visual fingerprint image A and the historical elastic visual fingerprint image B, divided by the square root of the product of the sum of squares of the pixels of A and the sum of squares of the pixels of B.

8. The method for intelligent recognition and annotation of handwritten remittances based on multi-engine game theory according to claim 1, characterized in that, The synchronization update process of the visual fingerprint index in step S4 includes: The final character label is obtained after user confirmation or correction; Generate a flexible visual fingerprint image and fingerprint identification value based on the corresponding final binarized mask; The fingerprint identifier value, visual fingerprint image, and final character label are written into the memory; if the same fingerprint identifier value does not exist in the memory, a new record is added; if it already exists, the corresponding record is updated.

9. A multi-engine game-based intelligent recognition and annotation system for handwritten overseas Chinese remittance documents, characterized in that, include: The image processing module is used to acquire image slices and extract multidimensional features; The pixel segmentation module is used to run deep learning models and generate binarized masks; The multi-channel recognition module includes a visual purification OCR unit and a flexible fingerprint retrieval unit; The game-theoretic decision-making module is used to select the best outcome based on confidence and similarity. The online learning module is used for memory updates and real-time evolution of model weights.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images