A method and system for identifying the crown code of multi-variety banknotes for an NPU

Abstract

The application discloses a kind of NPU-oriented multi-edition banknote crown code number identification method and system, for the crown code number identification in multi-edition, multi-pose and complex imaging condition banknote image, specifically includes the following steps: data annotation and pre-processing stage;Model training stage;NPU adaptation and model quantization stage.For the crown code number of Arab currency and the like in size and position variation, character adhesion / connected pen and background reflection, anti-counterfeiting underlay interference and the like, the traditional template two-stage OCR is prone to miss detection, false detection and sequence disorder, a template-independent end-to-end recognition method is proposed, which completes the robust positioning and sequence correct identification of crown code number in a single process, significantly reduces false positives and wrong sequence, and uses NPU to accelerate to meet the real-time and stability requirements of embedded and edge.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition technology, specifically to a method and system for recognizing serial numbers of multiple versions of banknotes for NPU. Background Technology

[0002] The serial number recognition method based on templates and two-stage OCR (Optical Character Recognition) typically completes the automatic reading of banknote serial numbers by first locating the Region of Interest (ROI) and then performing character recognition. Its core idea is to use a banknote layout template (such as the approximate position of the serial number, row direction, and tilt angle) and traditional image processing (edge / projection / morphology) to locate the text band in the entire banknote image, and then pass this area to a general OCR (such as SVM / HOG, CNN, or CRNN-CTC) to output a character sequence. This method achieves good results on banknotes with stable layouts and sufficient character spacing.

[0003] The recognition method has four steps: localization, preprocessing, recognition, and verification. These steps are detailed below: 1. Template / Prior Localization: Based on the currency format and size priors, geometric template matching, edge / projection analysis, or color / texture thresholding are used to coarsely locate the approximate area where the serial number is located. 2. Preprocessing and Fine Localization: Binarization, denoising, and morphological operations are performed within the ROI. Tilt / perspective correction is performed when necessary, and text lines or character candidates are segmented using connected components or projection. 3. Character Recognition: The segmented characters (or entire lines) are recognized using a general OCR model (such as SVM / HOG, CNN, or CRNN-CTC) to obtain candidate character sequences and confidence scores. 4. Rule Verification and Error Correction: Regularity verification and simple error correction are performed based on currency rules (number of digits, alphanumeric pattern, allowed character set), outputting the final serial number result.

[0004] Although two-stage OCR is relatively mature in engineering, it has obvious limitations in the Arabic coin scenario: (1) It is highly sensitive to changes in size and position. The two-stage template / projection / connected component positioning method relies on fixed format and threshold. The differences in font size, relative position drift and line direction and tilt angle caused by different versions and denominations make it difficult for fixed templates and thresholds to cover, which easily leads to missed detection and false detection; (2) The character sticking and segmentation is unstable. Affected by printing, collection and wear, Arabic letters and Arabic-Hindi numerals often stick together, are connected and obscured. The unstable segmentation of single characters leads to reading in sequence, missed reading and disordered order; at the same time, the differences in the writing direction of Arabic text RTL / LTR and the curved baseline It is difficult to adapt and relies on prior assumptions, which can easily lead to systematic misreading; (3) Background and reflection interference, complex anti-counterfeiting background, reflection, wrinkles and stains make binarization and connected component thresholds unstable, requiring a lot of manual rules and special cases to maintain, resulting in high engineering costs and difficulty in migration; (4) Strong template dependence and high maintenance costs: when crossing currencies, batches and devices, templates and thresholds need to be readjusted, making it difficult to cover long-tail changes and resulting in poor scalability; (5) It does not support NPU acceleration and has a slow inference speed. The existing technology process is a loosely coupled link that is not end-to-end. The errors in the positioning-segmentation-recognition links are easily amplified by cascading, making it difficult to achieve overall optimization through unified loss. It is difficult to meet the real-time requirements in batch recognition or online clearing scenarios.

[0005] The following are the corresponding Chinese names for the relevant English names: OCR: Optical Character Recognition.

[0006] ROI: Region of Interest.

[0007] SVM / HOG: Support Vector Machine / Square Gradient Histogram.

[0008] CNN: Convolutional Neural Network.

[0009] YOLOX-RBox / Quad: A target detection model that supports rotational rectangle or four-point bounding box regression.

[0010] CRNN-CTC: A line-level text recognition model that combines convolution, recurrent neural networks, and CTC decoding.

[0011] CTC: Connection Temporal Classification Algorithm, used for sequence decoding that does not require character-by-character segmentation.

[0012] RTL / LTR: Text direction markers, representing right-to-left and left-to-right respectively.

[0013] Geometric normalization (WarpAffine / Perspective Transformation): Uses four-point coordinates to straighten slanted text into a standard rectangle.

[0014] QAT: Quantization Awareness Training.

[0015] INT8 quantization: Compresses model weights and activations into 8-bit fixed-point numbers.

[0016] Conv+BN+Act fusion: Combines convolution, normalization, and activation operators into a single convolution operator.

[0017] NMS (Rotation NMS): Non-maximum suppression, a post-processing method used to remove overlapping candidate boxes.

[0018] RKNN: The NPU model format for the Rockchip platform, derived from the ONNX model.

[0019] NPU: Neural Network Processor, used for efficient inference acceleration of deep learning models.

[0020] Rotated box mAP@[0.50:0.95]: The average precision (AP) is calculated for each category within the range of IoU thresholds from 0.50 to 0.95 (10 categories in total). Then, the AP values ​​of all categories are averaged to obtain a final comprehensive performance score. Summary of the Invention

[0021] In view of this, it is necessary to provide a robust method and system for serial number recognition of Arabic coin banknotes that can be completed in a single process with end-to-end robust positioning and correct sequence recognition, and which utilizes NPU acceleration to meet the real-time and stability requirements of embedded and edge devices.

[0022] A serial number recognition method for multi-version banknotes for NPU is provided for serial number recognition in banknote images with multiple versions, poses, and complex imaging conditions. The method specifically includes the following steps: Step 1, Data Labeling and Preprocessing Stage: For the banknote image to be identified, the serial number text band is labeled using a four-point / rotated frame, and the corresponding complete label string is retained; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text bands of arbitrary size and tilt angle into a standardized ROI with a fixed height; Step 2, Model Training Phase: Utilize YOLOX's rotation / four-point regression to achieve robust text band localization, and train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, the segmentation difficulties caused by character adhesion are avoided at the sequence level. Step 3, NPU adaptation and model quantization stage; QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints are performed on the detection and recognition link to convert the exported ONNX model into an NPU-friendly RKNN model. Step four, edge-side inference stage: The integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" is executed on the NPU. The CPU only undertakes light CTC decoding and syntax constraints, outputting the serial number string and confidence score, thus meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

[0023] Preferably, step one, the data labeling and preprocessing stage, includes the following specific steps: Step 1.1: Data preparation and sample organization; according to the acquisition specifications, collect banknote images covering typical working conditions such as size differences, tilt angle changes, specular reflection, complex anti-counterfeiting background patterns, character adhesion / connection, and partial occlusion; generate a unique ID for each image and record acquisition metadata; the acquisition metadata includes device model, resolution, exposure, gain, light source conditions, and posture information; Step 1.2: Complete the geometric annotation, label recording, and direction marking of the serial number text band; for each banknote image, only annotate the serial number text band without character-level segmentation, and the annotation form is four-point coordinates or a rotated rectangle; the four points are stored in clockwise order according to the image coordinate system, with the starting point fixed at the upper left corner; if the text band has slight curvature, it is annotated in the form of an envelope quadrilateral, and the boundary is expanded by a predetermined number of pixels to avoid truncation in subsequent transformations; simultaneously record the entire string of character labels and text direction RTL / LTR for each annotation box: when it is RTL, it is entered from right to left according to the actual reading order; establish a one-to-one mapping table for Arabic-Hindi numerals and Arabic numerals, and save the original characters and mapped characters in two columns; Step 1.3: Supplement semantic priors and consistency checks; Based on the currency and version, generate and store the grammatical priors corresponding to the samples. The grammatical priors include the number of bits, allowed character sets, alphanumeric combination patterns, and optional check bit rules; Run an automatic verification script on the labeled data to check the order and closure of the four points, the minimum area and aspect ratio thresholds, and the consistency between the text direction and the label; Randomly sample a predetermined proportion of samples for manual review and calculate the quadrilateral IoU and string consistency rate. Samples below the threshold are entered into the rework queue. Step 1.4: Perform geometric normalization to form a standardized input; calculate the perspective or affine transformation matrix based on the four-point coordinates of each ROI, and normalize the text strip into a strip image with a fixed height H and adaptive width. Use bilinear interpolation to balance sharpness and smoothness; perform lightweight preprocessing without destroying the topology of the character edges, including local contrast enhancement, gamma correction, specular suppression, and little kernel denoising; cache the normalized text strip on disk as the direct input to the recognition sub-model. Step 1.5: Complete the dataset partitioning and hard example configuration; stratify the sampling by "currency × version" to divide the samples into three parts: training, validation, and testing, ensuring that the same physical banknote is not distributed across sets; keep the proportion of hard examples in each set basically consistent for various typical working conditions, and install a predetermined weighting ratio during training sampling to ensure the visibility of hard examples during the training process. Step 1.6: Construct a representative calibration subset for edge quantization and NPU compilation; screen samples covering various typical working conditions to form a calibration list of no less than 1000 normalized text bands and several original images for QAT or post-quantization calibration, ensuring that the accuracy of INT8 quantization does not significantly decrease in long-tail scenarios.

[0024] Preferably, step two, the model training phase, includes the following specific steps: Step 2.1: The detection part uses YOLOX rotation / four-point regression to achieve robust text band localization and obtain a standardized region of interest (ROI) with a fixed height; Step 2.2: The recognition part trains a row-level recognition model on the normalized ROI and uses forward / backward bidirectional decoding to adapt RTL / LTR. Step 2.3: Throughout the training process, the detection and recognition parts share the same set of data augmentations, which include scale adjustment, light perspective adjustment, noise / compression adjustment, and reflection simulation. For difficult cases of various typical working conditions, the weights are increased in the sampling or loss to enhance the learning ability for long-tail scenes. Step 2.4, in the validation phase, the detection model uses the rotated bounding box mAP@[0.50:0.95] and recall as the main indicators; the recognition model uses the character error rate (CER) and word error rate (WER) as the main indicators; the end-to-end model uses the whole string accuracy as the final indicator; at the same time, the above indicators are statistically analyzed on three subsets: size change, character adhesion, and strong reflection, to check the robustness and generalization performance of key difficulties; the inference deduplication uses the rotated version of nonmaximum suppression (NMS); Step 2.5: After training is completed, output the weight file for detection and recognition, geometric normalization parameters and input specifications, as well as the inference configuration, which includes character set / mapping, orientation adaptation and decoding strategy, detection threshold and NMS configuration.

[0025] Preferably, step 2.1, where the detection part utilizes YOLOX rotation / four-point regression to achieve robust text band localization and obtain a standardized region of interest (ROI) with a fixed height, includes the following specific steps: Step 2.1.1: Using the completed four-point / rotated box annotations, perform k-means++ clustering on the width, height and aspect ratio of the prefix target to generate a set of anchor boxes that match the thin text band; Step 2.1.2: Using YOLOX as the main framework, the detection head is modified to support the regression RBox / Quad that supports rotating boxes / four vertices. The regression loss uses the rotation intersection-union ratio SkewIoU / generalized intersection-union ratio GIoU, and is trained together with the classification / confidence branch. Step 2.1.3: Multi-scale training is used at the input end, and lightweight imaging enhancement is added to make the model more robust to size changes, shooting posture and lighting disturbances. Step 2.1.4: After the detection model converges, output the coordinates of the four points of the prefix, and then use perspective / affine transformation to unify the text into a standardized region of interest (ROI) of fixed height.

[0026] Preferably, step 2.2, where the recognition part trains a row-level recognition model on the normalized ROI, and the specific steps of adaptive RTL / LTR through forward / backward bidirectional decoding include: Step 2.2.1: The recognition part uses the normalized text strips mentioned above as input to train a row-level sequence recognition model of lightweight convolutional recurrent neural network CRNN+connected temporal classification CTC: front-end convolution / depthseparable convolution extracts temporal features, followed by bidirectional recurrent units to model the sequence, and the end uses CTC loss to complete the training without character segmentation. Step 2.2.2: Since the writing direction of Arabic scenes may be different, the same text band is decoded in both forward and reverse directions during inference, and the side with the higher score is selected by the average log probability of the sequence, so as to adaptively right-to-left / left-to-right RTL / LTR without explicit direction labels.

[0027] Preferably, step three, the NPU adaptation and model quantization stage, includes the following specific steps: Step 3.1, Model Export: Export the detection subnet YOLOX-RBox / Quad and the recognition subnet CRNN-CTC from PyTorch as ONNX models, unify and fix the input size, and specify the channel order and normalization method in the export script to ensure alignment of the end-side preprocessing; then perform small sample consistency verification against onnxruntime to confirm that the bounding box position / angle and recognition results are consistent with the PyTorch baseline; Step 3.2, Operator replacement and fusion; perform graph optimization on the exported ONNX model; Step 3.3, cascading continuous subgraphs; connecting "detection → geometric normalization → recognition" into a single computational graph and solidifying it; Step 3.4, Quantify perception training; The convolution weights are symmetrical INT8 based on the output channel, and the intermediate activations are asymmetric INT8 layer by layer. The scaling factor and zeros are accurately simulated with dummy quantization nodes, so that the forward pass is equivalent to the fixed-point operation on the end side, in order to achieve consistency with the target hardware quantization rules. During the fine-tuning phase, the batch normalized statistics are frozen, keeping the SkewIoU / GIoU and Focal loss on the detection side and the CTC loss on the recognition side unchanged. Training is driven by a calibration set with extreme coverage size, tilt angle variation, strong reflection and character adhesion, increasing the proportion of difficult examples in each batch to strengthen long-tail adaptation and improve numerical stability. A mixed-precision whitelist is used to retain the first, last, and bidirectional loop layers as FP16, while the remaining layers are executed as INT8 to balance precision and computing power, thereby reducing the fluctuations caused by quantization in critical layers. Use a small learning rate for short-cycle fine-tuning and set early stop, continuously monitor the end-to-end whole string accuracy, detect mAP and recognize CER / WER regression, and converge and export when the accuracy meets the requirements, so as to stabilize the convergence after quantization; After convergence, the scaling factor and zero point of each layer are exported synchronously, and the Q / DQ configuration is solidified as authoritative quantization parameters, which serve as direct inputs for subsequent RKNN transformation and compilation. This achieves near-floating-point accuracy and stable latency and power consumption performance on the NPU, thus enabling the deployment process.

[0028] Step 3.5, Model to RKNN: The quantized ONNX model is converted into an RKNN model using rknn-toolkit for subsequent board-side inference.

[0029] Preferably, step 3.2, the specific steps of operator replacement and fusion, includes: In step 3.2.1, the three operators—convolutional layer (Conv), normalization layer (BN), and activation layer (ReLU)—are merged into one operator, which mathematically combines them into an equivalent convolution, as shown in equation (1): (1); In this case, the scaling and translation of BN are directly folded into the weights and biases of the convolution; x is the input feature tensor / input feature map of the fusion operator of equivalent convolution, which comes from the output of the previous layer or from the tensor after preprocessing the input image; y is the output feature tensor / output feature map of the fusion operator, which is the result after equivalent convolution and activation, and is used by the next layer; In step 3.2.2, after the operator folding is completed, the operators in the model that are not supported by the NPU are replaced with NPU-friendly operators; Step 3.2.3 leaves post-processing that may not be supported by the NPU to the CPU / chip built-in unit to ensure that detection → geometric normalization → recognition can be connected into the same computation graph.

[0030] Preferably, step 3.3, the specific steps of concatenating continuous subgraphs, includes: In step 3.3.1, the rotation frame / four-point coordinates output by the detection head are unified in a clockwise order and normalized according to the input size; a geometric transformation matrix is ​​directly generated within the image based on the rotation frame / four-point coordinates, and a strip-shaped ROI with fixed height and adaptive width is output; In step 3.3.2, the strip ROI tensor is seamlessly connected to the recognition subnet input, and multiple targets in the same original image are stacked in batches within the image to reduce multiple startups and transfers. Zero-copy / shared buffering is enabled during runtime to prevent intermediate results from falling back to the CPU. Step 3.3.3 involves using a small sample to complete visualization and consistency verification, and recording the p50 / p90 latency and memory usage of the entire chain. Once a benchmark better than segmented execution is achieved, it is used as the deployment configuration.

[0031] Preferably, step four, the specific steps of the end-side inference stage, includes: In step 4.1, install the official chip SDK and rknn-toolkit runtime, and confirm that the development board firmware and driver version match. At the same time, pre-allocate input / output buffers and set zero-copy or shared buffer strategies. In step 4.2, an RKNN instance is created on the device side, the .rknn model file is loaded, the options for runtime thread count, core binding, and memory strategy are set, and nit_runtime() is called to complete the initialization. After initialization, an empty run is performed to warm up the device, fill the kernel cache and constant folding results, and stabilize the latency of the first frame. In step 4.3, after initialization is completed, the original image is input, scaled and filled according to the fixed input size of the detection network, and the same color channels and normalization as during training are performed. Then, inference is called to run the entire computation graph. In step 4.4, the rotating frame / four-point coordinates output by the detection head directly drive the WarpAffine within the image to generate a fixed-height strip ROI with adaptive width expansion, ensuring that the characters are aligned in the vertical direction; and directly enter the recognition subnet within the image, with intermediate tensors being transferred within the NPU without needing to return to the CPU; In step 4.5, after the NPU completes the recognition, the CPU or hardware unit completes the rotation NMS, sorts the data by confidence level, removes duplicates based on rotation IoU, and outputs the final four-point bounding box and its confidence level. In step 4.6, CTC decoding is performed on the recognition output, the beam width is set, and forward and reverse decoding are performed simultaneously. The better side is selected by log likelihood, and the adaptive RTL / LTR is used. If the confidence of LTR is greater, the string is output normally from left to right; otherwise, the string is output from right to left.

[0032] Furthermore, a serial number recognition system for multiple versions of banknotes for NPU, used to implement the serial number recognition method for multiple versions of banknotes for NPU as described above, comprising: The data annotation and preprocessing module is used to annotate the serial number text strips of the banknote image to be identified using a four-point / rotated frame and retain the corresponding complete label string; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text strips of arbitrary size and tilt angle into a standardized ROI of fixed height; The model training module is used to achieve robust text band localization using YOLOX rotation / four-point regression and to train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, it avoids the segmentation difficulties caused by character adhesion at the sequence level. The NPU adaptation and model quantization module is used to perform QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints on the detection and recognition link, and convert the exported ONNX model into an NPU-friendly RKNN model. The edge-side inference module is used to execute the entire graph of the integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" on the NPU. The CPU only undertakes lightweight CTC decoding and syntax constraints, outputting the serial number string and confidence score, thereby meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

[0033] In the aforementioned method and system for recognizing serial numbers of multiple banknote versions for NPU, considering the significant size and tilt differences in serial numbers of currencies such as Arabic coin under different versions and acquisition postures, this invention employs four-point / rotated bounding box annotation on the data side and uses YOLOX rotation / four-point regression to achieve robust localization of text bands during the detection stage. This unifies text bands of arbitrary size, tilt, and slight curvature into a standardized ROI of fixed height, eliminating the impact of inconsistent character sizes on recognition stability from the source. Addressing the common issues of character adhesion and ligatures in acquired banknote images that disrupt character-level segmentation and trigger cross-reading and sequence errors, this invention employs row-level sequence recognition on the standardized ROI and implements forward / reverse bidirectional decoding, adaptive RTL / LTR, bypassing pseudo-boundaries at adhesion points and ensuring correct sequence at the sequence modeling level. To achieve efficient program execution, this invention performs Quantization-Aware Training (QAT) after training convergence and completes Conv+BN+Act fusion, static input shape constraints, and replacement of unsupported operators, deriving an NPU-friendly RKNN model. During deployment, INT8 quantization, zero-copy memory, and multi-threaded pipelines are enabled, allowing the operator graph of detection → normalization → recognition to execute continuously on the NPU, avoiding CPU rollback and unnecessary data transfer. Finally, relying on "rotation / four-point positioning + geometric normalization" to solve inconsistent character sizes, and relying on "line-level recognition + bidirectional decoding" to solve character adhesion / sequence disorder, and using RKNN NPU acceleration to control edge-side inference latency and power consumption within target ranges, thus achieving robust localization and correct sequence recognition of serial numbers in a single process, meeting the dual requirements of online accuracy and embedded real-time performance. The method of this invention is simple, easy to implement, low-cost, and easy to promote. Attached Figure Description

[0034] Figure 1 This is a flowchart of a serial number recognition method for multiple versions of banknotes for NPU according to an embodiment of the present invention.

[0035] Figure 2 This is a flowchart of the data annotation and preprocessing stage of the serial number recognition method for multiple versions of banknotes for NPU according to an embodiment of the present invention.

[0036] Figure 3 This is a flowchart illustrating the NPU adaptation to the model quantization stage of the serial number recognition method for multi-version banknotes oriented towards NPU, according to an embodiment of the present invention.

[0037] Figure 4 This is a structural block diagram of a serial number recognition system for multiple versions of banknotes oriented towards an NPU according to an embodiment of the present invention. Detailed Implementation

[0038] This embodiment takes a serial number recognition method and system for multiple versions of banknotes for NPU as an example. The invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0039] Please see Figure 1 This invention illustrates a serial number recognition method for multi-version banknotes in NPU-based NPU, applicable to serial number recognition in banknote images with multiple versions, poses, and complex imaging conditions. The method specifically includes the following steps: Step 1, Data Labeling and Preprocessing Stage: For the banknote image to be identified, the serial number text band is labeled using a four-point / rotated frame, and the corresponding complete label string is retained; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text bands of arbitrary size and tilt angle into a standardized ROI with a fixed height; Step 2, Model Training Phase: Utilize YOLOX's rotation / four-point regression to achieve robust text band localization, and train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, the segmentation difficulties caused by character adhesion are avoided at the sequence level. Step 3, NPU adaptation and model quantization stage; QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints are performed on the detection and recognition link to convert the exported ONNX model into an NPU-friendly RKNN model. Step four, edge-side inference stage: The integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" is executed on the NPU. The CPU only undertakes light CTC decoding and syntax constraints, outputting the serial number string and confidence score, thus meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

[0040] Specifically, the integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" describes an end-to-end link: detect the output text with position (rotated box / four-point) → perform WarpAffine / perspective normalization based on geometric parameters → send it to the row-level recognition network to output sequence probability.

[0041] The aforementioned operator graph first prepares for graph optimization in the operator replacement and fusion steps: Conv + BN + Act fusion, and replacing NPU-unsupported operators with NPU-friendly operators such as WarpAffine, and externalizing rotation NMS / CTC decoding to the CPU / hardware unit to ensure chainability. Then, in the continuous subgraph concatenation step, "YOLOX detection → geometric normalization → row-level recognition" is solidified into a single computation graph within the graph (this is the key action of "forming integration"). Finally, in the edge-side inference stage, this integrated operator graph is run using the "whole graph inference" method of the RKNN / NPU runtime.

[0042] The overall process of the serial number recognition method for multi-version banknotes for NPU in this embodiment is as follows: Figure 1 As shown, in the data annotation and preprocessing stage, for banknote images with multiple versions, poses, and complex imaging conditions, four-point / rotated bounding boxes are used to annotate the serial number text bands, and the corresponding complete labels are retained. During inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text bands of arbitrary size and tilt angle into a standardized ROI with a fixed height. Subsequently, in the model training stage, YOLOX's rotation / four-point regression is used to achieve robust text band localization, and a row-level recognition model (such as a lightweight CRNN-CTC) is trained on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, the segmentation difficulties caused by character adhesion are avoided at the sequence level. Next, in the NPU adaptation and model quantization stage, QATINT8 quantization, Conv+BN+Act fusion, and static input shape constraints are performed on the detection and recognition links to derive the ONNX model and convert it into an RKNN NPU-friendly model. Finally, in the edge-side inference stage, the integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" is executed on the NPU. The CPU only handles lightweight CTC decoding and syntax constraints, outputting the serial number string and confidence score. This ensures that while size changes are controllable and character concatenation can be resolved, the real-time and stability requirements of embedded systems are met. The following is a detailed description of each stage: I. Data labeling and preprocessing stage.

[0043] Please see Figure 2 The flowchart shows the data annotation and preprocessing stage. In this stage, we complete the entire process from sample organization to geometric normalization under the conditions of multiple currencies, multiple versions, multiple denominations and multiple collection conditions (pose, lighting, wear, reflection) to support the training of subsequent detection and recognition models and the deployment of edge quantization.

[0044] First, data preparation and sample organization were carried out. Based on established acquisition specifications, banknote images were collected covering typical conditions such as size differences, tilt angle changes, specular reflection, complex anti-counterfeiting patterns, and character adhesion, continuous strokes, and partial occlusion. A unique ID was generated for each image, and acquisition metadata (device model, resolution, exposure, gain, light source conditions, pose information, etc.) was recorded for training statistics and subsequent problem tracking. To avoid information leakage, only representative difference frames (significantly different poses or lighting) were retained from multiple frames of the same banknote, and redundant frames were discarded.

[0045] Next, the geometric annotation and labeling of the prefix text bands are completed. For each image, only the prefix "text band" is annotated without character-level segmentation. The annotation format is four-point coordinates or a rotated rectangle; the four points are stored in clockwise order according to the image coordinate system, with the starting point fixed at the top left corner; if the text band has slight curvature, it is annotated using an envelope quadrilateral, and the boundary is extended outward by 1-10 pixels as needed to avoid truncation during subsequent transformations. Simultaneously, the entire string of character labels and text direction (RTL / LTR) are recorded for each annotation box. When it is RTL, it is entered from right to left according to the actual reading order. To unify the training character set, a one-to-one mapping table is established between Arabic-Hindi numerals and Arabic numerals, with the original characters and mapped characters stored in two columns for easy backtracking and evaluation.

[0046] Next, semantic priors and consistency checks are added. Based on the currency and version, the corresponding grammatical priors (number of bits, allowed character sets, alphanumeric combination patterns, and optional check bit rules) are generated and stored. These priors are used to constrain candidate paths and improve accuracy in subsequent decoding stages. An automatic verification script is run on the labeled data to check the order and closure of the four points, minimum area and aspect ratio thresholds, and consistency between text direction and label. At least 10% of the samples are randomly selected for manual review, and the quadrilateral IoU and string consistency rate are calculated. Samples below the threshold are entered into the rework queue.

[0047] Subsequently, geometric normalization is performed to form a standardized input. Based on the four-point coordinates of each ROI, a perspective or affine transformation matrix is ​​calculated, and the text strip is normalized into a strip image with a fixed height H (32 or 48 pixels recommended) and adaptive width. Bilinear interpolation is used to balance sharpness and smoothness. Lightweight preprocessing (local contrast enhancement, gamma correction, specular suppression, little kernel denoising, etc.) can be selectively performed without disrupting the topology of the character edges. To improve training and parameter tuning efficiency, the normalized text strip can be cached on disk as direct input to the recognition sub-model.

[0048] Next, the dataset was partitioned and difficult examples were configured. Stratified sampling was performed according to "currency × version" to divide the samples into three parts: training, validation, and testing, ensuring that the same physical banknote was not assigned across sets. The proportion of difficult examples (extreme size, obvious tilt angle, strong reflection, adhesion, occlusion) in each set was kept basically consistent, and appropriate weighting was given during training sampling to ensure the visibility of difficult examples during the training process.

[0049] Finally, a representative calibration subset is constructed for edge quantization and NPU compilation. The self-validation set is selected to cover samples with "extreme size, tilt angle variation, strong reflection, adhesion, and occlusion", forming a calibration list of no less than 1000 normalized text bands and several original images (≥200 images recommended) for QAT or post-quantization calibration, ensuring that the accuracy of INT8 quantization does not significantly decrease in long-tail scenes.

[0050] At this point, the output of the data annotation and preprocessing stage includes: original images and acquired metadata, four-point / rotation annotations for serial numbers, complete character labels and direction markers, syntax prior files, normalized text with cache, training / validation / test split lists, hard example statistics and quality control reports, and a quantization calibration subset index. This output will directly support subsequent model training, NPU adaptation, and edge deployment.

[0051] II. Model Training Phase 1) During the training phase, we first utilize the completed four-point / rotated bounding box annotations to perform k-means++ clustering on the width, height, and aspect ratio of the prefix targets, generating a set of anchor boxes that better match the slender text bands. Then, using YOLOX as the main framework, we modify the detection head to support regression (RBox / Quad) for rotated boxes / four-point vertices. The regression loss uses SkewIoU (Intersection over Union) / GIoU (Generalized IoU), and is trained in conjunction with classification / confidence branches. Multi-scale training is used on the input side (e.g., randomly selected between 608, 640, and 672), and lightweight image enhancement is added to make the model more robust to size changes, shooting pose, and lighting perturbations. After the detection model converges, it outputs the four-point coordinates of the prefix, and then uses perspective / affine transformations to unify the text band to a standardized ROI (Region of Interest) of a fixed height.

[0052] (1) The calculation process of anchor box clustering (k-means++, IoU distance) is as follows: To obtain a prior shape set that better matches the elongated text band and to make the regression more stable, this invention sets the size set of the truth boxes in the training set as follows: The anchor box centers are solved using k-means++ clustering with IoU as the similarity criterion. Wherein, the anchor frame center refers to the cluster center (prototype shape) in the width and height space, rather than the center point on the image coordinates; k is the cluster center / anchor frame prototype index; K is the number of cluster centers; : Width of the k-th annotation box (the thin text band with prefixes) (usually a pixel value or a value normalized to the input scale); : The height of the k-th bounding box. By using the width and height distribution of real text bounding boxes in the training set, we select K anchor box prototypes that best represent the "slender box shape", so that each ground truth box can be matched by a prototype with the highest possible IoU, thereby making the detection regression more stable and converge faster.

[0053] The goal of cluster optimization is:

[0054] The above formula sums the matching costs of each true sample to obtain the overall cost. Wherein, By adjusting all cluster centers (i.e., the width and height of each anchor frame prototype) to minimize the overall cost.

[0055]

[0056] Where K∈[6,9] is selected through the validation set, and the IoU distance 1−IoU(⋅,⋅) is used to highlight the consistency between the elongated box and the prior shape; D(⋅): Distance / cost, representing the "dissimilarity" between the truth box shape and a certain anchor box shape; the smaller the value, the more similar they are. IoU(⋅,⋅): Intersection over Union (IoU). A larger IoU value indicates a closer fit between the two bounding boxes. Converting "similarity (the higher the better)" into "distance (the smaller the better)" facilitates minimization optimization; box(w,h): A "shape box" constructed with the given width and height. In anchor box clustering, by default, it compares two boxes with the same center / position, only comparing the differences in width and height (without introducing positional factors). : The shape frame corresponding to the k-th anchor frame prototype.

[0057] (2) The calculation process of quad regression and decoding is as follows: Some currency serial numbers exhibit significant variations in tilt angle and aspect ratio (e.g., Arabic coin, Japanese yen). Using only horizontal rectangles degrades both IoU and localization gradient. RBox allows the network to directly predict the center, scale, and rotation angle, thus representing the tilted text band with minimal parameters. Sigmoid is used to limit the center offset to within the grid cells, exponential mapping ensures positive width and height, and tanh limits the angle. This is beneficial for convergence to numerical stability. The specific calculation formula is shown below:

[0058]

[0059] in, The center offset (logit or regression value) predicted by the network represents the offset trend of the center point within the current grid cell; (i, j) is the grid cell index (column / row number on the feature map) where the current prediction point is located. (i, j) determines the approximate location (coarse localization) of the corresponding input map of the prediction point. x, y: Center coordinates of the predicted bounding box (text band). In engineering, we usually first obtain the "normalized coordinates / feature map coordinates relative to the input map" and then map them to the pixel domain (multiplied by the input size or stride). w, h: Width and height of the prediction box (the actual size of the text band covering the image); tw, th: The width and height regression values ​​of the network output (usually logarithmic scale changes). The prior dimensions of the anchor (width and height) provide a reasonable benchmark for the width and height regression, thereby improving the convergence speed and stability. The scaling factor (related to feature map size / stride) that maps feature map coordinates to the input map coordinate system. Orientation angle The angle regression value is the output of the network.

[0060] (3) The calculation process of rotation IoU (SkewIoU) is as follows: If horizontal IoU is still used for matching and regression supervision, it will penalize insufficient angular bias. SkewIoU directly measures the geometric overlap between rotated boxes / quadrilaterals using polygon intersection-union ratio, which is closer to the actual coverage relationship of the target. During training, a differentiable polygon area approximation algorithm or an approximation of the angle can be used to stabilize the gradient.

[0061]

[0062] in, , For predicting and rotating polygons to the ground truth; Using rotational / four-point geometric relationships to measure text overlap is more suitable for tilted and slender targets than horizontal IoU; when used for matching (positive and negative sample assignment) and regression supervision, it can reduce evaluation distortion caused by angle changes; The region is the geometric intersection of the predicted value and the true value; the greater the overlap, the more accurate the localization. The area of ​​the polygonal region (pixel area or continuous plane area); The geometric union region of the prediction and the true value is the total area covered by both, which is used to normalize the intersection area to avoid overestimating the value by only looking at the intersection.

[0063] (4) The calculation process of the rotated GIoU regression loss is as follows: When the predictions are almost non-intersecting with the ground truth (common in the early stages of training), using only SkewIoU will produce zero gradients, making it difficult to optimize. Rotation GIoU, by introducing a minimum bounding rotation polygon, addresses this issue. Even without overlap, it still generates effective gradients, driving the predictions toward the true value, accelerating convergence and reducing degenerate boxes.

[0064]

[0065] The above formula addresses the issue of "weak gradient / slow learning" that often arises when using SkewIoU (similar to IoU) alone; it uses a blanking penalty term in the enclosing region to mitigate this problem. Even a small value or 0 can provide optimization signals such as "where to move and how much to move", thereby accelerating convergence and reducing the degradation box.

[0066] in, To predict polygons, For true polygons, (Minimum circumscribed rotating polygon / enclosing polygon) It is regression loss; To separate the penalty term, the less the prediction overlaps with the true value and the further apart they are, the greater the penalty becomes; even if they do not intersect, they can still provide an effective gradient direction.

[0067] 2) The recognition part uses the normalized text strips mentioned above as input to train a lightweight CRNN (Convolutional Recurrent Neural Network) + CTC (Connectionist Temporal Classification) row-level sequence recognition model: front-end convolutions / depthseparable convolutions extract temporal features, followed by bidirectional recurrent units to model the sequence, and finally CTC loss is used at the end to complete the training without character segmentation. Considering that the writing direction may be different in Arabic scenes, the same text strip is decoded in both forward and reverse directions during inference, and the side with the higher score is selected by the average log probability of the sequence, thus adaptively performing RTL / LTR (Right-to-Left / Left-to-Right) without explicit directional labels.

[0068] (1) The calculation process of CTC loss (connection time series classification) is as follows: Arabic letters and Arabic-Hindi numerals often exhibit overlap, ligatures, and partial occlusion in real-world data acquisition, leading to highly unstable character segmentation. CTC eliminates the need for frame-level alignment and character boundaries, directly summing across all possible alignment paths in the time series. This allows the model to "step" across overlaps to output the correct string, fundamentally bypassing the segmentation vulnerability. The specific formula is shown below:

[0069] in, The negative log-likelihood is used to identify the sequence of the sample. The smaller the value, the higher the probability that the model has correctly identified the prefix string. Let B be the target sequence, and B be the mapping for "removing duplicates + removing blanks". For all paths to it; The probability of the entire path is equal to the product of the probabilities of selecting that symbol in each frame, calculated as a product of time steps. At time step n, the model pairs the symbols The predicted probability; It is a probability distribution for "character list + blank"; To sum all valid paths, all possible alignments are included in the probability of a correct sequence, thus avoiding the need for character segmentation.

[0070] (2) Bidirectional decoding scoring (RTL / LTR adaptive) Different currencies and versions may use different writing directions. Instead of introducing an error-prone direction classifier, this scheme decodes the same text band twice during inference, both forward and backward, compares the average log-likelihood, and selects the more reliable side. This allows for adaptive RTL / LTR without direction labels and reduces "whole string reverse reading" caused by direction misjudgment.

[0071]

[0072]

[0073]

[0074] in, The optimal path log score obtained by decoding in the "left to right (LTR)" direction is the overall confidence of the model for the most likely aligned path (log domain summation) under the LTR assumption.

[0075] The overall confidence score under the RTL assumption is the logarithmic score of the optimal path obtained by decoding in the "right-to-left (RTL)" direction.

[0076] To select the side with the larger score as the final output direction, the decoding result of the direction that is "more believable overall" is output, thus adapting to RTL / LTR.

[0077] 3) Throughout the training process, detection and recognition share the same set of data augmentations (scale, light perspective, noise, compression, reflection simulation, etc.). For difficult examples such as "character adhesion / strong reflection", the weights are increased in sampling or loss (Focal Loss can also be used on the detection side) to enhance the learning ability for long-tail scenes.

[0078] (1) The calculation process of classification / confidence Focal Loss is as follows: True negative samples and easily classified true positive samples usually constitute the majority and dominate the cross-entropy loss, causing the model to ignore difficult examples such as "sticky," "reflective," and "large tilt angles." Focal Loss addresses this by... Dynamically scaling the loss for easy examples focuses on difficult examples with low confidence, significantly improving robustness to long-tailed problems. The parameter controls the suppression strength, and balances positive and negative samples. The formula is shown below:

[0079]

[0080] in, The loss value for the classification / confidence branch (for a single sample). y is the class balancing factor, where y is the label value, 1 represents a true negative sample, and 0 represents a true positive sample that is easy to classify. This represents the "correct class probability" after the model has processed the data based on the true labels, and is uniformly expressed as the "probability of predicting correctly".

[0081] 4) In the validation phase, the detection model uses the rotated bounding box mAP@[0.50:0.95] and recall as the main metrics; the recognition model uses CER (Character Error Rate) / WER (Word Error Rate) as the main metrics; and the end-to-end model uses the overall string accuracy as the final metric. Simultaneously, the above metrics are statistically analyzed on three subsets: "size variation, character adhesion, and strong reflection," to check robustness and generalization performance for key challenges. Inference deduplication uses a rotated version of NMS (Non-Maximum Suppression).

[0082] Specifically, in the rotated bounding box mAP@[0.50:0.95], the IoU threshold ranges from 0.50 to 0.95 (with a step size of 0.05, for a total of 10 thresholds). AP is calculated for each threshold, and then the thresholds and categories are averaged to obtain mAP@[0.50:0.95].

[0083] On the validation set, for each IoU threshold t ; 1. Use this threshold to determine TP / FP (for rotated frames, use rotation IoU / polygon IoU); 2. The PR curve (Precision-Recall) is obtained by varying the confidence threshold. 3. Integrate / interpolate the PR curve to obtain AP(t).

[0084] The final mAP@[0.50:0.95] is obtained by averaging the 10 AP(t) (and then averaging by category).

[0085] The industry-standard COCO-style comprehensive evaluation protocol (using multiple thresholds to avoid the randomness of "looking good only at the 0.5 threshold").

[0086] The numerical value is fixed, and mAP is the average precision, which ranges from 0 to 1.

[0087] 5) After training is completed, output the weight file (FP32 / FP16), geometric normalization parameters and input specifications for detection and recognition, as well as the inference configuration (character set / mapping, orientation adaptation and decoding strategy, detection threshold and NMS configuration, etc.), which will serve as direct inputs for subsequent NPU quantization and deployment.

[0088] III. NPU Adaptation and Model Quantization Stage Please see Figure 3 The flowchart illustrates the NPU adaptation and model quantization stages. The trained .pt model needs to be adapted to the NPU first. Therefore, this invention first exports the detection module (YOLOX-RBox / Quad) and the recognition module (CRNN-CTC) from PyTorch to ONNX, fixes the input size, and completes the necessary operator replacement / fusion to enable "detection → geometric normalization → recognition" to be chained into a continuous subgraph on the chip side. Then, quantization-aware training (QAT) is performed on the ONNX model: quantization / dequantization (Q / DQ) nodes are inserted into the training framework, and representative data is used for calibration and fine-tuning to make the INT8 precision close to floating point. After completion, the mixed precision settings of key layers (such as weight per-channel INT8, activation per-tensor INT8, and necessary layer FP16) are retained, and then the quantized ONNX is converted to RKNN format and compiled using rknn-toolkit.

[0089] 1) Model Export In this step, the patent first exports the detection subnet (YOLOX-RBox / Quad) and the recognition subnet (CRNN-CTC) from PyTorch to ONNX, unifies and fixes the input size, and clarifies the channel order and normalization method in the export script to ensure that the end-side preprocessing is aligned. Finally, it performs small sample consistency verification against onnxruntime to confirm that the box position / angle and recognition results are consistent with the PyTorch baseline before proceeding to the next step.

[0090] 2) Operator replacement and fusion After the model is exported, this patent performs graph optimization on the exported ONNX. First, it merges the three operators—convolutional layer (Conv), normalization layer (BN), and activation layer (ReLU)—into a single operator. These three can be mathematically combined into an equivalent convolution.

[0091] In this context, the scaling and translation of BN can be directly folded into the weights and biases of the convolution; x is the input feature tensor / input feature map of the fusion operator of the equivalent convolution, which comes from the output of the previous layer or from the tensor after preprocessing the input image; y is the output feature tensor / output feature map of the fusion operator, which is the result after equivalent convolution + activation and is used by the next layer.

[0092] The main functions of the above formula are: 1. Reduce the number of operators and memory access: Fold the three operators into a single convolution kernel for execution, reducing kernellaunch and reducing intermediate feature write-back / read-back.

[0093] 2. Improve NPU compilation and execution efficiency: NPU is better at regular convolution operators, and the fused graph is more regular and easier to be optimized by rknn-toolkit / NPU compiler.

[0094] 3. Reduce edge latency and power consumption fluctuations: Fewer operators and less tensor transport result in more stable edge graph inference.

[0095] After the operator folding is complete, unsupported operators in the model need to be replaced with NPU-friendly operators. For example, the grid_sample operator used to crop and normalize the rotated bounding box (RBox / Quad) to an upright rectangular region should be replaced with the NPU-friendly WarpAffine. Ideally, post-processing that may not be supported, such as rotation NMS and CTC decoding, should be left in the CPU / chip built-in unit to ensure that "detection → geometric normalization → recognition" can be connected into the same computation graph.

[0096] 3) Continuous subgraph concatenation After completing operator folding and fusion and externalizing rotation NMS and CTC decoding to the CPU / hardware unit, the next step is to string together "detection → geometric normalization → recognition" into a single computation graph and solidify it as follows: First, the rotation frame / four-point coordinates output by the detection head are unified in a clockwise order and normalized according to the input size; then, a geometric transformation matrix is ​​directly generated based on these four points in the image, and a strip-shaped ROI with fixed height and adaptive width is output. Then, the ROI tensor is seamlessly integrated into the recognition subnet input, and multiple targets in the same original image are stacked in batches within the image to reduce multiple startups and transfers. Zero-copy / shared buffering is enabled at runtime to prevent intermediate results from falling back to the CPU. Finally, visualization and consistency verification are completed with a small sample (ROI is not inverted / truncated, and the recognition string is consistent with the offline result). The p50 / p90 latency and memory usage of the entire chain are recorded. After reaching a benchmark that is better than segmented execution, it is used as the deployment configuration.

[0097] 4) Quantitative perception training To align with the target hardware quantization rules, the technical solution of this invention employs symmetric INT8 for convolution weights based on the output channel and layer-by-layer asymmetric INT8 for intermediate activations. It also uses dummy quantization nodes to accurately simulate scaling factors and zeros, making the forward pass equivalent to fixed-point operations on the edge. To improve numerical stability, the technical solution freezes batch normalized statistics during the fine-tuning phase, maintaining the SkewIoU / GIoU and Focal loss on the detection side and the CTC loss on the recognition side unchanged. Training is driven by a calibration set with extreme coverage sizes, tilt variations, strong reflections, and character adhesion, increasing the proportion of difficult examples in each batch to enhance long-tail adaptation.

[0098] To reduce fluctuations caused by quantization in critical layers, the technical solution of this invention uses a mixed-precision whitelist to retain the first, last, and bidirectional loop layers as FP16, while the remaining layers execute INT8 to balance accuracy and computational power. To stabilize convergence after quantization, the technical solution uses a small learning rate for short-cycle fine-tuning and sets early stopping, continuously monitoring end-to-end overall accuracy, detecting mAP, and recognizing CER / WER regression. Convergence is achieved when the accuracy requirements are met. To ensure a smooth deployment path, the technical solution synchronously exports the scaling factor and zero point of each layer after convergence, solidifying the Q / DQ configuration as authoritative quantization parameters for direct input to subsequent RKNN conversion and compilation, thereby achieving near-floating-point accuracy and stable latency and power consumption on the NPU.

[0099] 5) Model to RKNN In this step, the quantized ONNX model obtained in step 4) is converted into an RKNN model using the rknn-toolkit for subsequent board-side inference.

[0100] IV. End-side reasoning stage After obtaining the RKNN model, the first step is to install the chip's official SDK and rknn-toolkit runtime, and ensure that the development board firmware and driver versions are compatible. To avoid frequent memory allocation and release, input / output buffers are pre-allocated and zero-copy or shared buffer strategies are set. Then, an RKNN instance is created on the device, the .rknn model file is loaded, runtime options (number of threads, core binding, memory strategy) are set, and nit_runtime() is called to complete initialization. After initialization, a "no-load run" is performed to warm up the kernel cache and constant folding results, stabilizing the first frame latency.

[0101] After initialization, the original image is input, scaled and filled to the fixed input size of the detection network, and color channel normalization is performed in the same way as during training. Then, inference is called to run the entire computation graph. The rotated bounding box / four-point coordinates output by the detection head directly drive the WarpAffine within the graph to generate a fixed-height strip-shaped ROI, which is then directly entered into the recognition subnetwork within the graph. The intermediate tensors are passed within the NPU without needing to return to the CPU.

[0102] Specifically, the meaning of color channel refers to the channel dimension and its order / color space of the input image. The most common are RGB or BGR (three channels), and it may also include grayscale (single channel) or with alpha (four channels).

[0103] The role of color channels: to represent the multi-channel color information of the input image and keep it consistent with the preprocessing during the training stage, thereby ensuring the stability of the network input distribution, improving the ability to distinguish between characters and background, and avoiding the decrease in detection and recognition accuracy and the amplification of quantization error caused by inconsistent channel order / layout on the edge side.

[0104] After the NPU completes recognition, the CPU (or hardware unit) performs rotation NMS: sorting by confidence level, deduplication based on rotation IoU, and outputting the final four-point bounding box and its confidence level. Then, CTC decoding is performed on the recognition output, the beam width is set, and forward and reverse decoding are performed simultaneously. The log-likelihood is used to select the better side, and adaptive RTL / LTR is applied. If the confidence level of LTR is higher, the string is output normally from left to right; otherwise, the string is output from right to left.

[0105] The improvement of the prior art by the technical solution of the present invention: 1. Real-time performance and edge-side integrated reasoning By combining operator substitution and graph fusion, the entire "YOLOX detection → geometric normalization → row-level recognition" chain is solidified into an integrated operator graph, which runs as a whole within the NPU, enabling zero-copy and shared buffering to avoid frequent CPU intervention. Lightweight post-processing (rotation NMS, CTC decoding) is performed only at the end, achieving low latency, high throughput, and stable power consumption on the edge, meeting the requirements of real-time scenarios.

[0106] 2. Geometric normalization enhances character scale robustness. During the inference phase, perspective / affine geometry normalization is performed using the detected four-point coordinates to unify text bands of arbitrary size and angle to a fixed-height ROI, with adaptive width expansion to ensure consistent vertical alignment of characters. This ensures the recognition model always receives input of a uniform scale, significantly enhancing its robustness to characters of varying sizes and proportions.

[0107] 3. Two-way decoding solves the problem of character concatenation. The recognition subnet employs a lightweight CRNN-CTC, and bidirectional decoding (both forward and backward) is enabled during the decoding stage. A better path is selected based on log-likelihood, and adaptive RTL / LTR is used. This avoids the difficulty of character-by-character segmentation at the sequence level, even in cases of string concatenation or uncertain direction, achieving robust decoding.

[0108] The following are specific embodiments of this technical solution: (1) Constructing the dataset: The collected banknote images cover multiple versions, poses, and complex imaging conditions, including typical scenarios such as size differences, tilt variations, specular reflections, complex anti-counterfeiting patterns, and character adhesion / partial occlusion. Each image is uniquely identified and its metadata (device model, resolution, exposure parameters, light source conditions, pose information, etc.) is recorded. During the annotation phase, only the banknote serial number text band is annotated using a four-point / rotated frame method, and the corresponding complete character label and text direction (RTL / LTR) are saved simultaneously.

[0109] (2) Geometric normalization and preprocessing: Based on the four-point coordinates of the text strip, a perspective / affine transformation matrix is ​​calculated to normalize text strips of arbitrary size and tilt angle into standardized ROI images with a fixed height, and the width is adaptively expanded. Simultaneously, lightweight preprocessing (contrast enhancement, gamma correction, little kernel denoising, etc.) is performed as needed to improve the robustness of the subsequent recognition model.

[0110] (3) Model training: The detection subnet employs an improved YOLOX algorithm, adding a rotated bounding box / four-point regression branch to ensure robust localization of skewed and distorted text bands. The recognition subnet uses a lightweight CRNN-CTC to perform line-level character sequence recognition on normalized ROIs. The decoding stage uses bidirectional CTC (forward + backward), selecting the optimal path based on log-likelihood, thus adapting to both RTL and LTR directions and avoiding segmentation difficulties caused by inconsistent character sizes or character adhesion.

[0111] (4) NPU adaptation and model quantization: The trained detection and recognition subnets were exported as ONNX models, with standardized input dimensions and channel order to ensure alignment with edge preprocessing. During graph optimization, Conv+BN+Act were fused into a single convolution operator, and related operators were replaced to ensure the geometric normalization process was executed on the NPU. Quantization-Aware Training (QAT) was then performed, using symmetric INT8 for convolution weights and asymmetric INT8 for activations. Fine-tuning was driven by a calibration subset covering extreme scenarios, while retaining key layers (first, last, and recurrent layers) as FP16 to balance accuracy and performance.

[0112] (5) End-side inference and result output: The quantized ONNX model is converted to an RKNN model using the rknn-toolkit and loaded onto the embedded development board. During inference, the input image is preprocessed within the edge SDK and then fed into the integrated NPU for computation: the detection head outputs a rotated bounding box / four-point coordinates, directly driving WarpAffine to generate ROIs and sending them into the recognition subnet. Intermediate results are kept within the NPU with zero-copy transfer to avoid falling back to the CPU. Finally, rotation NMS and lightweight CTC decoding are performed on the CPU side, outputting the serial number string and its confidence score.

[0113] Furthermore, a serial number recognition system for multiple versions of banknotes for NPU, used to implement the serial number recognition method for multiple versions of banknotes for NPU as described above, comprising: The data annotation and preprocessing module is used to annotate the serial number text strips of the banknote image to be identified using a four-point / rotated frame and retain the corresponding complete label string; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text strips of arbitrary size and tilt angle into a standardized ROI of fixed height; The model training module is used to achieve robust text band localization using YOLOX rotation / four-point regression and to train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, it avoids the segmentation difficulties caused by character adhesion at the sequence level. The NPU adaptation and model quantization module is used to perform QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints on the detection and recognition link, and convert the exported ONNX model into an NPU-friendly RKNN model. The edge-side inference module is used to execute the entire graph of the integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" on the NPU. The CPU only undertakes lightweight CTC decoding and syntax constraints, outputting the serial number string and confidence score, thereby meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

[0114] The beneficial effects achieved by the technical solution of this invention are as follows: In response to the problems of missed detection, false detection and disordered sequence of serial numbers of currencies such as Arabic coin under conditions such as large changes in size and position, character adhesion / connection, and interference from background reflection / anti-counterfeiting patterns, the traditional templated two-stage OCR is prone to such problems. A template-independent end-to-end recognition method is proposed to complete the robust positioning and correct sequence recognition of serial numbers in a single process, significantly reducing false alarms and disordered sequence, and using NPU acceleration to meet the real-time and stability requirements of embedded / edge terminals.

[0115] First, because the serial numbers of currencies such as Arabic numerals exhibit significant size and tilt differences under different versions / collection postures, fixed rectangular ROIs are prone to truncation or offset, leading to the accumulation of downstream recognition errors. Therefore, this invention uses four-point / rotated box annotation on the data side and YOLOX rotation / four-point regression to achieve robust localization of text bands during the detection stage. After localization, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text bands of arbitrary size, tilt, and slight curvature into a standardized ROI of fixed height, eliminating the impact of inconsistent character sizes on recognition stability from the source. Secondly, because character adhesion / connected strokes commonly occur in real-world collections, disrupting character-level segmentation and triggering cross-reading and sequence errors, this invention does not perform single-character segmentation. Instead, it employs row-level sequence recognition (such as lightweight CRNN-CTC) on the standardized ROI and implements forward / backward bidirectional decoding, automatically selecting the actual writing direction with sequence logarithmic probability (adaptive RTL / LTR), bypassing pseudo-boundaries at adhesion points and ensuring correct sequence from the sequence modeling level. Subsequently, due to the strict constraints on real-time performance and power consumption in embedded / edge devices, general floating-point models and scattered operators are difficult to run efficiently. This invention performs Quantization-Aware Training (QAT) after training convergence and completes Conv+BN+Act fusion, static input shape constraints, and replacement of unsupported operators, deriving an NPU-friendly RKNN model. During deployment, INT8 quantization, zero-copy memory, and multi-threaded pipelines are enabled, allowing the operator graph of detection → normalization → recognition to be executed continuously on the NPU, avoiding CPU rollback and unnecessary data transfer. Finally, relying on "rotation / four-point positioning + geometric normalization" to solve the problem of inconsistent character sizes, and relying on "line-level recognition + bidirectional decoding" to solve the problem of character adhesion / sequence disorder, and using RKNN NPU acceleration to control the latency and power consumption of edge-side inference within the target range, thereby achieving robust positioning and correct sequence recognition of serial numbers in a single process, meeting the dual requirements of online accuracy and embedded real-time performance.

[0116] It should be noted that the present invention is not limited to the above-described embodiments. Based on the inventive spirit of the present invention, those skilled in the art can make other changes, and these changes made in accordance with the inventive spirit of the present invention should be included within the scope of protection claimed by the present invention.

Claims

1. A serial number recognition method for multi-version banknotes for NPU, used for serial number recognition in banknote images with multiple versions, multiple poses and complex imaging conditions, characterized in that, Specifically, the following steps are included: Step 1: Data labeling and preprocessing stage; For the banknote image to be identified, the serial number text band is marked using a four-point / rotated frame, and the corresponding complete label is retained; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text bands of arbitrary size and tilt angle into a standardized ROI with a fixed height; Step 2, Model Training Phase: Utilize YOLOX's rotation / four-point regression to achieve robust text band localization, and train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, the segmentation difficulties caused by character adhesion are avoided at the sequence level. Step 3, NPU adaptation and model quantization stage; QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints are performed on the detection and recognition link to convert the exported ONNX model into an NPU-friendly RKNN model. Step four, edge-side inference stage: The integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" is executed on the NPU. The CPU only undertakes light CTC decoding and syntax constraints, outputting the serial number string and confidence score, thereby meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

2. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 1, characterized in that, Step one, the data annotation and preprocessing stage, includes the following specific steps: Step 1.1: Data preparation and sample organization; according to the acquisition specifications, collect banknote images covering typical working conditions such as size differences, tilt angle changes, specular reflection, complex anti-counterfeiting background patterns, character adhesion / connection, and partial occlusion; generate a unique ID for each image and record acquisition metadata; the acquisition metadata includes device model, resolution, exposure, gain, light source conditions, and posture information; Step 1.2: Complete the geometric annotation, label recording, and direction marking of the serial number text band; for each banknote image, only annotate the serial number text band without character-level segmentation, and the annotation form is four-point coordinates or a rotated rectangle; the four points are stored in clockwise order according to the image coordinate system, with the starting point fixed at the upper left corner; if the text band has slight curvature, it is annotated in the form of an envelope quadrilateral, and the boundary is expanded by a predetermined number of pixels to avoid truncation in subsequent transformations; simultaneously record the entire string of character labels and text direction RTL / LTR for each annotation box: when it is RTL, it is entered from right to left according to the actual reading order; establish a one-to-one mapping table for Arabic-Hindi numerals and Arabic numerals, and save the original characters and mapped characters in two columns; Step 1.3: Supplement semantic priors and consistency checks; Based on the currency and version, generate and store the grammatical priors corresponding to the samples. The grammatical priors include the number of bits, allowed character sets, alphanumeric combination patterns, and optional check bit rules; Run an automatic verification script on the labeled data to check the order and closure of the four points, the minimum area and aspect ratio thresholds, and the consistency between the text direction and the label; Randomly sample a predetermined proportion of samples for manual review and calculate the quadrilateral IoU and string consistency rate. Samples below the threshold are entered into the rework queue. Step 1.4: Perform geometric normalization to form a standardized input; calculate the perspective or affine transformation matrix based on the four-point coordinates of each ROI, and normalize the text strip into a strip image with a fixed height H and adaptive width. Use bilinear interpolation to balance sharpness and smoothness; perform lightweight preprocessing without destroying the topology of the character edges, including local contrast enhancement, gamma correction, specular suppression, and little kernel denoising; cache the normalized text strip on disk as the direct input to the recognition sub-model. Step 1.5: Complete the dataset partitioning and hard example configuration; stratify the sampling by "currency × version" to divide the samples into three parts: training, validation, and testing, ensuring that the same physical banknote is not distributed across sets; keep the proportion of hard examples in each set basically consistent for various typical working conditions, and install a predetermined weighting ratio during training sampling to ensure the visibility of hard examples during the training process. Step 1.6: Construct a representative calibration subset for edge quantization and NPU compilation; screen samples covering various typical working conditions to form a calibration list of no less than 1000 normalized text bands and several original images for QAT or post-quantization calibration, ensuring that the accuracy of INT8 quantization does not significantly decrease in long-tail scenarios.

3. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 1, characterized in that, Step two, the specific steps of the model training phase, include: Step 2.1: The detection part uses YOLOX rotation / four-point regression to achieve robust text band localization and obtain a standardized region of interest (ROI) with a fixed height; Step 2.2: The recognition part trains a row-level recognition model on the normalized ROI and uses forward / backward bidirectional decoding to adapt RTL / LTR. Step 2.3: Throughout the training process, the detection and recognition parts share the same set of data augmentations, which include scale adjustment, light perspective adjustment, noise / compression adjustment, and reflection simulation. For difficult cases of various typical working conditions, the weights are increased in the sampling or loss to enhance the learning ability for long-tail scenes. Step 2.4, in the validation phase, the detection model uses the rotated bounding box mAP@[0.50:0.95] and recall as the main indicators; the recognition model uses the character error rate (CER) and word error rate (WER) as the main indicators; the end-to-end model uses the whole string accuracy as the final indicator; at the same time, the above indicators are statistically analyzed on three subsets: size change, character adhesion, and strong reflection, to check the robustness and generalization performance of key difficulties; the inference deduplication uses the rotated version of nonmaximum suppression (NMS); Step 2.5: After training is completed, output the weight file for detection and recognition, geometric normalization parameters and input specifications, as well as the inference configuration, which includes character set / mapping, orientation adaptation and decoding strategy, detection threshold and NMS configuration.

4. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 3, characterized in that, Step 2.1, the detection part utilizes YOLOX rotation / four-point regression to achieve robust text band localization, obtaining a standardized Region of Interest (ROI) with a fixed height. The specific steps include: Step 2.1.1: Using the completed four-point / rotated box annotations, perform k-means++ clustering on the width, height and aspect ratio of the prefix target to generate a set of anchor boxes that match the thin text band; Step 2.1.2: Using YOLOX as the main framework, the detection head is modified to support the regression RBox / Quad that supports rotating boxes / four vertices. The regression loss uses the rotation intersection-union ratio SkewIoU / generalized intersection-union ratio GIoU, and is trained together with the classification / confidence branch. Step 2.1.3: Multi-scale training is used at the input end, and lightweight imaging enhancement is added to make the model more robust to size changes, shooting posture and lighting disturbances. Step 2.1.4: After the detection model converges, output the coordinates of the four points of the prefix, and then use perspective / affine transformation to unify the text into a standardized region of interest (ROI) of fixed height.

5. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 3, characterized in that, Step 2.2, where the recognition part trains a row-level recognition model on the normalized ROI, and the specific steps of adaptive RTL / LTR through forward / backward bidirectional decoding include: Step 2.2.1: The recognition part uses the normalized text strips mentioned above as input to train a row-level sequence recognition model of lightweight convolutional recurrent neural network CRNN+connected temporal classification CTC: front-end convolution / depthseparable convolution extracts temporal features, followed by bidirectional recurrent units to model the sequence, and the end uses CTC loss to complete the training without character segmentation. Step 2.2.2: Since the writing direction of Arabic scenes may be different, the same text band is decoded in both forward and reverse directions during inference, and the side with the higher score is selected by the average log probability of the sequence, so as to adaptively right-to-left / left-to-right RTL / LTR without explicit direction labels.

6. The serial number recognition method for multi-version banknotes for NPU as described in claim 1, characterized in that, Step three, the NPU adaptation and model quantization stage, includes the following specific steps: Step 3.1, Model Export: Export the detection subnet YOLOX-RBox / Quad and the recognition subnet CRNN-CTC from PyTorch as ONNX models, unify and fix the input size, and specify the channel order and normalization method in the export script to ensure alignment of the end-side preprocessing; then perform small sample consistency verification against onnxruntime to confirm that the bounding box position / angle and recognition results are consistent with the PyTorch baseline; Step 3.2, Operator replacement and fusion; graph optimization of the exported ONNX model; Step 3.3, Continuous subgraph concatenation; concatenate "YOLOX detection → geometric normalization → row-level recognition" into a single computation graph and solidify it; Step 3.4, Quantify perception training; The convolution weights are symmetrical INT8 based on the output channel, and the intermediate activations are asymmetric INT8 layer by layer. The scaling factor and zeros are accurately simulated with dummy quantization nodes, so that the forward pass is equivalent to the fixed-point operation on the end side, in order to achieve consistency with the target hardware quantization rules. During the fine-tuning phase, the batch normalized statistics are frozen, keeping the SkewIoU / GIoU and Focal loss on the detection side and the CTC loss on the recognition side unchanged. Training is driven by a calibration set with extreme coverage size, tilt angle variation, strong reflection and character adhesion, increasing the proportion of difficult examples in each batch to strengthen long-tail adaptation and improve numerical stability. A mixed-precision whitelist is used to retain the first, last, and bidirectional loop layers as FP16, while the remaining layers are executed as INT8 to balance precision and computing power, thereby reducing the fluctuations caused by quantization in critical layers. Use a small learning rate for short-cycle fine-tuning and set early stop, continuously monitor the end-to-end whole string accuracy, detect mAP and recognize CER / WER regression, and converge and export when the accuracy meets the requirements, so as to stabilize the convergence after quantization; After convergence, the scaling factor and zero point of each layer are exported synchronously, and the Q / DQ configuration is solidified as authoritative quantization parameters, which serve as direct inputs for subsequent RKNN transformation and compilation. This achieves near-floating-point accuracy and stable latency and power consumption performance on the NPU, thus enabling the deployment process. Step 3.5, Model to RKNN: The quantized ONNX model is converted into an RKNN model using rknn-toolkit for subsequent board-side inference.

7. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 1, characterized in that, Step 3.2, the specific steps of operator replacement and fusion, include: In step 3.2.1, the three operators—convolutional layer (Conv), normalization layer (BN), and activation layer (ReLU)—are merged into one operator, which mathematically combines them into an equivalent convolution, as shown in equation (1): （1）； In this case, the scaling and translation of BN are directly folded into the weights and biases of the convolution; x is the input feature tensor / input feature map of the fusion operator of equivalent convolution, which comes from the output of the previous layer or from the tensor after preprocessing the input image; y is the output feature tensor / output feature map of the fusion operator, which is the result after equivalent convolution and activation, and is used by the next layer; In step 3.2.2, after the operator folding is completed, the operators in the model that are not supported by the NPU are replaced with NPU-friendly operators; Step 3.2.3 leaves post-processing that may not be supported by the NPU to the CPU / chip built-in unit to ensure that "YOLOX detection → geometric normalization → row-level recognition" can be connected into the same computation graph.

8. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 1, characterized in that, Step 3.3, the specific steps of concatenating continuous subgraphs, include: In step 3.3.1, the rotation frame / four-point coordinates output by the detection head are unified in a clockwise order and normalized according to the input size; a geometric transformation matrix is ​​directly generated in the figure based on the rotation frame / four-point coordinates, and a strip ROI with fixed height and adaptive width is output. In step 3.3.2, the strip ROI tensor is seamlessly connected to the recognition subnet input, and multiple targets in the same original image are stacked in batches within the image to reduce multiple startups and transfers. Zero-copy / shared buffering is enabled during runtime to prevent intermediate results from falling back to the CPU. Step 3.3.3 involves using a small sample to complete visualization and consistency verification, and recording the p50 / p90 latency and memory usage of the entire chain. Once a benchmark better than segmented execution is achieved, it is used as the deployment configuration.

9. The serial number recognition method for multiple versions of banknotes for NPU as described in claim 1, characterized in that, Step four, the specific steps of the end-side inference stage, include: In step 4.1, install the official chip SDK and rknn-toolkit runtime, and confirm that the development board firmware and driver version match. At the same time, pre-allocate input / output buffers and set zero-copy or shared buffer strategies. In step 4.2, an RKNN instance is created on the device side, the .rknn model file is loaded, the options for runtime thread count, core binding, and memory strategy are set, and nit_runtime() is called to complete the initialization. After initialization, an empty run is performed to warm up the device, fill the kernel cache and constant folding results, and stabilize the latency of the first frame. In step 4.3, after initialization is completed, the original image is input, scaled and filled according to the fixed input size of the detection network, and the same color channels and normalization as during training are performed. Then, inference is called to run the entire computation graph. In step 4.4, the rotating frame / four-point coordinates output by the detection head directly drive the WarpAffine within the image to generate a fixed-height strip ROI with adaptive width expansion, ensuring that the characters are aligned in the vertical direction; and directly enter the recognition subnet within the image, with intermediate tensors being transferred within the NPU without needing to return to the CPU; In step 4.5, after the NPU completes the recognition, the CPU or hardware unit completes the rotation NMS, sorts the data by confidence level, removes duplicates based on rotation IoU, and outputs the final four-point bounding box and its confidence level. In step 4.6, CTC decoding is performed on the recognition output, the beam width is set, and forward and reverse decoding are performed simultaneously. The better side is selected by log likelihood, and the adaptive RTL / LTR is used. If the confidence of LTR is greater, the string is output normally from left to right; otherwise, the string is output from right to left.

10. A serial number recognition system for multiple versions of banknotes for NPU, used to implement the serial number recognition method for multiple versions of banknotes for NPU as described in any one of claims 1-9, characterized in that, include: The data annotation and preprocessing module is used to annotate the serial number text strips of the banknote image to be identified using a four-point / rotated frame and retain the corresponding complete label string; during inference, perspective / affine geometry normalization is performed based on the four-point coordinates to unify text strips of arbitrary size and tilt angle into a standardized ROI of fixed height; The model training module is used to achieve robust text band localization using YOLOX rotation / four-point regression and to train a line-level recognition model on the normalized ROI. Through forward / backward bidirectional decoding and adaptive RTL / LTR, it avoids the segmentation difficulties caused by character adhesion at the sequence level. The NPU adaptation and model quantization module is used to perform QAT INT8 quantization, Conv+BN+Act fusion and static input shape constraints on the detection and recognition link, and convert the exported ONNX model into an NPU-friendly RKNN model. The edge-side inference module is used to execute the integrated operator graph of "YOLOX detection → geometric normalization → row-level recognition" on the NPU. The CPU only undertakes lightweight CTC decoding and syntax constraints, outputting the serial number string and confidence score, thereby meeting the real-time and stability requirements of embedded systems while ensuring controllable size changes and resolving character adhesion.

Images