Video Processing System and Method for Enhancing Recognition Performance via Multi-Resolution Feature Map Decomposition-Based Object Detection and Tracking and Iterative Character Region Warping

Abstract

Disclosed is an image processing system and method for improving recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing. The present invention relates to a multi-scale-based warping-corrected license plate recognition system and method for reliably recognizing license plates even in complex shooting environments. It includes a multi-scale object detection unit capable of detecting even small license plate objects in an input image, and reliably secures a license plate ROI even in situations of occlusion or motion blur through MOT tracking utilizing external features and movement trajectories. The secured ROI is precisely corrected for geometric distortion through STN-based warping correction and perspective transformation, and then restored into a high-resolution image using deep learning-based super-resolution technology. Character information is extracted from the restored license plate through OCR, and the reliability of the recognition result is enhanced using GAN-based correction and inter-frame mutual correction. Thus, the present invention provides high accuracy and stability in real-world environments by integrally configuring the entire recognition pipeline.

Description

Technology Field

[0001] The present invention relates to an image processing system and method with improved recognition performance, and more specifically, to an image processing system and method with improved recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing. Background Technology

[0002] Recently, CCTV-based video analysis technology has become essential in various fields such as traffic surveillance, smart parking systems, urban security, and road safety management; in particular, License Plate Recognition (LPR) technology is establishing itself as a core element within urban and industrial systems.

[0003] Such license plate recognition technology generally consists of steps such as vehicle or license plate area detection, geometric correction and normalization of license plate images, and optical character recognition (OCR), with each step often implemented as an independent module. However, the following problems frequently occur in complex shooting conditions, such as in real-world road environments, parking lots, and intersections.

[0004] Furthermore, the degradation of detection performance in environments with small and multiple objects is also a problem. Due to factors such as installation height, shooting distance, and changes in illumination, license plates often occupy a very small portion of the object size in CCTVs. Particularly in nighttime, backlit, or low-frame-rate environments, license plates appear as small pixels, making accurate detection difficult with typical CNN-based single-scale detectors. Additionally, in areas with a high concentration of vehicles, such as commercial districts, parking lots, and intersections, multiple objects are prone to overlapping or being obscured, leading to a high probability of detection failure.

[0005] In addition, the instability of tracking in environments with occlusion and motion blur is a concern. Existing object tracking algorithms suffer from the problem where the ID changes or disappears the moment the object detection score (confidence) drops. In particular, sudden motion blur occurring during vehicle movement, occlusion by vehicles in front or behind, and camera shake significantly degrade the continuity of license plate ROI extraction. Consequently, there was a problem where frame sequence data of the same vehicle license plate could not be accurately accumulated, and a temporary detection failure led to a complete recognition failure.

[0006] The degradation of license plate image quality caused by perspective distortion and geometric deformation is also a major problem. In most cases, CCTV recording cannot capture license plates from the front, and the following geometric issues are frequent: skew due to the vehicle's entry angle, perspective distortion caused by high-positioned CCTVs, and distortion of character areas due to light reflection, contamination, and sticker attachment. Traditional computer vision methods (based on Canny, Hough, and Contour) failed to accurately detect the four corners of the license plate when the border was unclear or partially obscured.

[0007] It has also been pointed out that the OCR recognition failure rate is high due to low resolution. Even if the resolution of the original CCTV footage is sufficient, license plates captured from a distance have a very small ROI, making it easy for character stroke information to be lost. In this case, even if geometric correction is performed, the character shape is excessively damaged, which has the limitation of being difficult to recognize with existing OCR engines.

[0008] The degradation of the overall recognition rate due to the fixed pipeline structure is also a problem. Conventional LPR systems typically consisted of modules with separate detection, correction, and recognition stages. In this structure, if an error occurred in one stage, there was no way to correct it in subsequent stages, leading to a sharp drop in the overall recognition rate. In particular, even when the same vehicle was captured across multiple frames, the system lacked inter-frame correction capabilities because each frame was processed individually.

[0009] This is a limitation issue regarding the handling of license plates damaged by reflection and contamination. If part of the license plate is obscured or damaged due to rain, snow, dust, dirt, or light reflection, existing algorithms cannot correct the missing parts, and OCR accuracy drops significantly.

[0010] As previously discussed, there are various technical limitations, and while deep learning-based restoration and correction technologies have recently emerged, there have been few proposals for an integrated structure that applies to the entire license plate recognition pipeline. Prior art literature

[0011] Patent Document 1: Korean Registered Patent No. 10-2223754 (Registration Date: February 26, 2021) Patent Document 2: Korean Registered Patent No. 10-2838099 (Registration Date: July 21, 2025) Patent Document 3: U.S. Registered Patent US10332264B2 (Registration Date: June 25, 2019) Patent Document 4: U.S. Registered Patent US11055854B2 (Registration Date: July 6, 2021)

[0012] Non-patent document 1: Silva, SM, & Jung, CR, “License Plate Detection and Recognition in Unconstrained Scenarios,” European Conference on Computer Vision (ECCV), 2018, Non-patent document 2: Chen, S.-L., Liu, Q., Ma, J.-W., & Yang, C., “Scale-Invariant Multidirectional License Plate Detection with the Network Combining Indirect and Direct Branches,” Sensors, 21(4):1074, 2021.Non-patent document 3: Nguyen, H., “A High-Performance Approach for Irregular License Plate Recognition in Unconstrained Scenarios,” International Journal of Advanced Computer Science and Applications (IJACSA), 14(3), 2023. The problem to be solved

[0013] The present invention, devised in response to the aforementioned necessity, proposes an image processing system and method that improve recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing to overcome the limitations of the prior art. Through this, it is designed to maximize step-by-step technical performance, such as enhancing detection sensitivity for small license plates, maintaining ID in obscured and low-reliability situations, precise correction of geometric deformation, and clear restoration of low-resolution license plates.

[0014] In addition, the present invention aims to provide a multi-scale based warping correction license plate recognition system and method that further improves recognition reliability by applying GAN-based restoration technology to compensate for cases where some characters are not recognized due to license plate contamination, reflection, or damage, and by including a frame-to-frame mutual correction function utilizing the characteristic that the same vehicle is captured in multiple frames. means of solving the problem

[0015] To achieve the aforementioned objective, an image processing system according to an embodiment of the present invention, which improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing, comprises: an object detection unit that detects location information of vehicle objects including small objects by analyzing vehicle or license plate candidate objects from an input image using multi-resolution feature map decomposition; a MOT-based ROI tracking unit that uses the object detection result to maintain the ID of the same object even in situations of occlusion, low resolution, and motion blur through a match association logic that combines the external features or movement trajectory information of the object, and generates a license plate candidate region as an ROI for each frame; and a geometric correction unit that performs a learning-based spatial transformation network (STN) or a perspective transformation matrix calculation module to automatically correct tilting, perspective distortion, and geometric deformation of the license plate for the ROI, and normalizes the ROI by applying an outlier removal-based model (RANSAC, etc.) to correct four reference point detection errors of the license plate. For the geometrically corrected ROI, a quality restoration unit that generates a high-resolution license plate image by performing deep learning-based super-resolution restoration to restore details of a low-resolution or small license plate image; an optical character recognition (OCR) processing unit that extracts a license plate string from the restored license plate image; and a license plate re-identification unit that determines whether there is an error by comparing the character recognition result with a previously stored license plate record of the same vehicle, and improves the reliability of license plate recognition by utilizing the existing record or generating correction information through a learning-based generative model in the event of recognition failure or license plate damage.

[0016] In this case, the object detection unit further includes a multi-scale based object detection unit having the feature of decomposing the input image into a feature pyramid feature map and combining high-resolution features of a shallow layer with semantic features of a deep layer to improve the detection sensitivity of small license plate objects.

[0017] Meanwhile, the above MOT-based ROI tracking unit includes a two-stage association logic that maintains the continuity of object IDs even in occlusion or motion blur situations by reconnecting low-reliability detection boxes that were not matched in the previous frame during an additional matching step.

[0018] Meanwhile, the above MOT-based ROI tracking unit further includes an appearance feature matching module that generates an object's appearance feature vector and determines whether it is the same object through cosine distance-based similarity measurement.

[0019] Meanwhile, the geometric correction unit includes a deep learning-based vertex detection network for detecting the four corner points of a license plate, and

[0020] An accurate perspective transformation matrix can be calculated by performing RANSAC-based outlier removal on the detected points.

[0021] Meanwhile, the geometric correction unit includes a perspective conversion module and a rotation normalization module that automatically correct the oblique shooting angle of the license plate into a form close to horizontal and front.

[0022] Meanwhile, the image quality restoration unit can improve the OCR recognition rate of the restored image by applying an edge-enhanced loss function that prioritizes the preservation of edge and stroke information of license plate characters by a super-resolution network.

[0023] Meanwhile, the image quality restoration unit includes a resolution-based conditional restoration (conditioned SR) method that automatically activates the SR restoration step when the geometrically corrected ROI is below a predefined resolution.

[0024] Meanwhile, the above OCR processing unit includes a sequence-based character recognition model (CTC-based or attention-based) optimized for the regional pattern (combination of separator lines, characters, and numbers) of the license plate.

[0025] Meanwhile, the above-mentioned license plate re-identification unit includes a match-based correction module that calculates a string similarity score between the recognition result of the current frame and the previously stored license plate information, and prioritizes the use of existing license plate information when the similarity is less than a threshold value.

[0026] Meanwhile, the above-mentioned license plate re-identification unit can restore missing characters or distorted parts using a generative adversarial network-based correction model when the license plate fails to be recognized due to contamination, reflection, or damage.

[0027] Meanwhile, the license plate re-identification unit includes a cross-frame enhancement function that utilizes high-resolution license plate information obtained from an initial frame in a situation of repeated entry of the same vehicle to improve the reliability of license plate recognition in a subsequent frame.

[0028] Meanwhile, to enable object detection, tracking, geometric correction, super-resolution restoration, and OCR recognition processes across the entire system to be performed in real time

[0029] It further includes an end-to-end integrated processing module that provides feature-level linkage between each processing unit.

[0030] An image processing method according to another embodiment of the present invention, which improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing, comprises: a step of detecting a vehicle or license plate candidate object using multi-resolution feature map decomposition in an input image; a step of calculating a license plate ROI per frame while maintaining an object ID even under environmental changes of occlusion and blurring through association analysis based on external features or movement trajectories for the detected object; a step of generating a normalized license plate image by correcting tilt and distortion / perspective transformation using a learning-based spatial transformation module or perspective transformation matrix calculation for the ROI and removing outlier points; a step of restoring the corrected license plate image to high resolution using a deep learning-based super-resolution model; a step of extracting a string from the restored license plate image; and a step of determining whether there is an error by comparing the extracted license plate information with a previous frame or past storage history, and, if necessary, outputting the corrected license plate information using existing information or a learning-based generative model. Effects of the invention

[0031] According to the present invention, the detection rate for small license plate objects can be significantly improved through multi-scale-based object detection, and the license plate ROI can be stably acquired by maintaining the ID of the same object even in complex environments such as occlusion, motion blur, and low-reliability detection through a robust ROI tracking function based on MOT. In addition, geometric distortion of the license plate can be precisely corrected by combining STN-based learning-type warping correction and perspective transformation with RANSAC-based reference point refinement technology, thereby effectively solving the problem of distortion in license plate images.

[0032] Furthermore, through a deep learning-based super-resolution restoration function, character strokes are clearly restored even in low-resolution license plate images, thereby significantly improving OCR recognition rates. Additionally, by applying GAN-based restoration and cross-frame mutual correction functions (e.g., cross-frame enhancement), highly reliable license plate information can be reconstructed by combining existing frames even in situations where some characters are not recognized due to contamination, reflection, or damage. Moreover, through an integrated architecture configured with an end-to-end approach for the entire pipeline, information loss between system components is minimized, and high recognition accuracy and reliability can be secured while maintaining real-time processing performance. Thus, the present invention has the effect of providing stable and efficient license plate recognition technology in various industrial fields, such as smart parking systems, traffic enforcement, ITS (Intelligent Transportation Systems), and control systems. Brief explanation of the drawing

[0033] FIG. 1 is a diagram exemplarily illustrating the overall configuration of an image processing system (100) that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition and character region warping and iterative reprocessing according to the present invention. FIG. 2 is a diagram exemplarily illustrating a multi-resolution feature map decomposition-based object detection unit (110) of a multi-scale based warping correction number plate recognition system according to an embodiment of the present invention. FIG. 3 is a diagram exemplarily illustrating an MOT-based ROI tracking unit (120) of a warping correction license plate recognition system based on multi-resolution feature map decomposition according to an embodiment of the present invention. FIG. 4 is a diagram exemplarily illustrating a geometric correction unit (130) of a warping correction number plate recognition system based on multi-resolution feature map decomposition according to an embodiment of the present invention. FIG. 5 is a diagram exemplarily illustrating the operation of an image restoration unit (140) of a warping correction license plate recognition system based on multi-resolution feature map decomposition according to an embodiment of the present invention. FIG. 6 is a diagram exemplarily illustrating the operation of the OCR processing unit (150) and the license plate re-identification unit (160) of a warping correction license plate recognition system based on multi-resolution feature map decomposition according to an embodiment of the present invention, and, FIG. 7 is a flowchart exemplarily illustrating a warping correction license plate recognition method based on multi-resolution feature map decomposition according to another embodiment of the present invention. Specific details for implementing the invention

[0034] Preferred embodiments of the present invention will be described below with reference to the drawings. The preferred embodiments of the present invention described below may be modified in various ways, and it is obvious that such modified embodiments fall within the scope of the technical spirit of the present invention.

[0035] FIG. 1 is a diagram illustrating the overall configuration of an image processing system (100) that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition and character area warping and iterative reprocessing according to the present invention. The system (100) of the present invention comprises: an object detection unit (110) that detects location information of vehicle objects including small objects by analyzing vehicle or license plate candidate objects from an input image using multi-resolution feature map decomposition in order to stably recognize a license plate in an input image; a MOT-based ROI tracking unit (120) that generates a license plate candidate area as an ROI for each frame while maintaining the ID of the same object even in situations of occlusion, low resolution, or motion blur through a match association logic that combines external features and movement trajectory information using the object detection results; a geometric correction unit (130) that normalizes the ROI by performing a learning-based spatial transformation network or a perspective transformation matrix calculation module to automatically correct tilting, perspective distortion, and geometric deformation of the license plate for the ROI and correcting reference point detection errors; an image quality restoration unit (140) that generates a high-resolution license plate image by performing deep learning-based super-resolution restoration to restore detailed information of a low-resolution or small license plate image for the geometrically corrected ROI; and a license plate string that extracts a license plate string from the restored license plate image. The system is configured to include an OCR processing unit (150) and a license plate re-identification unit (160) that determines whether there is an error by comparing the character recognition result with a previously stored license plate record of the same vehicle, and improves the reliability of license plate recognition by generating correction information using an existing record or a learning-based generative model in the event of recognition failure or license plate damage. An internal bus or network interface is formed between these components to exchange data streams and feature information, so that the entire pipeline can be designed to operate in real-time in a substantially end-to-end structure.

[0036] FIG. 2 is a diagram illustrating an exemplary object detection unit (110) based on multi-resolution feature map decomposition of a multi-scale-based warping correction license plate recognition system according to an embodiment of the present invention. The object detection unit (110) first generates multi-resolution feature maps from an input image using a backbone CNN (111). The generated multi-resolution feature maps are reconstructed into multiple resolutions through a feature pyramid network (FPN, 112) or a similar multi-scale feature combination structure. In this process, high-resolution spatial information at a shallow layer and rich semantic information at a deep layer are combined to effectively represent small license plate objects or vehicle objects within a complex background. Subsequently, the object detection head (113) receives the pyramid feature maps and calculates the location (bounding box) and class information of vehicle and license plate candidates for each scale.

[0037] By decomposing the multi-resolution feature map in this manner, the object detection unit (110) of the present invention provides enhanced detection sensitivity even for license plate objects that occupy a very small area within an image, such as those captured from a long distance, and can calculate stable candidate regions under various distance, illumination, and environmental conditions. The multi-resolution feature combination structure used here is not limited to FPN and can be replaced with BiFPN, PANet, Transformer-based multi-scale detection networks, etc., and the object detection method can also be modified to include various anchor-based or anchor-free deep learning models such as YOLO, RetinaNet, DETR, etc. The backbone network can also be freely replaced with a pre-trained model or a retrained model specialized for field data.

[0038] This is a diagram illustrating an exemplary MOT-based ROI tracking unit (120) of an image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character area warping, and iterative reprocessing. The MOT-based ROI tracking unit (120) calculates feature vectors using a CNN-based external feature extraction module (121) to extract external features for each object from the output of the object detection unit (110). Additionally, it evaluates the external similarity between objects using a similarity measure such as the cosine distance between the feature vectors. Meanwhile, the movement trajectory prediction module (122) predicts the position of an object in the current frame based on position and velocity information up to the previous frame, and the match association module (123) performs optimal detection-track matching by synthesizing the prediction result, detection result, external similarity, detection reliability, etc.

[0039] The tracking unit (120) according to the present invention does not simply discard low-reliability detection boxes, but rather performs a two-stage association on tracks or detection boxes that were omitted in the first matching, thereby preventing ID separation phenomena caused by occlusion, motion blur, or temporary decrease in reliability, and maintaining the continuity of the same object ID. The MOT-based ROI tracking unit (120) configured in this way stably generates license plate candidate regions in the form of ROI for each frame and provides them as input for subsequent geometric correction and image quality restoration steps. This tracking structure is not limited to algorithm implementations such as DeepSORT or ByteTrack, and can be implemented by replacing or modifying it with various multi-object tracking techniques such as SORT, OC-SORT, FairMOT, and Transformer-based TransTrack, as well as external feature matching technologies (Transformer embedding, Siamese network, graph-based tracking method, etc.), and the ROI generation cycle can also be changed to various forms such as all frames, frames at regular intervals, or event-based.

[0040] FIG. 4 is a diagram illustrating an exemplary geometric correction unit (130) of an image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing according to an embodiment of the present invention. The geometric correction unit (130) first includes a learning-based spatial transformation network module (131), and this STN module (131) predicts affine transformations or more complex warping parameters from an input license plate ROI to automatically correct the skew, rotation, distortion, etc. of the license plate. In addition, to accurately secure the four corner reference points of the license plate, it includes a deep learning-based vertex detection network (132), and for the detected reference point candidates, it removes outliers through a RANSAC-based reference point refinement module (133) and determines the optimal four corner points. Next, the perspective transformation module (134) calculates a perspective transformation matrix based on the refined reference point and repositions the obliquely captured license plate image into a form that is close to horizontal and frontal to generate a geometrically normalized license plate image suitable for subsequent character recognition. The implementation of this geometric correction unit (130) is not limited to the SNT method in the case of the STN module (131) and can be replaced with various learning type transformer structures such as Thin-Plate Spline (TPS) based transformers, Transformer-based geometric alignment networks, and CNN-based keypoint regression networks. The edge detection and refinement process can also be implemented using various algorithms such as Hough transform, LSD (Line Segment Detection), and edge alignment networks. In addition, the correction order and method can be varied in various ways, such as performing perspective transformation and rotation normalization separately or in combination.

[0041] FIG. 5 is a diagram illustrating the operation of an image quality restoration unit (140) of an image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing according to an embodiment of the present invention. The image quality restoration unit (140) receives a normalized license plate ROI output from a geometric correction unit (130) as input and improves the spatial resolution by a predetermined upscaling factor through a super-resolution network (141). This super-resolution network (141) may be composed of a deep learning model including residual blocks, upsampling layers, etc. The super-resolution network (141) applies an edge-enhanced loss function (142) to preferentially preserve character stroke and edge information during training, thereby ensuring that the character outlines in the restored license plate image remain clear. In addition, a resolution-based conditional restoration (conditioned SR) method is applied to automatically activate super-resolution restoration when the ROI resolution is below a predefined standard resolution, thereby efficiently using computational resources while performing high-quality restoration only when necessary. This image quality restoration unit (140) can be implemented with various super-resolution networks such as EDSR, SwinIR, Real-ESRGAN, in addition to SRGAN and ESRGAN, and may be modified to determine whether to restore based on image quality indicators such as the degree of blur or illumination conditions as well as resolution conditions, and may include a lightweight configuration based on quantization, pruning, and hardware acceleration (TensorRT, etc.) for real-time processing.

[0042] FIG. 6 is a diagram illustrating the operation of an OCR processing unit (150) and a license plate re-identification unit (160) of an image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing according to an embodiment of the present invention. Referring to FIG. 6, the OCR processing unit (150) includes a sequence-based character recognition model (151) designed considering the character arrangement structure of the license plate, character types (numbers, Korean characters, English characters, etc.), and dividing line patterns. For example, characters can be stably extracted from a license plate image in the form of a sequence using a CTC-based structure or an attention-based structure, and can be implemented in various ways such as a CRNN, OCR-Transformer, or Vision Transformer-based character recognition model. For the recognized string, post-processing techniques such as dictionary-based correction and N-gram-based verification can be applied to reduce additional errors.

[0043] The license plate re-identification unit (160) includes a comparison and verification module (162) that calculates string similarity or pattern matching by comparing the license plate string of the current frame transmitted from the OCR processing unit (150) with a license plate information database (161) stored for the same vehicle in the past. If the similarity value is greater than or equal to a reference value, the license plate re-identification unit (160) determines that the recognition result is reliable and outputs it as a final result; if it is less than the reference value, it determines that there is a high possibility of recognition error and performs a correction procedure. At this time, if it is determined that some characters are missing or distorted due to factors such as contamination, reflection, or damage to the license plate, a GAN-based restoration module (163) generates a corrected license plate image using the past high-resolution license plate image of the same vehicle or partial information obtained from multiple frames, and corrects the recognition result by re-OCRing it or replacing the partial string. In addition, the cross-frame enhancement module (164) utilizes the characteristic that the same vehicle is repeatedly captured across multiple frames to reflect high-quality license plate information obtained from the initial or highest quality frame into the recognition process of subsequent frames, thereby minimizing recognition failures caused by temporary image quality degradation or occlusion. The string similarity calculation method can be implemented with various character-based or vector-based algorithms such as Levenshtein Distance, Cosine Similarity, and Jaro-Winkler, and the GAN-based restoration model can also be freely replaced with Pix2Pix, CycleGAN, StyleGAN, and a diffusion-based model.

[0044] FIG. 7 is a flowchart exemplarily illustrating an image processing method that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing according to another embodiment of the present invention. Referring to FIG. 7, the license plate recognition method begins with a step (S710) of detecting a vehicle or license plate candidate object from an input image using multi-resolution features. In this step, the input image is decomposed into multiple layers of multi-resolution feature maps, and by combining high-resolution spatial information from shallow layers and semantic features from deep layers, improved detection sensitivity can be secured even for license plate objects that have a small size due to long-distance shooting or environmental changes. The detection method is not limited to an FPN-based multi-scale feature combination method and can be extended to apply BiFPN, PANet, HRNet, or Swin Transformer-based multi-scale attention structures. In addition, the detection model can be replaced not only with YOLO and DETR series but also with a complete Transformer-based DETR-like structure that does not generate object candidates, and can also be implemented as a lightweight detector based on MobileNet-EdgeTPU optimized for mobile environments to reduce computational load.

[0045] Next, in the step of calculating the ROI (S720), match association based on external features or movement trajectories is performed for each detected object to maintain the same object ID even with environmental changes such as occlusion, blurring, and motion blur. In this step (S720), external embeddings and movement trajectory predictions are combined through CNN-based feature extraction, and missing detection boxes are supplemented through a two-stage association method, thereby enabling stable calculation of the license plate candidate region (ROI) per frame. The ROI tracking step can be extended to a graph-based tracking method that models the relationships between objects using a Graph Neural Network (GNN), a method that extracts ReID (re-identification) embeddings into a Transformer structure, or a multimodal tracking method based on sensor fusion (radar / LiDAR), in addition to classical MOT methods such as DeepSORT or ByteTrack. In addition, when an object is missing, a missing recovery step may be added to restore the ROI by utilizing past N-frame information instead of a single frame.

[0046] In step (S730), a geometric normalization process is performed on the calculated ROI to correct tilt, distortion, and perspective deformation of the license plate. Specifically, geometric deformation of the license plate is automatically corrected by applying a learning-based spatial transformation module (e.g., STN) or a perspective transformation matrix calculation technique, and outliers that may occur during the license plate reference point detection process are removed through refinement techniques such as RANSAC. As a result, a normalized license plate image for subsequent character recognition is generated. The geometric correction step can be replaced with various geometric normalization models, such as the Thin Plate Spline (TPS) transformer, Geometry-aware Transformer, and HomographyNet, in addition to the STN-based transformation module or RANSAC-based perspective transformation. The method for detecting the four corners of the license plate can also be changed to a vertex regression-based network, an edge-line matching-based Hough Transform, or an edge heatmap-based unsupervised learning method.

[0047] For the normalized license plate image, a deep learning-based super-resolution restoration is performed in step (S740) to generate a high-resolution license plate image. At this time, the super-resolution process may be conditionally activated depending on the resolution or quality status of the input ROI, and OCR performance is improved by applying a boundary reinforcement-based learning loss to preserve character edge and stroke information. The super-resolution (SR) restoration step may be modified to include various models such as EDSR, ESRGAN, and SwinIR, and may be extended to a character structure-based loss function to prioritize the restoration of character stroke information of the license plate, or a prior-based SR method that applies license plate-specific prior learning. Additionally, if the license plate is excessively dark, an illumination compensation module may be added, or a reflection removal model to remove saturation caused by strong reflection may be placed before the SR.

[0048] In step (S750), OCR processing is performed to extract a character sequence from the restored high-resolution license plate image. The OCR process according to the present invention is configured to support various license plate formats (numbers, Korean characters, English characters, etc.) by including CTC-based or attention-based sequence models. In addition to a simple sequence recognition structure, the OCR recognition step can be extended to include a Vision Transformer-based character recognizer, an end-to-end OCR Transformer model other than a CNN-RNN-CTC combination, or a rule-based (post-processing rule set) and n-gram-based error correction model specialized for the Korean license plate format.

[0049] In step (S760), the reliability of the recognition result is determined by comparing and verifying the license plate information extracted through OCR with the previous frame or past storage history. If the similarity score is below a threshold value, it is determined to be a recognition error, and the license plate information is corrected using existing stored information or a learning-based generative model (GAN, etc.), and the corrected final license plate information is output. This correction process minimizes recognition errors caused by license plate damage, reflection, contamination, or temporary obscuration, and can enhance recognition consistency for the same vehicle through mutual correction techniques between multiple frames. The re-identification step is not limited to comparing string similarity; it can generate higher quality corrected images by using an image-based ReID that compares regional visual features of the license plate, a sentence-type license plate (e.g., special license plate) processing module, and a GAN-based character restoration, as well as a Diffusion Model-based license plate restoration model. Additionally, it is possible to automatically select the Best Reference Frame, in addition to past frames, and use it as reference data for the entire recognition process.

[0050] As illustrated in FIG. 7, the license plate recognition method according to the present invention sequentially performs a series of processing steps including multi-scale object detection, stable MOT-based ROI generation, geometric correction, super-resolution restoration, OCR processing, and re-identification correction, thereby enabling high-accuracy license plate recognition even in various image quality degradation situations that occur in real environments. Furthermore, it goes without saying that the detailed algorithms or neural network structures used in each step can be varied or replaced within the scope of the technical concept of the present invention.

[0051] The entire flow of the multi-scale-based warping correction license plate recognition method can be extended not only to an end-to-end approach but also to a distributed structure. For example, various system designs are possible, such as a cloud-edge cooperative structure where object detection and tracking are performed at edge devices and geometric correction and OCR are performed on a central server, or a method based on batch processing in a GPU / TPU environment.

[0052] As such, the license plate recognition method according to one embodiment of the present invention is not limited to the embodiments described above, and all or some steps of the image preprocessing, object detection, tracking, geometric correction, image quality restoration, OCR, re-identification, and correction processes may be freely modified or replaced according to various deep learning techniques, signal processing technologies, and hardware environments. All of these correspond to variations that do not deviate from the technical spirit and scope of the present invention.

[0053] As such, the image processing system and method according to the present invention, which improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing, integrates the entire process from object detection, tracking, geometric correction, super-resolution restoration, OCR, and re-identification into a single pipeline and enables feature-unit information exchange between each module, thereby simultaneously achieving real-time processing performance and high recognition accuracy. The above-described embodiment is merely one specific example, and it is obvious to those skilled in the art to which the present invention pertains that the internal structure of each module, network architecture, processing order, and distributed processing method (edge / server partitioning), etc., can be implemented by various modifications, substitutions, and extensions without departing from the technical spirit of the present invention. Explanation of the symbols

[0054] 100: Image processing system with improved recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing 110: Object detection unit 111 : Backbone CNN 112 : Feature Pyramid Network 113: Object Detection Head 120 : MOT-based ROI Tracking Unit 121 : CNN-based appearance feature extraction module 122: Movement Trajectory Prediction Module 123 : Association Analysis Module 130 : Geometric correction part 131 : Space conversion network module (131) 132 : Deep learning-based vertex detection network (132) 133 : RANSC-based reference point refinement module (133) 134 : Perspective Conversion Module 140 : Image quality restoration section 141 : Super-resolution network 142: Boundary Enhancement Loss Function 150 : OCR Processing Unit 151: Sequence-based character recognition module 160 : License Plate Re-identification Unit 161 : License Plate Information Database 162 : Comparison Verification Module 163 : GAN-based restoration module 164: Inter-frame mutual correction module

Claims

Claim 1 An image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing, comprising: an object detection unit that analyzes vehicle or license plate candidate objects from an input image using multi-resolution feature map decomposition to detect location information of vehicle objects including small objects; a MOT-based ROI tracking unit that uses the detection results of the object detection unit to maintain the ID of the same object even in situations of occlusion, low resolution, and motion blur through an association analysis logic that combines the object's external features or movement trajectory information, and generates a license plate candidate region as an ROI for each frame; a geometric correction unit that performs a learning-based spatial transformation network or a perspective transformation matrix calculation module to automatically correct the tilt, perspective distortion, and geometric deformation of the license plate for the ROI, and normalizes the ROI by applying an outlier removal-based model to correct the four reference point detection error of the license plate; and an image quality unit that generates a high-resolution license plate image by performing deep learning-based super-resolution restoration for the geometrically corrected ROI to restore detailed information of low-resolution or small license plate images. An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, comprising: a restoration unit; an OCR processing unit that extracts a license plate string from the restored license plate image; and a license plate re-identification unit that determines whether there is an error by comparing the license plate string of the OCR processing unit with a license plate record of the same vehicle stored prior to, and improves license plate recognition reliability by utilizing the existing record or generating correction information through a learning-based generative model in the event of recognition failure or license plate damage. Claim 2 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, further comprising: a multi-scale based object detection unit having the characteristic of decomposing an input image into a multi-size (feature pyramid) feature map and combining high-resolution features of a shallow layer with semantic features of a deep layer to improve detection sensitivity of small license plate objects in the object detection and tracking according to claim 1. Claim 3 In claim 1, the MOT-based ROI tracking unit includes a two-stage association logic that maintains the continuity of object IDs even in occlusion or motion blur situations by reconnecting low-reliability detection boxes that were not matched in the previous frame in an additional matching step, and is an image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing. Claim 4 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, wherein the MOT-based ROI tracking unit further comprises an appearance feature matching module that generates an appearance feature vector of an object and determines whether it is the same object by measuring similarity based on cosine distance. Claim 5 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, wherein the geometric correction unit includes a deep learning-based vertex detection network for detecting four corner points of a license plate and calculates an accurate perspective transformation matrix by performing RANSAC-based outlier removal on the detected points. Claim 6 In claim 1, the geometric correction unit comprises a perspective conversion module and a rotation normalization module that automatically correct the oblique shooting angle of a license plate to a form close to horizontal and front, and an image processing system that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character area warping, and iterative reprocessing. Claim 7 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, wherein the image quality restoration unit improves the OCR recognition rate of the restored image by applying an edge-enhanced loss function that prioritizes the preservation of license plate character edge and stroke information by a super-resolution network. Claim 8 In claim 1, the image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, wherein the image quality restoration unit includes a resolution-based conditional restoration (conditioned SR) method that automatically activates an SR restoration step when the geometrically corrected ROI is below a predefined resolution. Claim 9 In claim 1, the OCR processing unit comprises a sequence-based character recognition model (CTC-based or attention-based) optimized for the regional pattern (combination of dividing lines, characters, and numbers) of a license plate, and is an image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing. Claim 10 In claim 1, the license plate re-identification unit includes a matching-based correction module that calculates a string similarity score between the recognition result of the current frame and the previously stored license plate information, and prioritizes the use of existing license plate information when the similarity is less than a reference value, thereby improving recognition performance through multi-resolution feature map decomposition-based object detection and tracking and character region warping and iterative reprocessing. Claim 11 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, wherein the license plate re-identification unit restores missing characters or distorted parts using a generative adversarial network-based correction model when the license plate fails to be recognized due to contamination, reflection, and damage. Claim 12 In claim 1, the license plate re-identification unit includes a cross-frame enhancement function that utilizes high-resolution license plate information obtained from an initial frame in a situation of repeated entry of the same vehicle to improve the reliability of license plate recognition in a subsequent frame, and is an image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing. Claim 13 An image processing system that improves recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing, further comprising: an end-to-end integrated processing module that provides feature-unit linkage between each processing unit so that object detection, tracking, geometric correction, super-resolution restoration, and OCR recognition processes throughout the system can be performed in real time, in claim 1. Claim 14 An image processing method that improves recognition performance through object detection and tracking based on multi-resolution feature map decomposition, character region warping, and iterative reprocessing using an image processing system comprising an object detection unit, an MOT-based ROI tracking unit, a geometric correction unit, an image quality restoration unit, an OCR processing unit, and a license plate re-identification unit, wherein the object detection unit detects a vehicle or license plate candidate object in an input image by multi-resolution feature map decomposition; the MOT-based ROI tracking unit calculates a license plate ROI per frame while maintaining an object ID even under environmental changes such as occlusion and blurring through association analysis based on external features or movement trajectories for the detected object; the geometric correction unit corrects tilt and distortion / perspective transformation for the ROI using a learning-based spatial transformation module or perspective transformation matrix calculation, and generates a normalized license plate image by removing outlier points; the image quality restoration unit restores the corrected license plate image to high resolution using a deep learning-based super-resolution model; and the OCR processing unit extracts a string from the restored license plate image. The image processing method for improving recognition performance through multi-resolution feature map decomposition-based object detection and tracking, character region warping, and iterative reprocessing is characterized by including the step of: the license plate re-identification unit determining whether there is an error by comparing the extracted license plate information with a previous frame or past storage history, and, if necessary, outputting corrected license plate information using existing information or a learning-based generative model.

Images