Method and system for OCR object recognition based on 3D image

Abstract

A 3D image-based OCR object recognition system includes a memory for storing instructions and a processor, and the instructions, when executed by the processor, can control the system to detect an object containing text in a 3D image, perform camera distortion correction and projection normalization on the detected object to convert it into a 2D plane, analyze the placement pattern of text blocks in the converted image to automatically perform alignment and rotation, perform OCR by merging multiple lines of text, perform re-inference if the confidence score of the OCR result is below a preset threshold, and perform semantic vector-based correction if an inconsistency or error is detected in the text recognition result.

Description

Technology Field

[0001] The present invention relates to the field of computer vision and optical character recognition (OCR) technology.

[0002] More specifically, the invention relates to an intelligent OCR system and a control method thereof that detects text objects located at various angles and distances within a 3D image, such as slates, signboards, and product labels, corrects camera distortion and projection distortion to normalize them into a 2D plane, automatically analyzes the placement pattern of text blocks to align them in an optimal direction, processes multiple line text to improve OCR accuracy, and automatically corrects text recognition errors through reliability-based re-inference and semantic vector-based correction. Background Technology

[0004] With the recent advancement of computer vision technology and deep learning-based OCR engines, attempts at text recognition in various environments are becoming more active. However, existing OCR systems are primarily optimized for flat images taken from the front, so they have limitations in processing text objects located at various angles and distances in 3D space.

[0005] In particular, existing OCR systems suffer from a significant decrease in accuracy when text is tilted or distorted due to physical factors such as camera angle, lighting, and lens distortion. High false positive rates still exist regarding text tilt and angle issues in 3D space, as well as the recognition of text displayed on irregular surfaces.

[0006] Furthermore, existing technologies suffered from inefficiencies, such as the need for manual preprocessing due to a lack of capabilities to automatically determine and align text rotation, or the requirement to recognize each line individually because they could not process multi-line text integrally. The mechanisms for correcting errors in OCR results during the post-processing stage were also inadequate, leading to the limitation that the entire process had to be restarted in the event of a recognition failure. The problem to be solved

[0008] The present invention aims to provide a 3D space-2D normalization mapping algorithm that automatically detects various text objects, such as slates, signboards, and product labels, within a 3D image, and mathematically corrects lens distortion, perspective distortion, and tilt distortion using the 3D coordinates and camera parameters of the detected objects.

[0009] In addition, the present invention aims to provide an automatic correction rotation algorithm using the HEAD (Horizontal Enhancement and Alignment Detection) method, which automatically analyzes the arrangement direction and reference line of characters within a text block in a converted 2D image, calculates the rotation angle of the text, and aligns it in the optimal direction.

[0010] The present invention aims to provide a multi-line merging processing algorithm that merges two or more lines of text into a single integrated block and normalizes the spacing between lines and the character size to preprocess them into an input form optimized for an OCR engine.

[0011] The present invention aims to automatically correct misrecognized text by evaluating the reliability score of OCR results in real time and automatically performing re-inference if it falls below a threshold, and by utilizing CNN-based image feature extraction and semantic vector analysis using a CLIP (Contrastive Language-Image Pre-training) model. means of solving the problem

[0013] A 3D image-based OCR object recognition system includes a memory for storing instructions and a processor, and the instructions, when executed by the processor, can control the system to detect an object containing text in a 3D image, perform camera distortion correction and projection normalization on the detected object to convert it into a 2D plane, analyze the placement pattern of text blocks in the converted image to automatically perform alignment and rotation, perform OCR by merging multiple lines of text, perform re-inference if the confidence score of the OCR result is below a preset threshold, and perform semantic vector-based correction if an inconsistency or error is detected in the text recognition result.

[0014] The system can be replaced by electronic devices (e.g., terminals, computers). Effects of the invention

[0016] The 3D image-based OCR object recognition method and system according to the present invention can significantly improve recognition accuracy compared to existing flat image-based OCR by automatically correcting camera distortion and projection distortion for text objects at various angles and distances in 3D space and normalizing them into a 2D plane.

[0017] In addition, the present invention can recognize text in various directions in an optimal state without manual preprocessing through a HEAD-based automatic correction algorithm that automatically analyzes the placement pattern of text blocks, calculates the rotation angle, and aligns them.

[0018] The present invention has the effect of simultaneously improving processing efficiency and accuracy compared to existing methods that process multi-line text individually, through an algorithm that integrates, merges, and normalizes multi-line text. Furthermore, the present invention can minimize the OCR failure rate and increase the reliability of the final recognition result through a multi-error correction system that combines reliability-based automatic re-inference and CNN-CLIP-based semantic vector correction. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram showing the configuration of a 3D image-based OCR object recognition system according to one embodiment. Figure 2 is a flowchart illustrating a 3D image-based OCR object recognition method. Figure 3 is a flowchart illustrating a 3D image-based OCR object recognition method. Figure 4 is a flowchart illustrating a 3D image-based OCR object recognition method. Specific details for implementing the invention

[0021] The terms used in the embodiments are for illustrative purposes only and should not be interpreted restrictively. The terms "include" or "have" mean that the mentioned features or elements are present and do not exclude the possibility of other elements being present. Terms not specifically defined are to be interpreted according to their common meaning in the relevant technical field and are not to be interpreted in an overly formal sense. The embodiments may be implemented in various forms of products, such as personal computers, smartphones, and smart home appliances.

[0022] Artificial intelligence (AI) systems are computer systems that mimic human intelligence and are equipped with the ability for machines to learn and make decisions on their own. Existing rule-based systems are gradually being replaced by deep learning-based AI.

[0023] Artificial intelligence technology consists of machine learning and component technologies based on it. These component technologies include fields such as linguistic understanding, visual understanding, reasoning and prediction, knowledge representation, and motion control. The practical application of machine learning algorithms requires learning through trial and error, and deep learning, in particular, is carried out through simulation in a virtual environment on a computer.

[0024] Artificial intelligence learning models can be designed to mimic the structure of the human brain, with network nodes simulating the signal transmission of neurons. In particular, Convolutional Neural Networks (CNNs) can effectively process 2D input data with minimal preprocessing and demonstrate excellent performance in image and voice processing. Currently, development is moving in the direction of increasing test accuracy by expanding the network size through the combination of large amounts of training data and enhanced computational power.

[0026] FIG. 1 is a block diagram showing the configuration of a 3D image-based OCR object recognition system according to one embodiment.

[0027] A system (100) according to one embodiment may include a processor (120) and a memory (130), and some of the illustrated components may be omitted or substituted. A system (100) according to one embodiment may be a server or a terminal. According to one embodiment, the processor (120) is a component capable of performing operations or data processing regarding the control and / or communication of each component of the system (100), and may be composed of one or more processors. The memory (130) may store information related to the method described above or store a program in which the method described above is implemented. The memory (130) may be volatile memory or non-volatile memory. The memory (130) may store various file data, and the stored file data may be updated according to the operation of the processor (120).

[0028] According to one embodiment, the processor (120) can execute a program and control the device (100). The code of the program executed by the processor (120) can be stored in memory (130). Operations of the processor (120) can be performed by loading instructions stored in memory (130). The system (100) can be connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown in the drawing) and exchange data.

[0029] A system (100) according to one embodiment includes a processor (120) and a memory (130). A system (100) according to one embodiment may be the server or terminal described above. The memory (130) may store information related to the method described above or store a program in which the method described above is implemented. The memory (130) may be volatile memory or non-volatile memory.

[0030] A system (100) according to one embodiment includes a processor (120) and a memory (130). A system (100) according to one embodiment may be the server or terminal described above. The memory (130) may store information related to the method described above or store a program in which the method described above is implemented. The memory (130) may be volatile memory or non-volatile memory.

[0032] Figure 2 is a flowchart illustrating a 3D image-based OCR object recognition method.

[0033] In operation 210, the system (100) can detect objects containing text in a 3D image and convert them into a 2D plane. The system (100) can automatically identify text areas such as slates, signboards, and product labels through a computer vision algorithm. The system (100) can extract the 3D coordinates of the detected objects and the intrinsic and extrinsic parameters of the camera. The system (100) can mathematically inversely transform lens distortion, perspective distortion, and tilt distortion to create a state where distortion is removed. The system (100) can generate a 2D plane image of the front view through projection transformation. The system (100) can apply bilinear interpolation to minimize the loss of resolution that may occur during the conversion process.

[0034] In operation 220, the system (100) can automatically perform alignment rotation by analyzing the placement pattern of text blocks. The system (100) can detect text blocks in a converted 2D image and analyze the placement direction and baseline of the characters within each block. The system (100) can calculate the rotation angle of the text based on the relative positional relationship of the characters. The system (100) can align the text horizontally or vertically by rotating the image in the reverse direction by the calculated angle. The system (100) can automatically achieve an optimal text alignment state through an orientation check using the HEAD (Horizontal Enhancement and Alignment Detection) method.

[0035] In operation 230, the system (100) can perform OCR by merging multiple lines of text and perform confidence-based correction. When two or more lines of text are detected in an aligned image, the system (100) can merge them into a single block based on the start and end points of each line. The system (100) can normalize the spacing between lines and the character size within the merged block to preprocess them into an input form optimized for the OCR engine. The system (100) can input the preprocessed image into the OCR engine to perform text recognition and calculate a confidence score for each result. If the confidence score is below a preset threshold, the system (100) can perform re-inference by adjusting the image preprocessing parameters. If an inconsistency or error is detected in the text recognition result, the system (100) can perform automatic correction through CNN-CLIP-based semantic vector analysis.

[0036] According to one embodiment, the system (100) can detect an object containing text in a 3D image. The system (100) can convert the detected object into a 2D plane by performing camera distortion correction and projection normalization. The system (100) can automatically perform alignment rotation by analyzing the placement pattern of text blocks in the converted image. The system (100) can perform OCR by merging multiple lines of text. The system (100) can perform re-inference if the reliability score of the OCR result is below a preset threshold. The system (100) can perform semantic vector-based correction if an inconsistency or error is detected in the text recognition result.

[0037] According to one embodiment, the system (100) can automatically identify text areas within 3D space through a computer vision algorithm. The system (100) can distinguish and detect various forms of text objects, such as slates, signboards, and product labels. The system (100) can determine the spatial location and orientation of objects by utilizing depth information and surface normal vectors. Through this detection process, the system (100) can obtain three-dimensional spatial information that is impossible in planar images.

[0038] According to one embodiment, the system (100) can eliminate lens distortion and perspective distortion by mathematically inverse transforming them. The system (100) can correct the inherent radial distortion of the lens using intrinsic parameters of the camera. The system (100) can normalize perspective deformation caused by the shooting angle by utilizing extrinsic parameters. The system (100) can minimize resolution loss by applying bilinear interpolation during the transformation process of mapping 3D coordinates to a 2D plane. Through this transformation, the system (100) can make tilted text look the same as if it were photographed from the front.

[0039] According to one embodiment, the system (100) can automatically analyze the arrangement direction of characters within a text block. The system (100) can calculate the main direction of the text by connecting the center points and boundary lines of the characters. The system (100) can align the text horizontally or vertically by applying a reverse rotation equal to the calculated rotation angle. Through this automatic alignment, the system (100) can make text in various directions recognizable without manual preprocessing.

[0040] According to one embodiment, the system (100) can process text composed of multiple lines into a single integrated block. The system (100) can merge text by setting text boundaries based on the start and end points of each line. The system (100) can normalize the spacing between lines and the character size to create a consistent form. Through this merging process, the system (100) can derive an accurate OCR result that considers the overall context compared to individual recognition of each line.

[0041] According to one embodiment, the system (100) can adjust preprocessing parameters and retry for recognition results showing low reliability. The system (100) can sequentially apply various filters and correction techniques to improve image quality. The system (100) can recover from initial recognition failures through re-inference and gradually improve accuracy.

[0042] According to one embodiment, the system (100) can perform correction by combining image features and a language model when pattern mismatch or grammar error occurs. The system (100) can represent the visual form of a character as a vector through a CNN and reflect contextual information through a CLIP. The system (100) can achieve meaning-based text correction beyond simple form matching by fusing the two pieces of information. Through this multimodal approach, the system (100) can prevent misrecognition between characters with similar forms.

[0044] Figure 3 is a flowchart illustrating a 3D image-based OCR object recognition method.

[0045] In operation 310, the system (100) can extract 3D coordinates and camera parameters of a text area within a 3D image. The system (100) can automatically detect text areas including slates, signboards, and product labels by utilizing a computer vision algorithm. The system (100) can obtain 3D spatial coordinate information of the detected area. The system (100) can extract intrinsic parameters including the camera's focal length, principal point coordinates, and lens distortion coefficients. The system (100) can extract extrinsic parameters including the camera's rotation matrix and translation vectors. The system (100) can measure the 3D tilt angle of an object by calculating depth information and surface normal vectors.

[0046] In operation 320, the system (100) can inverse transform the distortion and project it onto a 2D plane of the front view. The system (100) can mathematically correct the lens distortion using the extracted 3D coordinates and camera parameters. The system (100) can inverse transform the perspective distortion and tilt distortion to make the text identical to a frontal view. The system (100) can generate a transformation matrix that maps 3D spatial coordinates to a 2D normalized plane. The system (100) can re-project the tilted text onto the front view by applying the generated transformation matrix. The system (100) can generate an input optimized for OCR processing by converting the distortion-removed object into a 2D plane image of the front view.

[0047] In operation 330, the system (100) can automatically align text by performing a HEAD-style orientation check. The system (100) can detect text blocks in a converted 2D image. The system (100) can analyze the arrangement direction and reference line of the characters within each text block. The system (100) can determine the main direction of the text by connecting the center point of the characters, the top boundary line, and the bottom boundary line. The system (100) can calculate the angle between the determined main direction and the horizontal or vertical axis. The system (100) can automatically align the text horizontally or vertically by rotating the image in reverse by the calculated angle. The system (100) can verify the alignment result to achieve an optimal alignment state.

[0048] In operation 340, the system (100) can merge multi-line text and perform OCR optimization preprocessing. The system (100) can check whether two or more lines of text are detected in an aligned image. The system (100) can merge text lines into a single block based on the start and end points of each line. The system (100) can adjust the spacing between lines within the merged block to be consistent. The system (100) can standardize the character size of each line to normalize it into a consistent form. The system (100) can analyze the contrast of the text area to detect illuminance imbalance and perform adaptive histogram equalization. Through these preprocessing steps, the system (100) can prepare the OCR engine to recognize text under optimal conditions.

[0049] According to one embodiment, the system (100) can correct distortion using 3D coordinates of a text area within a 3D image and camera parameters, project and transform the corrected object onto a 2D plane, analyze the arrangement direction of the text block to calculate a rotation angle for text alignment, and control the optimization of OCR processing by merging multiple lines of text into one block.

[0050] According to one embodiment, the system (100) can detect text areas including slates, signboards, and product labels within a 3D image using a computer vision algorithm. The system (100) can extract 3D coordinates and camera parameters of the detected area. The system (100) can mathematically inversely transform lens distortion, perspective distortion, and tilt distortion using the extracted 3D coordinates and the intrinsic and extrinsic parameters of the camera. The system (100) can project and transform the distortion-removed object into a 2D planar image of a frontal view. The system (100) can detect text blocks in the transformed 2D image and calculate the rotation angle of the text by analyzing the arrangement direction and reference line of the characters within each text block. The system (100) can perform an orientation check using the HEAD (Horizontal Enhancement and Alignment Detection) method, which automatically aligns the text horizontally or vertically by rotating the image in the reverse direction by the calculated angle. The system (100) can merge text lines into a single block based on the start and end points of each line when two or more lines of text are detected in an aligned image. The system (100) can normalize the spacing between lines and the character size within the merged block to preprocess them into an input form optimized for the OCR engine.

[0051] According to one embodiment, the system (100) can automatically identify the boundaries of text areas within a 3D image through an object detection algorithm. The system (100) can distinguish and recognize flat objects such as slates and objects installed at various angles such as signboards. The system (100) can also include text attached to curved surfaces, such as product labels, as a detection target.

[0052] According to one embodiment, the system (100) can determine the position and orientation of an object by obtaining three-dimensional spatial coordinates of a detected area. The system (100) can extract focal length, principal point position, and lens distortion coefficients from camera intrinsic parameters. The system (100) can obtain camera rotation information and movement vectors from extrinsic parameters.

[0053] According to one embodiment, the system (100) can correct radial distortion and tangential distortion caused by the physical properties of the lens. The system (100) can inversely calculate and remove deformation caused by perspective effects according to the shooting angle. The system (100) can perform a transformation that unfolds into a planar state by considering the three-dimensional tilt of the object. Through this multi-stage distortion correction, the system (100) can make the text appear in a distortion-free frontal state.

[0054] According to one embodiment, the system (100) can generate and apply a transformation matrix that maps 3D spatial coordinates to a 2D plane. The system (100) can relocate all pixels of a text area to a new position during the projection transformation process. The system (100) can adjust the image after transformation so that it takes the form of a frontal view that is optimal for processing by an OCR engine. Through this projection transformation, the system (100) can make text in a 3D environment the same as flat image OCR.

[0055] According to one embodiment, the system (100) can geometrically analyze the arrangement pattern of characters within a text block. The system (100) can calculate the slope of a line segment connecting the center points of the characters. The system (100) can determine the main direction of the text by extracting the upper and lower boundary lines of the characters. The system (100) can align the text to a reference axis by applying the calculated rotation angle in reverse. Through this automatic alignment, the system (100) can standardize text tilted in various directions.

[0056] According to one embodiment, the system (100) can process text in both horizontal and vertical directions through the HEAD method. The system (100) can automatically determine the main direction of character placement and select an appropriate alignment criterion. After alignment, the system (100) can verify whether the text matches the reading direction and perform additional adjustments if necessary.

[0057] According to one embodiment, the system (100) can automatically identify each line boundary of a multi-line text. The system (100) can determine the vertical order by analyzing the vertical arrangement relationship of the lines. The system (100) can establish an integrated area containing all identified lines to form a single processing unit.

[0058] According to one embodiment, the system (100) can evenly adjust the vertical spacing between lines within a merged block. The system (100) can measure the character size of each line and normalize it to a standard size. The system (100) can increase the recognition stability of the OCR engine by making the character thickness and contrast consistent. Through this normalization, the system (100) can make multi-line text capable of being processed with the same quality as a single line.

[0060] According to one embodiment, the system (100) calculates a confidence score for the recognition probability value of each character or word output by the OCR engine, and if the confidence score is less than a preset threshold, it performs re-recognition by adjusting preprocessing parameters, generates a semantic vector of the text using CNN-based feature extraction and a CLIP model, and can correct the misrecognized text based on the generated semantic vector.

[0061] According to one embodiment, the system (100) can perform text recognition by inputting a preprocessed image into an OCR engine. The system (100) can calculate a confidence score for each recognition result as a value between 0 and 1. If the confidence score is below a preset first threshold, the system (100) can re-perform OCR by adjusting image preprocessing parameters. If the OCR result does not match a preset pattern or if grammatical errors are detected, the system (100) can generate a semantic vector of the text by utilizing a Convolutional Neural Network (CNN)-based feature extraction and a Contrastive Language-Image Pre-training (CLIP) model. The system (100) can automatically correct the misrecognized text by searching a database for text candidates most similar to the generated semantic vector. The system (100) can classify the final recognition result as normal processing if the confidence score is above a preset second threshold. If the confidence score is below the second threshold, the system (100) can mark it as subject to manual review and suspend processing.

[0062] According to one embodiment, the system (100) can analyze text within an image character by character through an OCR engine and generate recognition results. The system (100) can calculate a quantitative reliability index by evaluating the morphological consistency of each recognized character. The system (100) can provide consistent evaluation criteria by expressing the reliability as a normalized numerical value.

[0063] According to one embodiment, the system (100) can improve recognition accuracy by repeating a preprocessing process for results with low reliability. The system (100) can automatically change parameters such as image contrast, brightness, and sharpness. The system (100) can find optimal preprocessing conditions by trying various combinations of parameters. The system (100) can compensate for initial recognition failures and improve final accuracy through re-execution.

[0064] According to one embodiment, the system (100) can perform correction by combining image and language information when detecting pattern mismatches or grammatical errors. The system (100) can convert visual shape information of characters into numeric vectors through a CNN. The system (100) can predict contextually appropriate words by utilizing a CLIP model. The system (100) can achieve meaning-based correction beyond simple character recognition by fusing visual information and language information. The system (100) can reduce the possibility of misrecognition through this multimodal approach.

[0065] According to one embodiment, the system (100) can automatically select the correct text by calculating the similarity between semantic vectors. The system (100) can find the closest item by measuring the distance to the correct answer candidates stored in the database. The system (100) can correct the final result by replacing the misrecognized text with a candidate with high similarity. The system (100) can learn recurring misrecognition patterns to strengthen its ability to correct the same error in the future.

[0066] According to one embodiment, the system (100) can verify the reliability of the result by re-evaluating the final reliability after the calibration is completed. The system (100) can determine whether automatic processing is possible based on a second threshold. The system (100) can immediately output results with high reliability and classify results with low reliability for manual review. The system (100) can achieve a balance between automation and quality assurance through this dual threshold system.

[0068] Figure 4 is a flowchart illustrating a 3D image-based OCR object recognition method.

[0069] In operation 410, the system (100) can input a preprocessed image into an OCR engine to perform text recognition and calculate a reliability score. The system (100) can recognize text within the image character by character through the OCR engine. The system (100) can calculate a reliability score based on morphological consistency for each recognition result. The system (100) can express the reliability score as a normalized value between 0 and 1. The system (100) can calculate both the average reliability of the entire text and the reliability of individual characters. The system (100) can use the reliability score as a criterion for judgment in a subsequent processing step.

[0070] In operation 420, the system (100) can re-perform OCR by adjusting image preprocessing parameters if the reliability score is below a first threshold. The system (100) can compare the calculated reliability score with a preset first threshold. The system (100) can automatically adjust the parameters of the preprocessing process if the reliability falls below the standard. The system (100) can change parameters such as image contrast, brightness, sharpness, and noise removal intensity. The system (100) can re-perform OCR after re-performing preprocessing with the adjusted parameters. The system (100) can find the optimal combination of parameters by evaluating whether the reliability of the re-performed result has improved.

[0071] In operation 430, the system (100) can generate a semantic vector of text based on CNN-CLIP when an error is detected in the OCR result. The system (100) can check whether the OCR result does not match a predefined pattern or if there are grammatical errors. The system (100) can input an image of the text area where an error was detected into a CNN-based feature extractor. The system (100) can extract a feature vector that quantifies the pixel pattern and shape information of the image. The system (100) can generate a language model vector by inputting the contextual information before and after the text into a CLIP model. The system (100) can generate an integrated semantic vector by combining the image feature vector and the language model vector.

[0072] In operation 440, the system (100) can automatically correct misidentified text by calculating the similarity between the generated semantic vector and the correct text in the database. The system (100) can compare the integrated semantic vector with the vectors of the correct texts stored in the database. The system (100) can quantify the similarity using cosine similarity or Euclidean distance. The system (100) can select the text with the highest similarity as a candidate. The system (100) can automatically replace the misidentified text with the selected candidate. The system (100) can improve future automatic correction accuracy by learning the pattern when the same type of misidentification is repeated.

[0073] In operation 450, the system (100) can classify the reliability score of the final recognition result as normal processing or pending by comparing it with a second threshold. The system (100) can recalculate the reliability score of the final text after correction is complete. The system (100) can check whether the recalculated reliability is greater than or equal to a preset second threshold. If the reliability is greater than or equal to the second threshold, the system (100) can classify it as normal processing and output the result. If the reliability is less than the second threshold, the system (100) can mark it as subject to manual review. The system (100) can suspend processing for items classified as subject to manual review and send a notification to the administrator.

[0074] According to one embodiment, the system (100) can extract tilt information of an object from a 3D image, generate a 3D-2D conversion matrix based on the extracted tilt information, correct the illuminance imbalance of the converted image, and estimate camera distortion parameters to perform real-time calibration.

[0075] According to one embodiment, the system (100) can measure the three-dimensional tilt angle of an object by calculating depth information and surface normal vectors of text areas detected in a 3D image. The system (100) can generate a transformation matrix that maps 3D spatial coordinates to a 2D normalized plane for objects with irregular angles where the measured tilt angle exceeds a specified threshold. The system (100) can reproject the tilted text into a frontal view by applying the generated transformation matrix. The system (100) can apply bilinear interpolation to minimize the loss of resolution of the original 3D image. The system (100) can detect illumination imbalance by analyzing the contrast of the text area in the reprojected 2D image. The system (100) can improve text readability by performing adaptive histogram equalization on the detected illumination imbalance area. The system (100) can automatically perform calibration to correct distortion by estimating the radial distortion coefficient and tangential distortion coefficient of the camera lens in real time.

[0076] According to one embodiment, the system (100) can obtain distance information to a text area through a depth sensor or stereo vision. The system (100) can determine the direction in which the object surface faces the camera by calculating a surface normal vector. The system (100) can quantify the three-dimensional tilt of the object by calculating the angle between the normal vector and the camera view vector. Through this tilt measurement, the system (100) can identify text with severe angles that are difficult to process with only planar transformation.

[0077] According to one embodiment, the system (100) can classify irregular angle objects whose measured slope exceeds a threshold as special processing targets. The system (100) can represent transformation relationships connecting each point in 3D space to a corresponding position in a 2D plane in the form of a matrix. The system (100) can integrate complex transformations including rotation, translation, and scaling into a single matrix.

[0078] According to one embodiment, the system (100) can reposition tilted text to a frontal state by applying a transformation matrix to each pixel. The system (100) can handle cases where the original pixel location is mapped to real coordinates rather than integers during the reprojection process. The system (100) can perform bilinear interpolation to generate a new pixel value by calculating a weighted average of surrounding pixels. Through this interpolation technique, the system (100) can maintain the sharpness and quality of the image after transformation.

[0079] According to one embodiment, the system (100) can quantitatively measure the difference in brightness between text and background in a reprojected image. The system (100) can automatically detect areas within the text area where lighting is unevenly distributed. The system (100) can detect local changes in brightness caused by shadows or reflected light.

[0080] According to one embodiment, the system (100) can perform independent brightness adjustment for each region of the image. The system (100) can achieve overall balance by making dark areas brighter and bright areas relatively darker. The system (100) can maximize brightness contrast and sharpen text boundaries through histogram equalization. Through this adaptive processing, the system (100) can secure stable OCR performance even in environments with poor lighting conditions.

[0081] According to one embodiment, the system (100) can analyze lens distortion characteristics by capturing a checkerboard pattern or a known shape. The system (100) can estimate a radial distortion coefficient by measuring the degree to which a straight line is distorted into a curve. The system (100) can calculate tangential distortion at a position off-center from the lens. The system (100) can immediately correct the input image by applying the estimated distortion coefficient in real time.

[0083] According to one embodiment, the system (100) generates a recognition probability value output by an OCR engine for each character when performing OCR as a confidence score, generates an integrated semantic vector combining image features and context information for words containing characters whose confidence score is less than a preset threshold, corrects text by comparing the generated vector with reference data in a database, and improves the automatic correction function by learning misrecognition patterns.

[0084] According to one embodiment, the system (100) can generate individual confidence scores calculated based on the morphological match of each character along with the OCR output when the OCR engine performs text recognition. The system (100) can identify a character as a potentially misidentified character if the confidence score of a specific character among the entire recognized text is measured to be lower than a preset threshold. The system (100) can input an image region of a word containing a potentially misidentified character into a CNN-based feature extractor to extract a feature vector that quantifies the pixel pattern and morphological information of the image. The system (100) can input contextual information before and after the sentence to which the word belongs into a CLIP (Contrastive Language-Image Pre-training) model to generate a language model vector that predicts a contextually appropriate word. The system (100) can combine the extracted image feature vector and the language model vector to generate an integrated semantic vector indicating what the word is likely to be. The system (100) can correct misrecognized characters by calculating the similarity between the generated integrated semantic vector and the vectors of correct texts stored in the database to find the most similar text. The system (100) can automatically correct the misrecognized characters by learning the mapping relationship between the misrecognition pattern and the correct pattern when the misrecognition of the same type of character is repeated more than a specified number of times.

[0085] According to one embodiment, the system (100) can quantitatively evaluate the shape matching accuracy when the OCR engine recognizes each character. The system (100) can quantify the degree of agreement between the recognized character and the learned character template and express it as a reliability. The system (100) can calculate the reliability for each character individually to accurately identify uncertain parts within the entire text.

[0086] According to one embodiment, the system (100) can select characters with a high risk of misrecognition by comparing a reliability score with a threshold. The system (100) can identify confusion between characters with similar shapes or ambiguity caused by damaged characters. By identifying characters that may be misrecognized, the system (100) can focus on areas requiring additional verification. Through this selective processing, the system (100) can prevent the inefficiency of reprocessing the entire text.

[0087] According to one embodiment, the system (100) can extract morphological features such as edges, curves, and intersections of a character image through a CNN. The system (100) can convert pixel-level image information into high-dimensional feature vectors. The system (100) can make the extracted feature vectors numerically represent the visual essence of the character.

[0088] According to one embodiment, the system (100) can evaluate the contextual appropriateness of a word within a sentence by utilizing a CLIP model. The system (100) can predict a word likely to appear in a given position by analyzing semantic associations with preceding and succeeding words. The system (100) can utilize grammar rules and word co-occurrence relationships learned by a language model.

[0089] According to one embodiment, the system (100) can generate an integrated vector by weightedly combining image features and linguistic context information. The system (100) can make a more accurate judgment by considering visual evidence and linguistic evidence simultaneously. When the two pieces of information conflict, the system (100) can give greater weight to the side with higher reliability. Through this multimodal fusion, the system (100) can reduce the possibility of error compared to a single source of information.

[0090] According to one embodiment, the system (100) can calculate the distance between an integrated vector and correct answer candidates in a database. The system (100) can find the closest item using cosine similarity or Euclidean distance. The system (100) can correct misrecognition by selecting the text with the highest similarity as the final answer.

[0091] According to one embodiment, the system (100) can analyze patterns by accumulating repeated misrecognition cases. The system (100) can learn the tendency for a specific character to be frequently misrecognized as another specific character. The system (100) can store the learned misrecognition patterns in a lookup table for immediate reference. When the same error recurs, the system (100) can quickly correct it by automatically applying the learned mapping. Through this experience-based learning, the system (100) can continuously improve the accuracy of misrecognition correction over time.

[0093] According to one embodiment, the system (100) can detect cases where the same text object appears in multiple frames at different angles or distances in a 3D image. The system (100) can generate multiple recognition results by performing OCR independently on each frame. The system (100) can compare the generated multiple recognition results character by character and select the character with the highest confidence score at each location. The system (100) can determine the final text by increasing the confidence of matching characters and decreasing the confidence of mismatching characters in the multiple results. The system (100) can automatically infer the domain to which the text belongs by analyzing the 3D position, size, and relationship with surrounding objects of the detected text object. The system (100) can perform OCR post-processing by applying a language model and text pattern rules specialized for the inferred domain. The system (100) can track the movement of text objects between consecutive frames in a video. The system (100) can narrow the OCR processing range and improve processing speed by using the text result recognized with high confidence in the previous frame as the initial value of the current frame. The system (100) can calculate a frame quality score by quantifying the clarity, lighting uniformity, and degree of motion blur of each frame. The system (100) can prioritize selecting the OCR result of the frame with the highest quality score.

[0094] According to one embodiment, the system (100) can identify the same text object across multiple frames through an object tracking algorithm. The system (100) can utilize differences in shooting conditions in frames with different angles and distances. The system (100) can utilize complementary characteristics in which parts that are blurry or obscured in some frames appear clear in others.

[0095] According to one embodiment, the system (100) can obtain independent recognition results by individually executing OCR on each frame. The system (100) can obtain various results reflecting different preprocessing results and recognition conditions for each frame. The system (100) can reduce accidental errors that may occur in a single attempt through multiple independent results. The system (100) can improve recognition reliability through such parallel processing.

[0096] The system (100) can compare the recognition results and reliability for characters at the same location per frame. The system (100) can select the result showing the highest reliability as the final character at that location. The system (100) can further increase the reliability by determining that characters matching across multiple frames are likely to be correct recognitions. The system (100) can decrease the reliability by considering that characters that do not match have high uncertainty.

[0097] According to one embodiment, the system (100) can estimate the use of an object through 3D spatial information where the text is located. The system (100) can distinguish between a movie slate, a road sign, a product label, etc. by analyzing the object size and surrounding environment. The system (100) can identify expected text formats and vocabulary based on the inferred domain. The system (100) can provide context-appropriate interpretation through such domain inference.

[0098] According to one embodiment, the system (100) can apply a domain-specific language model and pattern rules. The system (100) can verify the scene number and take number formats in the case of a slate. The system (100) can verify the barcode or manufacturing date formats in the case of a product label. Through this domain-specific processing, the system (100) can increase accuracy compared to general-purpose OCR.

[0099] According to one embodiment, the system (100) can determine the movement trajectory by tracking changes in the position of a text object in consecutive frames. The system (100) can utilize the recognition results of the previous frame as prior information for processing the current frame. The system (100) can reduce the amount of computation by reconfirming only the changed parts instead of recognizing the entire text from the beginning.

[0100] According to one embodiment, the system (100) can evaluate the sharpness of each frame by edge intensity and high-frequency components. The system (100) can measure lighting uniformity by brightness dispersion and contrast. The system (100) can detect motion blur by pixel changes between consecutive frames. The system (100) can determine the priority per frame by combining these quality indicators.

[0102] According to one embodiment, the system (100) can measure and quantify the rate of change in character stroke thickness, the uniformity of character spacing, and the contrast value between the background and the text in a text area detected in a 3D image. The system (100) can calculate a text quality index as a weighted sum of these values. If the calculated text quality index falls within the upper range, the system (100) can perform a single OCR processing. If the text quality index falls within the lower range, the system (100) can apply an adaptive processing depth that selects the result with the highest reliability after performing multiple OCR processing with different preprocessing parameters in parallel. The system (100) can analyze the character shape, size, and spacing patterns of the visible text portion when part of the text is obscured by another object. Based on the analyzed patterns, the system (100) can generate a group of character candidates that can fit into the obscured area and perform partial text restoration to select the character most appropriate in context. The system (100) can calculate the radius of curvature of the surface when detecting text located on a cylindrical or spherical surface in 3D space. The system (100) can perform OCR by dividing the text into units of a certain angle, individually projecting each divided area onto a plane, and then performing OCR. The system (100) can connect each recognized divided result in order according to the original curved surface position. The system (100) can classify and store cases that were reprocessed due to low reliability during the OCR processing, cases where semantic vector correction was applied, and cases where multiple frame integration was performed. The system (100) can track the final recognition success rate for each case to automatically learn and prioritize the most effective processing path.

[0103] According to one embodiment, the system (100) can evaluate text quality by measuring whether the thickness of the character strokes is uniform or highly variable. The system (100) can determine the alignment state by analyzing whether the character spacing is maintained at a constant level. The system (100) can quantify readability by quantifying the difference in brightness between the background and the text. The system (100) can derive an overall quality index by weighting and averaging these individual indicators.

[0104] According to one embodiment, the system (100) can quickly complete recognition with minimal processing for high-quality text. For low-quality text, the system (100) can perform intensive processing that attempts various preprocessing options. The system (100) can apply various parameter combinations, such as brightness adjustment, contrast enhancement, and noise removal, in parallel. The system (100) can compare the results of each combination and select the one showing the highest reliability as the final result.

[0105] According to one embodiment, the system (100) can calculate the average width and height of characters in a visible text portion. The system (100) can estimate the expected position of the next character by analyzing the regularity of the spacing between characters. The system (100) can predict the number of characters that can be included from the size of the hidden area. The system (100) can generate semantically appropriate character candidates by considering the context of the visible characters. Through such partial restoration, the system (100) can infer and complete even completely invisible text.

[0106] According to one embodiment, the system (100) can geometrically calculate the curvature of a cylindrical or spherical surface. Based on the curvature information, the system (100) can approximate the surface into several small planar pieces. The system (100) can minimize distortion by independently unfolding each piece into a 2D plane. The system (100) can obtain partial results by performing individual OCR on each divided area. The system (100) can reconstruct the entire text by connecting the partial results in the order of their original positions on the curved surface. Through this division process, the system (100) can recognize curved text that is difficult to process using only planar projection.

[0107] According to one embodiment, the system (100) can collect statistics by separately recording cases to which each processing method is applied. The system (100) can calculate the success rate of each method, such as reprocessing, semantic vector correction, and multi-frame integration. The system (100) can analyze patterns to determine which method is most effective under specific conditions. Based on the accumulated statistics, the system (100) can automatically select the optimal processing path for a new input.

[0108] According to one embodiment, the system (100) can analyze environmental lighting conditions in real time by utilizing an illuminance sensor and a color temperature sensor together when detecting text objects in a 3D image. The system (100) can quantify the illuminance level affecting text readability by measuring the brightness and color temperature of the current lighting. The system (100) can perform lighting-adaptive preprocessing that strengthens noise removal in low-light environments and prioritizes the restoration of overexposed areas in high-light environments. The system (100) can maintain consistent text recognition performance in various lighting conditions, such as fluorescent lights, incandescent lights, and natural light, by automatically adjusting the white balance according to the color temperature.

[0109] According to one embodiment, the system (100) can restore low-resolution or damaged text by applying a Generative Adversarial Network (GAN)-based image quality enhancement model before OCR processing. The system (100) can upscale text images that are blurry or have compression loss to high resolution. The system (100) can improve OCR accuracy by predicting and restoring lost details through the generator network of the GAN. The system (100) can verify whether the restored image is similar to actual high-quality text by utilizing a discriminator network. Through this prior quality enhancement, the system (100) can overcome the limitations of the original image and improve it to a recognizable state.

[0110] According to one embodiment, the system (100) can rapidly identify text regions in complex backgrounds by utilizing real-time object detection algorithms such as YOLO or SSD when detecting text. The system (100) can simultaneously detect text of various sizes, orientations, and styles through a deep learning-based detector. The system (100) can determine priority by generating a confidence score for the detected text regions. The system (100) can prevent false detections by performing additional verification on regions with low confidence. The system (100) can recognize text in real-time even in video streams through these high-speed detection algorithms.

[0111] According to one embodiment, the system (100) can provide real-time text translation and overlay functions in an augmented reality (AR) environment. The system (100) can accurately determine the position and orientation of text recognized in 3D space and display the translated text in the same location as the original. The system (100) can analyze the font size, color, and style of the text to render the translated text with similar visual attributes. The system (100) can track changes in the user's viewpoint and adjust so that the overlaid translated text is always displayed in the correct 3D location.

[0112] According to one embodiment, the system (100) can perform multimodal verification in conjunction with voice recognition technology. The system (100) can analyze whether text content is mentioned in voice recorded together with video recording. The system (100) can check for a match by comparing a word spoken aloud with visually recognized text. If the two pieces of information match, the system (100) can significantly increase the reliability of the recognition result. If a discrepancy occurs, the system (100) can perform text correction by utilizing voice information as an additional clue.

[0113] The embodiments described above may be implemented as hardware components, software components, or a combination thereof. For example, the described device, method, and components may be implemented using one or more computers such as a processor, controller, arithmetic logic unit (ALU), digital signal processor (DSP), microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, etc., and the processing unit may perform an operating system (OS) and software applications running on said operating system.

[0114] Software may include computer programs, code, instructions, or combinations thereof, and may be stored on one or more computer-readable recording media or distributed and stored and executed on computer systems connected to a network. Although embodiments have been described through specific drawings, a person skilled in the art may apply various technical modifications and variations based thereon.

Claims

Claim 1 A system for 3D image-based OCR object recognition, comprising: a memory for storing instructions; and a processor, wherein, when the instructions are executed by the processor, the system detects an object containing text in a 3D image, performs camera distortion correction and projection normalization on the detected object to convert it into a 2D plane, detects a text block in the converted image, determines a rotation angle of the text based on the center point, top boundary line, and bottom boundary line of the characters within the text block, rotates the converted image based on the rotation angle to generate a rotated alignment image, and if multiple lines of text are identified in the alignment image, merges the multiple lines of text into a single block based on the start and end points of each identified multiple line, normalizes the spacing between lines and the character size within the merged block to perform OCR, performs re-inference if the reliability score of the OCR result is below a preset threshold, and controls the system to perform semantic vector-based correction if an inconsistency or error is detected in the text recognition result. Claim 2 In claim 1, the above instructions are a system that, when executed by the processor, corrects lens distortion, perspective distortion, and tilt distortion using 3D coordinates of a text area within a 3D image and camera intrinsic and extrinsic parameters, measures the 3D tilt angle of an object based on depth information and surface normal vectors, and controls the system to project and transform the corrected object onto a 2D plane of a frontal view. Claim 3 In claim 1, the above instructions, when executed by the processor, the system calculates a confidence score for the recognition probability value of each character or word output by the OCR engine, and if the confidence score is less than a preset first threshold, performs re-recognition by adjusting preprocessing parameters including image contrast, brightness, sharpness, and noise removal intensity, and if the OCR result does not match a preset pattern or a grammatical error is confirmed, generates a semantic vector of the text using CNN-based feature extraction and a CLIP model, and controls the system to correct the misrecognized text based on the similarity between the generated semantic vector and the correct text vector stored in the database. Claim 4 In claim 1, the above instructions control the system to extract tilt information of an object from a 3D image, generate a 3D-2D conversion matrix based on the extracted tilt information, measure the contrast of the converted image to detect illuminance imbalance, perform adaptive histogram equalization on the detected illuminance imbalance area, and estimate camera distortion parameters to perform real-time calibration. Claim 5 A system that, when executed by the processor, when the same text object is detected in multiple frames at different angles or distances, performs OCR independently for each frame to generate multiple recognition results, compares the generated multiple recognition results character by character to select the character having the highest confidence score at each location, determines the final text by increasing the confidence for matching characters and decreasing the confidence for non-matching characters in the multiple recognition results, reduces the OCR processing range by using the text result recognized with high confidence in the previous frame as the initial value of the current frame, determines the domain to which the same text object belongs based on the 3D position, size, and relationship with surrounding objects of the same text object, and controls the system to perform OCR post-processing by applying a language model and text pattern rules corresponding to the determined domain.

Images