A Deep Learning-Based Character Recognition Method for Highly Reflective Metal Surfaces

Abstract

This invention discloses a deep learning-based method for recognizing characters on highly reflective metal surfaces. The method involves filtering an image containing text on a metal surface, then performing Retinex image enhancement on the filtered image, determining the region of interest (ROI) using template matching, applying an affine transformation to fix the character positions within the ROI, detecting the text using an improved DB algorithm, and finally recognizing the text using a CRNN. ​​This invention enables effective recognition of characters on highly reflective metal surfaces.

Description

Technical Field

[0001] This invention belongs to the field of industrial robot technology, and relates to character recognition technology, particularly to a method for recognizing characters on highly reflective metal surfaces based on deep learning. Background Technology

[0002] With the advent of the digital information age, character recognition in images has become particularly important. Manual recognition suffers from inefficiency, high time costs, and a high rate of repetitive errors, easily leading to work mistakes. Optical Character Recognition (OCR), an indispensable branch of computer vision, solves the problem of automatically, quickly, and accurately inputting character information into electronic devices. The concept of OCR technology was first proposed in 1929 by the German scientist Tausheck. Research on OCR technology began worldwide in the 1960s and 70s. OCR is an information processing method that simulates human vision—a process of extracting and digitizing text from target images containing text content. As an important human-computer interaction window, it can replace manual processing of a large number of simple and repetitive tasks.

[0003] OCR character recognition technology has undergone years of development, resulting in numerous OCR algorithms such as LeNet, RRPN, DMPNet, and CTPN. However, OCR technology is sensitive to changes in lighting conditions; recognition performance is difficult to guarantee under varying or low-light environments. Furthermore, for OCR recognition on metal surfaces, characters on different metal materials will appear differently under the same lighting, affecting recognition accuracy. Additionally, the large number of neurons in deep learning leads to computational complexity and time consumption. Moreover, recognition performance is unsatisfactory when recognizing characters with styles significantly different from those in the training set, resulting in unstable recognition in complex and variable environments. Summary of the Invention

[0004] The purpose of this invention is to address the problem of different reflective properties caused by different metal materials in the prior art, and to provide a deep learning-based method for recognizing characters on highly reflective metal surfaces. This method can recognize characters on metal surfaces, simplifies the network structure, and has a short processing time and high accuracy.

[0005] For characters on the surface of parts in complex industrial settings, the small size of the parts and the high resistance of the recognition environment, such as unstable lighting conditions and varying surface defects, lead to low image quality during character recognition and image acquisition. The images themselves are distorted, occluded, blurry, and cluttered. To overcome environmental interference and improve the accuracy of character recognition, this invention provides a deep learning-based method for recognizing characters on highly reflective metal surfaces, comprising the following steps:

[0006] S1 acquires images containing characters on a metal surface;

[0007] S2 performs filtering on images containing characters on metallic surfaces;

[0008] S3 performs Retinex image enhancement processing on the filtered image;

[0009] S4 determines the region of interest using a template matching method;

[0010] S5 performs an affine transformation on the region of interest;

[0011] S6 performs character detection, including the following steps:

[0012] S61 performs feature extraction on the image obtained by affine transformation;

[0013] S62 obtains a probability map and a dynamic threshold map of the predicted text based on extracted features;

[0014] S63 obtains an approximate binarized image based on a dynamic threshold image and a text probability image. The outline of the approximate binarized image is the text box.

[0015] S7 will take the text box input into the CRNN network and recognize the characters.

[0016] In step S2 above, to address the issue of noise affecting image recognition caused by external interference and other factors during image acquisition, median filtering is performed on the image. Median filtering is a nonlinear signal processing technique based on ordinal statistics theory that effectively suppresses noise. It replaces the value of a point in a digital image or sequence with the median value of all points in its neighborhood, making the surrounding pixel values ​​closer to the true value, thereby eliminating isolated noise points. It does not rely on values ​​in the neighborhood that differ significantly from typical values ​​and has advantages such as simple computation, high speed, and effective suppression of salt-and-pepper noise. The median filtering calculation formula is as follows:

[0017] F(x,y)=Med{f(x-θ,y-θ),…,f(x,y),…,f(x+θ,y+θ)} (1)

[0018] Where Med represents the calculated median, and θ represents the size of the filter window.

[0019] In step S3 above, insufficient lighting may lead to unclear details, affecting subjective visual effects and making regional features indistinct, thus posing difficulties for target detection and recognition. To address the issue of the reflective properties of different materials and the impact of lighting conditions on image quality, this invention employs the Retinex image enhancement method to enhance the image and reduce the influence of lighting. The Retinex image enhancement method is an image enhancement algorithm based on the constancy of human color perception, simultaneously achieving dynamic range compression, detail enhancement, and color constancy. Based on color consistency, Retinex theory effectively improves local image contrast and better maintains image naturalness by eliminating or suppressing ambient light. Multi-scale Retinex image enhancement is implemented according to the following formula:

[0020]

[0021] Among them, R i This represents the reflection component of the i-th channel, where i = 1, 2, 3 represent the R, G, and B channels respectively; I i This represents the component of the i-th channel of the image obtained after filtering in step S2; N is the number of scale parameters and the weighting coefficient ω. k satisfy Represents the Gaussian blur filter at the k-th scale:

[0022]

[0023] Among them, c k This represents the standard deviation of the k-th Gaussian filter.

[0024] In step S4 above, characters on the metal surface may appear at any position on the image due to the workflow. Although character detection algorithms already exist, the tilt angle of the characters affects character detection and recognition. This invention improves the character recognition rate by finding the region of interest through template matching. Based on the highly reflective metal part to be identified (i.e., the metal workpiece to be tested), a template containing the framework of the region of interest is constructed. Then, the region of interest is matched from the image after image enhancement using the template matching (shape matching) method.

[0025] The shape matching process first counts the total number M edge points of the template. Then, within the Region of Interest (ROI) to be matched, the cosine value (0-1) of the angle between each point on the template boundary and the direction vector of the corresponding point in the image to be matched is calculated. Assume the matching cosine values ​​for the M points are a1, a2, a3, ..., a... M The matching score within this ROI region can then be expressed as:

[0026]

[0027] The P-value ranges from 0 to 1, with a higher match rate indicating a closer match to 1. If P exceeds the set threshold (set to 0.85), the ROI region is recorded, and the center point of this ROI region is the location of the target object in the image to be matched.

[0028] In step S5 above, based on the template used in step S4 and the matched region of interest, an affine transformation matrix can be constructed to obtain image-related information, such as the scaling factor relative to the template, position changes, and rotation angles. By using the affine transformation matrix to perform an affine transformation on the region of interest, the position and orientation of the characters are fixed, thereby improving recognition accuracy.

[0029] In step S6 above, many segmentation-based methods require complex post-segmentation processing, such as grouping pixel-level prediction results into the detected text, like pixel clustering. This process is time-consuming. To address this time-consuming drawback, the DB algorithm incorporates a differentiable binarization operation into the segmentation network for joint optimization. This allows for adaptive setting of the binarization threshold, effectively distinguishing pixels from the foreground and background, simplifying post-segmentation processing, and significantly improving the speed and accuracy of text detection. It can also handle irregularly shaped text, such as curved or multi-directional text. Therefore, this invention chooses the DB algorithm for text detection. Since the standard binarization function is non-differentiable, this invention proposes an approximate function for binarization.

[0030] In step 61, the image obtained after affine transformation is fed into the Feature Pyramid Network (FPN) structure. The output of the feature pyramid is transformed to the same size by upsampling and then cascaded to generate feature F.

[0031] In step 62, the predicted text probability map and dynamic threshold map are obtained using feature F.

[0032] In step 63, an approximate binarized image is calculated from the dynamic threshold image and the predicted text probability image. The outline of the approximate binarized image is the text box.

[0033]

[0034] In the formula, (u,v) represents the image coordinates. P represents the approximate binarization result. u,v T represents the probability of the text. u,v This represents the dynamic threshold, and K represents the amplification factor.

[0035] For detailed instructions on steps S61-S63 above, please refer to Minghui Liao et al., Real-time Scene Text Detection with Differentiable Binarization, AAAI 2020.

[0036] DB provides highly robust binarized graphs, greatly simplifying post-processing and performing well in lightweight backbones, balancing the improvement of text detection efficiency and accuracy.

[0037] In step S7 above, traditional character recognition algorithms segment the entire line of strings into individual characters for recognition, training, and adjustment. These methods require training a powerful character detector, and even then, they cannot be applied to image-based sequence recognition. To address this issue, the novel CRNN neural network model is specifically designed to recognize variable-length sequential objects in images. It treats an entire line of characters as a unit, recognizing characters within an image region and returning the specific content of the character line in sequence form. This non-segmented, end-to-end recognition of character sequences within units allows for direct line-level annotation learning, without being limited by sequence length. Through recurrent convolutional recognition of the entire line, not only is the recognition accuracy significantly improved, but the annotation of the training set is also much simpler.

[0038] The CRNN network used in this invention consists of three parts, from input to output: a convolutional layer, a recurrent layer, and a transcription layer. The convolutional layer automatically extracts feature sequences from each input image; the recurrent layer predicts each frame of the feature sequence and outputs the predictions; subsequently, the transcription layer converts the predictions from each frame of the recurrent layer into label sequences, essentially finding the label sequence with the highest probability combination for each frame predicted by the RNN. For specific results of the CRNN network, please refer to Baoguang Shi et al., "An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume: 39, Issue: 11, November 2017).

[0039] This invention reduces interference from lighting, metal materials, and other external noise through image preprocessing, weakening the influence of lighting and different metal materials; it corrects characters through template matching and affine transformation, facilitating subsequent character detection and recognition, and improving character recognition accuracy; finally, it locates the character region, segments, and recognizes the character. Compared with existing OCR algorithms, the deep learning-based character recognition method for highly reflective metal surfaces provided by this invention has the following advantages:

[0040] 1) This invention first filters and enhances the image, then matches the region of interest using a template, then performs an affine transformation on the region of interest to fix the character positions, then uses an improved DB algorithm to detect the text, and finally uses CRNN to recognize the text, which can effectively recognize characters on highly reflective metal surfaces.

[0041] 2) This invention reduces the influence of lighting and different metal materials by using Retinex image enhancement, which enhances the robustness of the network to lighting changes and reduces the influence of the external environment on metal workpieces. The whole method has strong transferability and can be used for OCR character recognition on metal surfaces with different lighting changes and different materials, which improves recognition efficiency and has high robustness.

[0042] 3) This invention combines DB text detection and CRNN text recognition, which greatly reduces the parameter scale of the neural network and shortens the recognition calculation time, making it a lightweight OCR recognition algorithm.

[0043] 4) This invention introduces template matching and affine transformation to correct character regions, which greatly improves the accuracy of recognition and thus helps to improve the production efficiency of batch metal workpieces. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the process for a deep learning-based character recognition method for highly reflective metal surfaces, provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the image acquisition principle involved in an embodiment of the present invention; wherein, 1-CCD camera, 2-ring light source, 3-metal workpiece to be measured.

[0047] Figure 3This is a schematic diagram of the character detection steps provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0050] Example 1

[0051] The deep learning-based character recognition method for highly reflective metal surfaces provided in this embodiment is as follows: Figure 1 As shown, it includes the following steps:

[0052] S1 captures images containing characters on a metal surface.

[0053] This embodiment is based on, as follows Figure 2 The image acquisition principle shown is used to acquire images of characters on a highly reflective metal surface. A CCD camera 1 is positioned directly above the metal workpiece 3 to be tested, with its imaging window covering the workpiece. Simultaneously, a ring light source 2 is placed below the CCD camera lens to provide large-area, uniform illumination for the workpiece, reducing shadows and improving contrast. Furthermore, a robotic arm can be used to hold the workpiece, positioning it directly below the CCD camera lens.

[0054] S2 performs filtering on images containing characters on metallic surfaces.

[0055] In this embodiment, median filtering is used to process the image acquired in step S1, thereby eliminating isolated noise points. The median filtering calculation formula used is as follows:

[0056] F(x,y)=Med{f(x-θ,y-θ),…,f(x,y),…,f(x+θ,y+θ)} (1)

[0057] Where Med represents the calculated median, θ represents the size of the filter window, and (x,y) represents the image pixel coordinates.

[0058] S3 performs Retinex image enhancement processing on the filtered image;

[0059] The multi-scale Retinex image enhancement used in this embodiment is implemented according to the following formula:

[0060]

[0061] Among them, R i This represents the reflection component of the i-th channel, where i = 1, 2, 3 represent the R, G, and B channels respectively; I i This represents the component of the i-th channel of the image obtained after filtering in step S2; N is the number of scale parameters and the weighting coefficient ω. k satisfy Represents the Gaussian blur filter at the k-th scale:

[0062]

[0063] Among them, c k This represents the standard deviation of the k-th Gaussian filter.

[0064] The enhanced image is obtained by merging the three-channel reflection components.

[0065] S4 determines the region of interest using a template matching method.

[0066] In this embodiment, a template containing a region of interest frame is constructed based on the metal workpiece to be tested, and then the region of interest is matched from the image after image enhancement processing by the template matching (shape matching) method.

[0067] The shape matching process first counts the total number M edge points of the template. Then, within the Region of Interest (ROI) to be matched, the cosine value (0-1) of the angle between each point on the template boundary and the direction vector of the corresponding point in the image to be matched is calculated. Assume the matching cosine values ​​for the M points are a1, a2, a3, ..., a... M The matching score within this ROI region can then be expressed as:

[0068]

[0069] The P-value ranges from 0 to 1, with a higher match rate indicating a closer match to 1. If P exceeds the set threshold (set to 0.85), the ROI region is recorded, and the center point of this ROI region is the location of the target object in the image to be matched.

[0070] S5 performs an affine transformation on the region of interest.

[0071] Based on the template used in step S4 and the matched region of interest, an affine transformation matrix can be constructed to obtain image-related information, such as the scaling factor relative to the template, positional changes, and rotation angles. By performing an affine transformation on the region of interest using the affine transformation matrix, the position and orientation of the characters can be fixed, thereby improving recognition accuracy.

[0072] S6 performs character detection.

[0073] In this embodiment, an improved DB module is used to detect characters. The improved DB module's functions include extracting features F through a Feature Pyramid Network (FPN) structure, using features F to obtain a predicted text probability map and a dynamic threshold map, and calculating an approximate binarized map from features F and the text probability map. Based on this, this step includes the following sub-steps:

[0074] S61 performs feature extraction on the image obtained by affine transformation.

[0075] In this embodiment, the image obtained after affine transformation is fed into a Feature Pyramid Network (FPN) structure. The output of the feature pyramid is transformed to the same size through upsampling, and then cascaded to generate features F. This step uses a Feature Pyramid Network (FPN) structure already disclosed in the art.

[0076] S62 obtains the predicted text probability map and dynamic threshold map based on extracted features.

[0077] In this embodiment, feature F is used to obtain the predicted text probability map and dynamic threshold map.

[0078] S63 obtains an approximate binarized image based on a dynamic threshold image and a text probability image. The outline of the approximate binarized image is the text box.

[0079] In this embodiment, an approximate binarized image is calculated from the dynamic threshold image and the predicted text probability image. The outline of the approximate binarized image is the text box.

[0080]

[0081] In the formula, (u,v) represents the image coordinates. P represents the approximate binarization result. u,v T represents the probability of the text. u,v This represents the dynamic threshold, and K represents the amplification factor, which is set to 50 here.

[0082] For detailed instructions on steps S61-S63 above, please refer to Minghui Liao et al., Real-time Scene Text Detection with Differentiable Binarization, AAAI 2020.

[0083] S7 will take the text box input into the CRNN network and recognize the characters.

[0084] The CRNN used in this embodiment consists of three parts, from input to output: a convolutional layer, a recurrent layer, and a transcription layer. The convolutional layer automatically extracts feature sequences from each input image; the recurrent layer predicts each frame of the feature sequence and outputs the predictions; subsequently, the transcription layer converts the predictions from each frame of the recurrent layer into label sequences, essentially finding the label sequence with the highest probability combination for each frame predicted by the RNN. For specific results of the CRNN network, please refer to Baoguang Shi et al., "An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume: 39, Issue: 11, November 2017).

[0085] In the application process, a training set is first constructed using the existing metal workpiece recognition results containing characters, and the improved DB module and CRNN network are then trained and optimized.

[0086] Then, the characters on the metal workpiece to be tested are identified according to steps S1-S7 above. Existing OCR algorithms are also used to identify characters on the same metal workpiece. Comparing the two identification results, the deep learning-based character recognition method for highly reflective metal surfaces provided by this invention improves the recognition accuracy by 11%, while the recognition time for a single metal image (1200W pixel resolution) is only about 1.17 seconds, improving production efficiency and reducing the error rate of character recognition. Furthermore, the improved DB module and CRNN network model provided by this invention have only 11.6M parameters, making it a lightweight network structure.

[0087] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for character recognition on highly reflective metal surfaces based on deep learning, characterized in that, Includes the following steps: S1 acquires images containing characters on a metal surface; S2 performs filtering on images containing characters on metallic surfaces; S3 performs Retinex image enhancement on the filtered image; multi-scale Retinex image enhancement is implemented according to the following formula: (2)； in, This represents the reflection component of the i-th channel, where i = 1, 2, 3 represent the R, G, and B channels, respectively. This represents the component of the i-th channel of the image obtained after filtering in step S2; The number of scale parameters and the weighting coefficients. satisfy , represents the Gaussian blur filter at the k-th scale: ； in, This represents the standard deviation of the k-th Gaussian filter; S4 determines the region of interest (ROI) using a template matching method. Specifically, based on the highly reflective part to be identified, a template containing the ROI framework is constructed. Then, the ROI is matched from the image after image enhancement using a shape matching method. The shape matching process first counts the total number M edge points of the template. Within the ROI to be matched, the cosine of the angle between each point on the template boundary and the direction vector of the corresponding point in the image to be matched is calculated. Assuming the matching cosine values ​​for the M points are... Then the matching score within this ROI region can be expressed as: The P-value ranges from 0 to 1, with a higher match rate indicating a closer match to 1. If P exceeds the set threshold, the ROI region is recorded. S5 performs an affine transformation on the region of interest. Specifically, based on the template used in step S4 and the matched region of interest, an affine transformation matrix is ​​constructed; the region of interest is then subjected to an affine transformation using the affine transformation matrix. S6 performs character detection, including the following steps: S61 performs feature extraction on the image obtained by affine transformation; specifically, the image obtained after affine transformation is fed into the feature pyramid FPN network structure, and the output of the feature pyramid is transformed to the same size by upsampling, and then cascaded to generate feature F; S62 obtains a probability map and a dynamic threshold map of the predicted text based on extracted features; S63 obtains an approximate binarized image based on a dynamic threshold image and a text probability image. The outline of the approximate binarized image is the text box. S7 will take the text box input into the CRNN network and recognize the characters.

2. The method for recognizing highly reflective metal surfaces based on deep learning according to claim 1, characterized in that, In step S2, median filtering is performed on the image containing characters on a metal surface.

3. The method for recognizing highly reflective metal surfaces based on deep learning according to claim 2, characterized in that, The formula for median filtering is as follows: (1)； Where Med represents the calculated median. Indicates the size of the filtering window.

4. The method for recognizing highly reflective metal surfaces based on deep learning according to claim 1, characterized in that... In step S63, an approximate binarized image is calculated from the dynamic threshold image and the predicted text probability image. The outline of the approximate binarized image is the text box. ； In the formula, (u,v) represents the image coordinates. This represents the approximate binarization result. Represents the probability of text. This represents the dynamic threshold, and K represents the amplification factor.

5. The method for recognizing highly reflective metal surfaces based on deep learning according to claim 1, characterized in that, In step S7, the CRNN used consists of three parts: a convolutional layer, a recurrent layer, and a transcriptional layer, from the input end to the output end.

Images