Tire Sidewall Imaging Methods

Abstract

A computer-implemented method is provided for generating a region of interest on a digital image of a tire sidewall, the sidewall having one or more embossed and / or engraved patterns. The method includes generating a histogram of oriented gradients feature map of the digital image, inputting the histogram of oriented gradients feature map into a trained convolutional neural network, wherein the trained convolutional neural network is configured to output a first probability that a pixel region of the digital image contains an embossed and / or engraved pattern based on the input histogram of oriented gradients feature map, and accepting the pixel region as the region of interest if the first probability is equal to or above a first predetermined threshold.

Description

Technical Field

[0001] The present invention relates to methods of reading embossed and / or engraved patterns on a tire sidewall and, more particularly, to a computer-implemented method for generating regions of interest. Background Art

[0002] The outer surface of a vehicle's tire, known as the tire sidewall, carries a text-based code. The code carries information about, for example, the tire brand, manufacturer, age, tire type, expected load, speed rating and size, manufacturing batch information, manufacturer details, and other product information. The code may include, for example, a mixture of one or more letters, numbers, logos, symbols, pictographs, and / or any other visual representation of information. For vehicle users, and especially fleet operators, this information is critical because it provides a consistent and reliable method of tracking tire usage and condition across a fleet, greatly improving the fleet operator's ability to perform data analysis on a fleet's tire inventory and detect when a tire has failed and / or is beginning to fail.

[0003] There have been attempts to automate the process of reading tire sidewalls, however, such systems are either 3D scanner-based systems for indoor and controlled inspection tasks (available from MicroEpsilon, Cognex, and Numetrix) or handheld laser devices for indoor and outdoor applications. Given the structure of such systems, laser components are expensive to manufacture, difficult to calibrate, prone to breakage, and / or still require human operator assistance, so they cannot be said to be truly automated and cost-effective.

[0004] Applying optical character recognition (OCR) to images captured without structured light would significantly reduce hardware costs. However, due to the wear of the sidewall text caused by outdoor use of tires (e.g. due to material corrosion, dust, dryness and / or humidity) and due to the very low contrast of the text (black on black), which is sometimes difficult even for human observers, let alone automated systems, previous attempts at OCR based on color or grayscale images have been unsuccessful.

[0005] The challenge in producing such a system is to be able to read the text quickly enough as a moving vehicle tire drives past the system, or to read the text on a stationary tire quickly enough as the system's field of view moves across the tire. It is also desirable to be able to compensate for changing conditions (e.g., varying weather conditions outdoors at a fleet warehouse and / or dusty / dirty conditions indoors at a fleet warehouse) and produce accurate and reproducible results without the assistance of a human operator.

[0006] An imaging system providing enhanced contrast images is proposed in WO2017060739A1. In particular, lighting is very important for reading embossed or engraved text (such as tire sidewall codes) because the contrast and readability of the text can be improved by shadow casting. Although WO2017060739A1 suggests using image analysis software to perform OCR on these images to read the embossed patterns, the traditional OCR technology described in WO2017060739A1 performs poorly due to being too slow and / or having low accuracy in non-ideal non-laboratory environments.

[0007] Recent developments in deep learning based image classification and text recognition have pushed deep convolutional neural networks (CNNs) to the top of performance tables for text recognition on benchmark datasets, so that now almost all top-ranked results in image processing use deep learning instead of hand-crafted features. However, the deep convolutional neural networks that rank at the top of the performance tables were tuned to work well on benchmark datasets that do not include tire image data. In fact, this does not mean that such networks are successful when used on data such as tire images obtained with a camera in the field. Such deep networks have been used to detect and read text in the wild (i.e., in high noise environments), but in these cases, the data used exhibited a reasonable degree of contrast and color difference between the text being read and the background of the image. Therefore, there is a need for an improved system and method for recognizing text on a tire sidewall. Summary of the invention

[0008] In summary, the present invention relates to a method for more accurately and efficiently identifying regions of interest on images with low contrast and low color difference, such as images of tire sidewalls. The method combines the Histogram of Oriented Gradients (HOG) technique with convolutional neural network layers to improve efficiency and accuracy compared to known techniques, despite low contrast and color difference.

[0009] By more accurately and efficiently identifying regions of interest on low contrast, low color difference images, faults such as cracks near the embossed and / or engraved patterns of the tire that would otherwise be missed can be more easily identified and associated with a specific tire, and these faults are tracked, thereby providing the effect of improving tire safety. In addition, a central database of tire identification information and related tire fault and / or safety information can be updated. It can be used to determine when a tire needs to be repaired or replaced.

[0010] More specifically, the present invention relates to a method for generating a region of interest associated with a user-specified character sequence on a tire sidewall, wherein a HOG of an input image is generated to obtain a HOG feature map, which is used as an input to a convolutional neural network that classifies from the HOG features to determine whether the user-specified character sequence exists. In one embodiment, HOG features can be generated outside and separately from the convolutional neural network using one of the two methods (DalalTriggs and UoCTTI) provided by the VLFeat open source library, for example. In another embodiment, they can be generated using a CNN-implemented HOG analog (such as described in Mahendran and Vedaldi (2015), Understanding DeepImage Representations by Inverting Them, IEEE Conference on Computer Vision and Pattern Recognition, IEEE Compt. Soc). This paper points out that the HOG feature generation using CNN is digitally indistinguishable from the HOG feature generation method provided by the VLFeat open source library, except that it also allows the calculation of HOG feature derivatives, which advantageously reduces the complexity of any subsequent processing operations. The terms HOG features and HOG used here refer to HOG features and HOG generated using the VLFeat open source library and / or CNN methods such as those described in Mahendran and Vedaldi (2015) and other papers mentioned therein.

[0011] Additionally, the following terms are defined as follows:

[0012] “Synthetically generated / synthetic data” – data generated using an algorithm to increase the total amount of data available for training, e.g. when only limited data is available from other sources;

[0013] “Tire Detection” - Identifying which pixels in a tire image correspond to the tire and which pixels correspond to the background (such as a hubcap or vehicle body);

[0014] “Straighten” – maps an image of a curved tire sidewall to one in which the curve has been removed or straightened;

[0015] “Stacked convolutional filters” - a cascade of image processing operations comprising multiple convolutional filters, together forming a convolutional neural network or a portion of a convolutional neural network;

[0016] "Fully connected convolutional layer" - a convolutional filter whose mask size (height, width and number of channels) is the same as the feature map size of the previous layer. It produces an output feature map of the same size as a fully connected layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A five-phase approach according to one embodiment is shown.

[0018] FIG. 2 shows a correction scheme, on which the inner and outer radii of the tire are indicated.

[0019] Figure 3 is a flowchart of a candidate / region of interest generator method according to an embodiment.

[0020] Figure 4 It is a flowchart of a method for generating a histogram of oriented gradients and a corresponding feature map using a HOG-CNN architecture according to one embodiment.

[0021] Figure 5 It is a flowchart of a method for generating a histogram of oriented gradients and a corresponding feature map using a HOG-MLP architecture according to an embodiment.

[0022] FIG6( a ) is a block diagram of a CNN architecture according to an embodiment.

[0023] FIG6( b ) is a block diagram of a CNN architecture according to an embodiment.

[0024] FIG6( c ) is a block diagram of a CNN architecture according to an embodiment.

[0025] Figure 7 is a flowchart of a method for verifying a region of interest according to an embodiment.

[0026] FIG8( a ) is a block diagram of a network architecture according to an embodiment.

[0027] FIG8( b ) is a block diagram of a network architecture according to an embodiment.

[0028] Fig. 9 is a flow chart of a method for locating / verifying a tire sidewall code according to an embodiment.

[0029] Fig.10 is a block diagram of a network architecture according to an embodiment.

[0030] Fig.11 is a block diagram of a network architecture according to an embodiment. DETAILED DESCRIPTION

[0031] like Figure 1As shown, a five-stage method of an embodiment of the present invention is proposed. The five-stage method includes target illumination and high frame rate image acquisition 101, tire detection 102, tire unwarping 103, text detection 104 (wherein the text on the tire sidewall is located by looking for a user-specified character sequence, such as "D", "O", "T"), and code reading 105 (wherein the tire sidewall code containing product information about the tire is detected and recognized).

[0032] These stages can be used together as one system or separately and / or in combination with systems not described herein, for example with the imaging system described in WO2017060739A1, or with a mobile imaging system using a camera and a flash of a smartphone, tablet or other similar device. In other cases where daylight provides sufficient illumination, the flash can be omitted entirely.

[0033] In particular, at the text detection 104 stage, a candidate (i.e., region of interest) generator method 104a is provided that identifies regions of interest that may contain a user-specified sequence of characters. As will be described in more detail below, in a first step, the candidate generator method 104a generates multiple histogram of directional gradients (HOG) feature maps (each feature is a HOG) based on an input image of the sidewall of the tire using a method such as provided by the VLFEAT open source library or using a CNN. In a second step, multiple HOG features are input into a CNN classifier architecture. By first generating multiple HOG features and using them as input to a CNN classifier architecture, it is found that the candidate generator is superior in accuracy to methods based solely on manual features, methods with a single classifier, or methods based solely on deep CNNs in efficiency.

[0034] Furthermore, by first generating multiple HOG features and feeding them into the CNN architecture, the learning task is simplified to classifying multiple HOG inputs into one or more categories to generate regions of interest, rather than classifying multiple low-contrast back-on-back images whose features (i.e., pixel values) are difficult to determine and learn patterns from. This means that the overall system is more generalizable to unseen datasets, such as those found in the wild. In contrast, if a pure CNN architecture (without HOG input) is fed low-contrast back-on-back images, the architecture is more complex (e.g., more layers and / or more complex connections between layers) so that the CNN can learn the image features. The increased complexity leads to increased computational resource overhead, increased memory resource overhead, and decreased efficiency. Therefore, while the pure deep CNN architecture may still outperform the currently proposed architecture combining HOG and CNN (measured only in terms of accuracy), when applied to the real-world problem of tire sidewalls, the pure deep CNN architecture fails in terms of efficiency for real-time applications and low-memory systems. Furthermore, in terms of resource usage, using deep CNNs for candidate generation significantly increases the resource overhead of the system and therefore cannot use resources efficiently in low-resource systems, especially when the proposed HOG-CNN architecture described in this paper can generate multiple candidates that are equally relevant (with orders of magnitude improvement in computational efficiency and reduced memory footprint, thereby overcoming the need for expensive GPUs, memory, and other hardware required by deep CNN architectures).

[0035] It is contemplated that the candidate generator method 104a described herein may therefore improve the performance of any system for generating regions of interest on tire sidewalls based on embossed and / or engraved patterns. The inventors contemplate its use as a standalone invention and / or in conjunction with any known OCR technology.

[0036] The details of the other stages: target illumination and high frame rate image acquisition 101, tire detection 102, tire correction 103, verification of the region of interest proposed by the candidate generator 104b, and text reading 105 are not necessary to realize the advantages provided by the candidate generator method 104a. Before explaining the exemplary implementation of the candidate generator method in more detail, the details of these stages will be described below.

[0037] Image Acquisition 101

[0038] As described above, an imaging system such as that proposed in WO2017060739A1 can be used to obtain a digital image of a tire sidewall on which there is embossed and / or engraved text or pattern. The system only captures a portion of the tire sidewall in any given image, so as the tire rolls past, a series of images are usually taken to ensure that the entire circumference of the tire sidewall is captured, thereby also capturing any portion of the sidewall with embossed and / or engraved patterns.

[0039] Tire Inspection 102 and Correction 103

[0040] Once one or more images have been acquired, a circular Hough transform (CHT) or other suitable technique may be used to detect the circular portion of the tire (i.e., locate its inner and outer radii). Prior to performing CHT, the image may be preprocessed using a Difference of Gaussian (DoG) filter, which not only normalizes the illumination but also enhances the edges. As part of the preprocessing, the image may be selectively downsampled to 1 / 4-1 / 8 of the original size, which improves the efficiency and accuracy of tire detection. The downsampled image (since the center of the tire may be outside the image frame captured by the camera) is then padded with black pixels (i.e., black pixels are added to provide a coordinate system of appropriate size in which the CHT can identify the tire center). Once preprocessing has occurred, the circular connection of the hub cap is detected using CHT, thereby detecting the tire inner radius 204 and outer radius 203 with a certain safety offset, as shown in FIG. 2( a), and the tire inner radius 204 and outer radius 203 correspond to the actual inner radius 201 and actual outer radius 202 of the tire as shown in FIG. 2( b). However, sometimes, false circles are detected due to the presence of another dominant roundness in the image (such as the roundness of the wheel arch or hub cap as shown in Figure 2(b), which sometimes may be more dominant due to greater contrast). To avoid this, all captured images related to a particular tire (axle) are processed in n radius ranges (in parallel threads). The detected circles are then used to generate a radius range histogram. The radius corresponding to the radius range bin in which the number of detected circles is the largest is selected as the best detected tire inner radius 201. This approach is simple (i.e., resource-efficient) and is able to effectively and successfully remove any outliers due to the consistency generated by the moving tire, where the tire roundness dominates due to the field of view of a given image.

[0041] Once the junction of the hubcap and the tire (i.e., the tire inner radius 201) is detected, a second circle is selected corresponding to the outer radius 202 of the tire 200 at a fixed offset from the first radius. This is sufficient to cover the area where tire sidewall text (e.g., text of a DOT code) is expected to appear, because tire sidewall text typically falls near or in the middle of the inner radius of the tire 200, rather than near the tread near the outer radius 202 of the tire 200. Since it is close to the inner radius, the detected inner radius is also reduced by a fixed number of pixels, as shown in Figure 2(a), to ensure that boundary cases are handled correctly.

[0042] After tire inspection, the radial image slices between the inner radius 201 and the outer radius 202 are rectified to a rectangular grid using a Polar-to-Cartesian mapping. This not only corrects the roundness but also only displays the necessary parts of the image, thereby improving the efficiency of the next stage.

[0043] The first three stages of the process, namely target illumination and image acquisition 101, tire detection 102 and correction 103, can be implemented in any suitable computer language (either all algorithms from scratch or preferably using OpenCV), but other computer vision libraries and vision processing techniques can also be used.

[0044] Text detection: DOT detection 104

[0045] In the text detection 104 stage, a machine learning based method for text detection and localization is used. The rectified image from the tire rectification stage 103 is used. Due to industry regulations, most commercial tire sidewall codes are preceded by the character sequence "D", "O", and "T", which stands for the United States Department of Transportation. In this example, the DOT character sequence is used as an anchor to locate the text associated with the tire sidewall code. However, it is contemplated that other character sequences, letters, numbers, logos, symbols, pictographs, and / or any other visual representation of information may also be used as an anchor with which the text of the tire sidewall code can be located. For example, if a fleet operator only uses one brand of tires, the associated brand logo or trademark may be used to locate the text on the tire sidewall.

[0046] The purpose of the anchor is to narrow the search space because, in most cases, it precedes the rest of the text of the tire sidewall code. The text detection 104 stage has two cascades, namely a collection of subsequent image processing operations, candidate (i.e., region of interest) generation 104a, and subsequent verification or text localization 104b. As described above, it is contemplated that the candidate generator method 104a described herein can be used as a stand-alone method, the output of which is processed separately using known image processing techniques (e.g., using third-party owned equipment) that rely on candidate (i.e., region of interest) generation to detect and / or recognize text on the tire sidewall.

[0047] Candidate Generation 104a

[0048] Since the contrast of text is very low, low-level feature-based methods (such as the edge boxes proposed by "Zitnick and Dollar, EdgeBoxes: Locating object Proposals from Edges, ECCV, European Conference on Computer Vision, 2014") are found by the inventors to be inappropriate for candidate generation because the strong edges of other parts of the tire (most of which do not contain text) dominate, resulting in a large number of candidates that do not contain any text. Determining which of these candidates contains text significantly increases resource overhead.

[0049] In addition, although hand-crafted features have been successfully used for text detection (e.g., “Wang et al, End-to-end Scene Text Recognition, Proceedings of the 2011 International Conference on Computer Vision, IEEE Computer Society, Washington, ICCV'11 pp 1457-1464 DOI10.1109 / ICCV.2011.6126402”, “Mishra et al, Top-down and bottom-up cues for scene text recognition, 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp 2687-2694, DOI 10.1109 / CVPR.2012.6247990”, and “Mishra et al, Image Retrieval Using Textual Cues, 2013 IEEE International Conference on Computer Vision and Pattern Recognition, pp 3040-3047", but these techniques are too slow for industrial applications to recognize tire sidewall text in a reasonable time.

[0050] In particular, when the inventors attempted to use HOG in combination with a support vector machine (SVM) classifier in a sliding window manner, fairly accurate text detection results (i.e., detecting the character sequence "D", "O", "T") were produced, but the size of the images (500x 2000 to 4000 pixels) still meant that several minutes were required to scan each image, and each tire had several images associated with it. This time scale is too long and is unacceptable for industrial applications. In industrial applications, if the system is to outperform a system in which a human operator manually reads and records the tire sidewall code, it is not reasonable to expect fleet operators to wait that long for each tire to be scanned. Ideally, a suitable system would need to obtain end-to-end results in less than a minute. In addition, such a system should be able to run with only CPU-based processing (because the cost of GPUs would be very expensive for this application). Lower resolution images, such as those obtained by lower resolution cameras or by downsampling higher resolution images, are not suitable for such small and low-contrast text recognition.

[0051] Deep CNN-based branching architectures, such as Faster-RCNN (described, for example, in "Ren et al, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Advances in Neural Information Processing Systems 28, Curran Associates, Inc., pp 91-99, 2015"), are an alternative approach that uses a region proposal network to scan the image and generate proposals for the localization branch. Faster-RCNN has been shown to be accurate while maintaining GPU efficiency. However, it is too expensive to use deep backbone networks (such as those typically required for Faster-RCNN (e.g., VGG16 or ResNet50)) on a CPU to generate feature maps and proposals on images of the size used in tire sidewall imaging, thus requiring a large memory GPU (11GB or larger), which increases the overall system cost to the point that it would be more cost-effective for fleet operators to hire human operators to manually read and record tire sidewall codes. In addition, the GPU may require additional cooling, which potentially limits its use in outdoor scenarios on hot weather.

[0052] As described above, the present invention provides a solution to this problem by combining the generation of HOG features with a CNN-based classifier to efficiently generate candidates. In one architecture, the HOG features are generated using a known method, such as the method provided by the VLFeat open source library, and then input into a CNN-based classifier. In another architecture, the HOG features are generated by a CNN and input into a CNN-based classifier. The first architecture is described herein as HOG-MLP (Multi-layer Perceptron) and the second architecture is described as HOG-CNN.

[0053] Training Run

[0054] All CNN training runs discussed in this article are performed in Matlab using the MatConvNet library of A. Vedaldi and Lenc (2015) using back-propagation using stochastic gradient descent as the optimizer, as described in A Vedadi and Lenc (2015) MatConvNet-Convolutional Neural Networks for Matlab, Proceedings of the ACM, Int. Conf. on Multimedia. However, it is contemplated that any suitable alternative training and optimization techniques and libraries may also be used, such as TensorFlow, Caffe, Torch, etc. In addition, in one example, text class training data may be synthetically generated, while background class training data may be extracted from actual tire images. However, it is contemplated that synthetic data generation may not be required at all, such as when sufficient data from real tire images is available. In addition, a drop out layer may be used to prevent overfitting. In addition, while the network described herein uses one or more 50% dropout layers during training to prevent overfitting, it is contemplated that other techniques for preventing overfitting may be used instead, such as cross-validation, training with more data, removing features, early stopping regularization, etc. A difference of Gaussian (DoG) filter is applied to the input data for illumination normalization and edge enhancement. Other contrast normalization techniques such as histogram equalization or adaptive histogram equalization may also be used.

[0055] Synthetic Data Generation

[0056] As described above, if there is not enough real image data available, synthetic data generation can be optionally used. Since the automatic tire sidewall text reader deployed in the wild must read the sidewall text under different light, weather and wear conditions, a large amount of training data may be required to achieve good generalization. Collecting large data sets in the wild is a very expensive and time-consuming process. Instead, training data can be synthetically generated using several different fonts and text rendering engines. Initially, a black and white text mask is created using fonts of various random sizes. The mask can then be incrementally applied (adding multiple copies or moving the rendering position in a small neighborhood (dx, dy pixels)). This occurs in different directions (representing rotated shadows) and lengths (representing different shadow lengths). The image mask is then fused with the tire background to produce a realistic embossed / engraved text image as they should appear on a real tire sidewall image. Assuming that the directional gradient histogram features are used as input to the CNN classifier, in some embodiments, the training data may include multiple directional gradient histogram feature maps generated from multiple digital images of the tire sidewall.

[0057] Implementing the Candidate Generator Method 104a

[0058] Figure 3 is a flowchart showing the steps of the candidate generator method 304, which is similar to Figure 1 An embodiment of the present invention corresponding to the candidate generation step 104a in . A digital image 300 of a portion of a rectified tire sidewall, such as obtained as described above, is used as input. The tire sidewall has one or more embossed and / or engraved patterns, such as a tire sidewall code. A directional gradient histogram of the digital image and its associated feature map 301 are generated. The generated directional gradient histogram is input into a trained neural network 302. The trained neural network is configured to output a first probability 303 that a pixel region of the digital image contains an embossed and / or engraved pattern based on the input directional gradient histogram and its associated feature map. If the first probability is equal to or higher than a first predetermined threshold 305a, the pixel region is accepted as a region of interest and output 306. Otherwise, the pixel region is rejected 305b. In this way, the candidate generator method 104a can generate a region of interest associated with one or more embossed and / or engraved patterns on a digital image.

[0059] As described above, two alternative ways to generate HOG features are provided. In the first method, the HOG features are generated outside the trained CNN classifier and separately from the CNN classifier (e.g., using the method provided by the VLFeat open source library). This is described in this article as HOG-MLP. In the second method, the HOG features are generated by CNN. This is described in this article as HOG-CNN.

[0060] Figure 4 401 and 402. It is a flow chart of a method for using a CNN-generated directional gradient histogram and its feature map according to the above HOG-CNN architecture. Specifically, after receiving an input 400 of a digital image of a portion of a tire sidewall, a stacked convolution filter 402 is used to generate a directional gradient histogram and a corresponding HOG feature map, which is output 403.

[0061] Figure 5 It is a flow chart of a method for generating a histogram of oriented gradients 501 and a corresponding HOG feature map using a separate external HOG generator 502 according to the above HOG-MLP architecture. Specifically, after receiving an input 500 of a digital image of a portion of a tire sidewall, the HOG generator is used to generate a histogram of oriented gradients and a corresponding HOG feature map, which is output 503.

[0062] The effect provided by combining HOG features with CNN classifiers is that the total number of generated candidates / regions of interest is significantly less and there are fewer false positives compared to pure manual techniques such as HOG+SVM (i.e., support vector machine classifiers in a spatial sliding window manner). Another advantage is that the overall scanning / computation time is much shorter because the generation of HOG features is shallower and / or more efficient than using a deep convolutional network alone to generate candidate / regions of interest. As described above, the inventors believe that one reason is that HOG generation provides image substructures (or feature maps) to CNN classifiers without the CNN classifier having to learn image substructures (or feature maps) from raw image data, thereby effectively skipping the need for deep CNN architectures. The deep architecture will need to learn image substructures only from training data, which is particularly difficult when the image has low contrast between foreground and background because it will require more orders of magnitude of data and training time. In contrast, training HOG-CNN can be performed with a relatively much smaller data set and can be performed very efficiently with a CPU.

[0063] HOG-CNN

[0064] FIG6( a) is a block diagram of a HOG-CNN architecture according to an embodiment. A fully connected convolutional network is inserted at the end of the stacked convolutional filters (i.e., a cascade of multiple image processing operations that constitute the HOG feature generation layer ending with a HOG decomposition layer). Together, this provides a complete CNN architecture that terminates with a cross entropy loss layer (for training) or a softmax layer (for testing or application or operation) that outputs the probability that a given input image contains an embossed and / or engraved pattern. This network is shallow, having fewer convolutional layers and channels than deep networks (e.g., those in which CNN layers are used to generate deep features). As a result, shallow networks greatly increase speed, making them more suitable for reading tire sidewall text in the field.

[0065] As described in “Mahendran and Vedaldi (2015), Understanding Deep Image Representations by Inverting Them, IEEE Conference on Computer Vision and Pattern Recognition, IEEE Compt. Soc”, although a CNN layer for HOG is used, it is contemplated that any suitable CNN-based HOG layer implementation may be used instead. In addition, any HOG method described in “Dalal and Triggs (2005), Histograms of Oriented Gradients for Human Detection, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'o5)–Volume 1–pp886-893, DOI 10.1109 / CVPR.2005.177” and “Felzenszwalb et al (2010), UoCTTI, Object Detection with Discriminatively Trained Part-Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence 32(9):1627-1645, DOI 10.1109 / TPAMI.2009.167” can also be used.

[0066] The example architecture shown in FIG6 has a 60×130 pixel DOT text input 601 image (preprocessed using the Gaussian difference technique discussed previously). As described in “Mahendran and Vedaldi (2015), Understanding Deep Image Representations by Inverting Them, IEEE Conference on Computer Vision and Pattern Recognition, IEEE Compt. Soc”, stacked convolutional filters are used to extract HOG features, where the directional filter K=2 times the number of directions (O), where K is an index, K=1,...k. The Kth directional filter is given by:

[0067] G k =G x u 1k +G y u 2k in

[0068] and

[0069] The directional filter takes the input along the direction vector u k The projection is converted to gu k (where g is a constant). After directional filtering, HOG binning 602 may be performed by the following activation function:

[0070]

[0071] The stacked convolutional filters are shown in FIG. 6( a ) as starting with a Conv(3x3x1x2*0) architecture 601, however, it should be understood that other filter architectures (e.g., Conv(5x5x1x2*0) or Conv(7x7x1x2*0)) may also be used. Other examples of filters may be found in the HOG implementation described above.

[0072] In HOG feature extraction, the binned gradients are pooled into multiple cells and then combined into 2×2 blocks. This is done by stacking linear filters 603. After normalization 604 (L2 norm), the blocks are decomposed back into a cell structure and the values ​​are clamped 605 to 0.2 (i.e., max{x, 0.2}). In the example implementation described herein, the number of directions (h do ) to box the directional gradients, accompanied by a set of undirectional gradients (h uo). Therefore, a total of 3×O channels are used in the HOG decomposition layer 606.

[0073] Using the example above, for an input image with 60(H) x 130(W) pixels, the CNN-based HOG generates a feature map of 7 x 16 x 27 with a cell size of 8 x 8 and 9 directions. Other cell sizes and numbers of directions may also be used.

[0074] The HOG output is then input to a classifier (e.g., a multilayer perceptron or MLP) 607a, 607b. In this example, the classifiers 607a, 607b include a randomly initialized fully connected (FC) layer 607a with a mask size of 7×16×27 CHs (CHs represents the number of channels in the current layer). This is followed by a 50% dropout and another FC layer 607b, as shown in Figure 6(a). Dropout is a regularization technique that prevents overfitting by simply skipping some neurons. It is contemplated that other techniques may also be applied to prevent overfitting, examples of which are described above, such as cross-validation, training with more data, removing features, early stopping, regularization, etc. Since the HOG feature generation and subsequent classification are performed using the FCs connected to each other as a unified CNN architecture, the term HOG-CNN is used.

[0075] A final cross entropy loss layer 608 is also provided to train the CNN classifier by back propagation to recognize the “DOT” text 609. In a similar manner to OverFeat (Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & Lecun, Y. (2014). Overfeat: Integrated recognition, localization and detection using convolutional networks. In International Conference on Learning Representations (ICLR 2014)), the architecture in FIG. 6(a) uses convolutional layers as FC layers, and the HOG-CNN network can scan the entire image if it is larger than the minimum required size (i.e., 60x130 pixels).

[0076] Training such a network can be difficult since few layers are predefined and the final classifier is randomly initialized. In this case, it was trained on a dataset containing less than 600K images (60x130 pixels in size) in total, where the DOT class was synthetically generated. The training set contains a synthetically generated DOT class and a background class consisting of a mixture of non-DOT text, edges, textures, and pure background. A total of 80-90 training epochs are considered sufficient when saturation is reached. Continuing training tends to further overfit the model. However, since the network is shallow and uses sparse filters, it can be trained efficiently even on a CPU (with a training time of approximately less than 5 hours).

[0077] It should be understood that the above exemplary architecture is for illustrative purposes.

[0078] As mentioned above, the problem of high computational overhead and cost of techniques such as HOG+SVM (in a spatial sliding window manner) or deep CNN techniques for recognizing tire sidewall text in the wild can therefore be addressed by using the concept of inputting the output of the HOG implementation into a shallow CNN.

[0079] HOG-MLP

[0080] For HOG-MLP (instead of using a unified CNN architecture), a separate HOG implementation 610 (such as those in the VLFeat library (Vedaldi and Fulkerson 2008, An Open and Portable Library of Computer Vision Algorithms, ver(0.9.16), p, http: / / www.vlfeat.org)) can be used to extract HOG features from the input 601 and then feed them into a multi-level MLP (HOG-MLP) network, as shown in FIG6(b). In the VLFEATHOG implementation 610 used in this example, the gradients are binned with 3*0+4 texture components. Therefore, for an input 601 of image size 60(H) x 130(W), with an 8x 8 HOG cell size and 12 directions (40 components in total), the first layer 611 in the network is 8x 16x40CHs. The cell size and number of directions are selected by systematic search to achieve the best possible detection accuracy on the cross-validation dataset. Other cell sizes and numbers of directions may also be used. It is trained on a dataset of over 1 million images, including 11 classes (nC=11) (including 7 synthetic DOT classes (round / square / narrow and wide fonts, clear and blurred appearances, long and short shadows, single and double spacing between characters, and other variations), and 4 background classes (divided into pure background, non-DOT text, and edges / textures)). A second layer 612 and a cross-entropy loss layer 613 are also provided. The output 614 is mapped to a binary class probability (i.e.: DOT / non-DOT), which is done by predetermining which of the output classes of the cross-entropy loss layer correspond to DOT codes and which do not. This multi-class representation enables prior knowledge to be incorporated into training, and thereby increases, for example, the generalization of the network, enabling it to cope with changes in the lighting configuration of image acquisition, for example during installation, calibration, and / or hardware product development.

[0081] If no changes to image acquisition and / or lighting are required, the light / shadow directions in the acquired images are more consistent. In this case, as shown in Figure 6(c), an alternative illustrative example of a HOG-MLP candidate generator is provided, where the cell size = 8×8, O = 16 (constituting a total of 52 components), but with only four output classes nC = 4 (i.e., DOT, pure background, edge / texture, non-DOT). The output 615 is again mapped to a binary classification (DOT / non-DOT). In this example, the network was trained on a dataset of just over one million images, where the DOT class was synthetically generated as described above. For these two illustrative HOG-MLP networks, satisfactory results were obtained after 30-50 epochs of training. Like HOG-CNN, these sparse networks can also be trained efficiently on the CPU, which is not possible in deep CNN implementations.

[0082] As with HOG-CNN, it should be understood that the above example of a HOG-MLP architecture is for illustrative purposes. Thus, by using the concept of inputting the output of a HOG implementation into a shallow CNN, the high computational overhead and expense of techniques such as HOG+SVM (Support Vector Machine Classifier in a Spatial Sliding Window manner) or deep CNN for identifying tire sidewall text in the wild can be addressed.

[0083] Compare

[0084] Comparing two candidate generation methods, HOG-CNN and HOG MLP, the scanning time for a 500x 3000 pixel image is about 550 and 250ms on an Intel Corei7 3.6GHz CPU, respectively. For both HOG-CNN and HOG-MLP, this is significantly faster than the minutes of manual HOG+SVM implementations or deep CNN-based implementations in a sliding window manner.

[0085] In HOG-MLP, it is not possible to backpropagate through the feature extraction stage because the HOG part of the architecture is not part of the CNN architecture. In contrast, in HOG-CNN, backpropagation through the entire network is possible, increasing the ability of the HOG-CNN implementation to adapt to changes in the data.

[0086] The inventors have observed that the accuracy of HOG-CNN and HOG-MLP structures using the same cell size and number of orientations is comparable, although HOG-CNN produces fewer candidates and therefore generalizes better than HOG-MLP (e.g., due to backpropagation).

[0087] Text positioning: DOT positioning 104b

[0088] To finally locate and verify the tire sidewall text (ie, tire sidewall DOT code) from the filtered candidates, a classifier may be applied to the generated regions of interest to accept or reject one or more of them as false positives.

[0089] Figure 7 It is shown that according to the corresponding Figure 1 Flowchart of method 704 of the embodiment of step 104b in . The region of interest 700 output by candidate generator method 104a is input into classifier 701. Classifier 701 locates the text in the region of interest, thereby verifying the real region of interest and false positive. In other words, for each region of interest, it determines which are false positives and which are not. False positives are discarded, and the real region of interest is selected. Classifier 701 can be a deep neural network, which outputs the probability 702 that a given region of interest does contain an embossed / engraved pattern (such as the above-mentioned "D", "O", "T" character sequence). If the probability is lower than a predetermined threshold, the given region of interest is determined to be a false positive and rejected 703b. Otherwise, the given region of interest is accepted 703a as a real region of interest and output 704.

[0090] An illustrative example of a deep network 801 that can be used as a classifier 701 is shown in FIG8( a). It is contemplated that other similar architectures can be used, for example, the architecture described in “Jaderberg et al (2016), Reading Text in the Wild with Convolutional Neural networks, International Journal of Computer Vision 116(1): 1-20 DOI 10.1007 / s11263-015-0823-z” can be used. In fact, the method of rejecting false positives is irrelevant to the advantages provided by step 104a and is not necessary to achieve the advantages provided by step 104a. In order to compare the detection probability with a predefined threshold, a Softmax layer at the end of the CNN classifier can be used.

[0091] The training set for the illustrative network 801 contains multiple DOT and background classes (1.6 million images, 10 classes: 7 DOT classes and 3 background classes (pure background, edge / texture, and non-DOT text)). In the example shown in Figure 8(a), an input 32×100 pixel DOT text image 800 is used, that is, the detection result of the HOG-CNN or HOG-MLP is 60×130 pixels, which is downsampled to 32×100 pixels. The classification result 802 is then mapped to a binary output (DOT / non-DOT). Similar to HOG-MLP, when the imaging and lighting settings no longer need to be changed during, for example, installation, calibration, and / or hardware product development and to make the dataset more consistent, the text localization network 1001 can be reduced to a 4-branch classifier 803 (DOT, pure background, non-DOT text, and edge / texture), as shown in Figure 8(b). As a result, many false positives generated by the candidate generator can be rejected and only a few strong candidates are retained. If necessary, false positives that creep in at this stage can be resolved by text recognition in the code reading stage 105 .

[0092] Code Reading 105

[0093] Code reading 105 may include, for example Figure 1 Two stages are shown: text or character detection / localization 105a (where the characters of the code are located) and text or character recognition 105b (where the characters are recognized and output). Steps 105a and 105b can be performed in a single step by the same classifier, or by separate multiple classifiers. The code slice of the image (i.e., the portion of the image containing the DOT code and the characters after the "DOT" anchor position) is first pre-processed to crop the code slice to the text height using low-level edge filtering. The code slice height is then adjusted to 40-50 pixels based on the stride of the code detection network (the number of pixels skipped between two consecutive detection windows on the input image).

[0094] Fig. 9 is a flow chart of a method 901 for locating and / or classifying a tire sidewall code (i.e., reading an embossed and / or engraved pattern of a tire sidewall), the method 901 using a method corresponding to Figure 1 A single classifier of steps 105a and 105b is used. An area adjacent to the verified area of ​​interest is selected and input into a single classifier 901. The classifier 901 can then locate the character / symbol of the code within the selected area and output a probability 903 that a given character / symbol is recognized as, for example, a character (such as a letter or a number), based on which a reading result of the embossed and / or engraved pattern (i.e., the tire sidewall code) can be output.

[0095] Alternatively, Fig.10 and Fig.11 Independent multiple networks are shown that can independently perform steps 105a and 105b. It will be appreciated that a variety of OCR techniques exist, and it is contemplated that any such technique may be used once the candidate / region of interest has been generated (as described in step 104a).

[0096] refer to Fig.10 , since the text has very low contrast with the background, it is necessary to Fig.10 The dense prediction mechanism provided by the architecture 1001 shown. In CNN, the max pooling layer downsamples the image to increase the span of the network. Removing the max pooling layer will allow dense (per pixel) prediction, but will greatly increase the parameter space, which will have an impact on efficiency and accuracy. Regularization techniques, such as DropOut, combined with MaxOut activation, can help improve accuracy. Therefore, Fig.10 As shown, MaxOut is used in this architecture. The inventors observed that if ReLU is before the MaxOut layer, the network converges quickly to the minimum during training. Fig.10 The input 1000 is shown as a DoG image with a size of 32×32 pixels. Other network architectures are also envisioned, such as "Goodfellow et al (2013), Maxout Networks, Proceedings of the 30 th International Conference on Machine Learning–Volume28, JMLR.org, ICML’13, pp III-1319-III-1327” and those described in “Jaderberg et al (2014), Deep Features for Text Spotting, European Conference on Computer Vision”. Finally, in the same way as in HOG-CNN and HOG-MLP, a fully connected (FC) layer consisting of convolutional layers allows the network to slide over the entire code piece, detecting and localizing text on the way and avoiding any need for a spatial sliding window mechanism.

[0097] In this example, training is performed on a 700K image dataset, where text classes are synthetically generated as described above. The background class is extracted from an actual tire piece. It contains a single edge, ridge pattern, casting or mold shape and a pure background. The output is mapped to a binary class probability (i.e., text / non-text). As previously described, the character detector generates a bounding box by scanning the entire code piece through convolution. The box detected is thus centered in the area where the probability of text occurrence is highest. Non-maximum suppression is applied to the detected box to filter candidates. A character classifier can also be optionally used for character detection. However, the inventors have found that a dedicated classifier for code character detection, separate from the character classifier for code text recognition, performs better.

[0098] As mentioned above, in this illustrative example, Fig.11 The separate character recognition network 1101 shown is used to perform step 105b. In step 105a, for example, Fig.10 After performing positioning according to the architecture shown in , the detected code character positions are used to extract characters that are fed into the character classifier network 1101. Other character classifiers may also be used, such as the character classifier described in "Jaderberg et al (2016), Reading Text in the Wild with Convolutional Neuralnetworks, International Journal of Computer Vision 116 (1): 1-20 DOI 10.1007 / s11263-015-0823-z". The network has categories for numbers 0 to 9, uppercase letters A to Z (excluding I, Q, S and O that are not used in tire DOT codes) and seven background categories, constituting a 39-branch classifier that maps to 33 categories (32 character categories and 1 background category). The model is trained on a synthetic character dataset of approximately 700,000 images of the inventors. If desired, the classifier can also be trained to recognize specific brands, logos or symbols in tire sidewall codes.

[0099] Advantages

[0100] Since the proposed system is an industrial system, accuracy and efficiency are equally important. In particular, the candidate / region of interest generator described above in step 104a provides a significant increase in the efficiency of the tire sidewall reading system without suffering a significant decrease in accuracy. The inventors contemplate that the candidate / region of interest generator can therefore be used with any known computer vision and OCR technology applied to tire sidewall reading, whose methods in turn require the generation of candidate / regions of interest.

[0101] Accuracy

[0102] The accuracy ultimately depends on the data sample analyzed. The training error of the architecture described here is below 5%. If synthetic training data is mixed with real image data and / or training time data augmentation such as affine transformations are added, the overfitting of the network can be reduced even further. As a result, HOG-CNN and HOG-MLP provide less than 5% false positive rate for the generation of regions of interest on tire sidewall text. This is despite large differences in tire height, radius and position relative to the wheel arch.

[0103] efficiency

[0104] For an industrial system where the end user waits for results, efficiency is critical. GPUs (Graphics Processing Units) have been widely used in deep learning-based systems, but deploying GPUs means expanding the total system cost because they are deployed in each imaging site. As demand increases and two GPUs are required for each site (one each on the right and left side of the vehicle), keeping costs low becomes key. Therefore, as mentioned above, a CPU-based system is ideally sought.

[0105] Scanning an entire rectified image (average size 500 x 3000 pixels) using a deep network takes more than 20 seconds on a Core i7 3.6GHz CPU (requiring 496MB of parameter memory). In fact, when some of the best performing object / text detection algorithms (i.e., those ranked high on the benchmark dataset) are applied to tire sidewall text imaging, they quickly become computationally bottlenecked.

[0106] In contrast, the proposed shallow networks (HOG-CNN or HOG-MLP) require only 1 to 3MB of parameter memory. When only the candidates thus generated are subsequently scanned in depth, the total scanning time is reduced to about 3 seconds. This is an order of magnitude improvement in efficiency (almost 95% speedup), as well as a significant reduction in the overall cost and complexity of the system (since it is based solely on CPU operations). Since the recall of HOG-CNN or HOG-MLP is close to 100%, there is no noticeable compromise in accuracy. In this way, the end-to-end results of processing the image (tire detection and correction, then the rectified image of 500x 3000 pixels obtained by scanning at three different scales, and then detecting and reading the code) on the above CPU takes an average of 3 to 5 seconds.

[0107] Although the present invention has been described according to the preferred embodiments as described above, it should be understood that these embodiments are illustrative only, and the claims are not limited to those embodiments. Those skilled in the art can make modifications and substitutions according to the present disclosure, and these modifications and substitutions are considered to fall within the scope of the appended claims. Each feature disclosed or shown in this specification may be incorporated into the present invention individually, or with any other feature disclosed or shown herein, in any appropriate combination.

[0108] For example, while FIG. 6( a) contemplates two fully connected convolutional layers 607 a, 607 b, this can be reduced to one layer, thereby further reducing computational overhead at the expense of accuracy. In other words, the convolutional neural network may include one or two fully connected convolutional layers. Alternatively, to increase accuracy, the number of fully connected layers may be increased to three or more layers at the expense of computational complexity. However, it is contemplated that using more than two layers may increase computational complexity to such an extent that computational time is increased to levels that are unacceptable to fleet operators and / or require a GPU, thereby reducing or entirely eliminating any advantage gained by using a HOG-CNN or HOG-MLP. While this may not be a problem for an ideal laboratory environment, it is a problem for industrial systems where cost and efficiency are priorities, and therefore shallow networks offer greater advantages.

Claims

1. A computer-implemented method for generating a region of interest on a digital image of a tire sidewall having one or more embossed and / or engraved patterns, characterized in that: The method comprises: Correcting the digital image, comprising: Detecting n radius ranges related to tire roundness, where n is multiple; Generate a radius range histogram; identifying a tire inner radius and a tire outer radius from the radius range histogram; Generate a directional gradient histogram feature map of the rectified digital image; Inputting the histogram of oriented gradients feature map into a trained neural network, wherein the trained neural network is configured to output a first probability that a pixel region of the rectified digital image contains the embossed and / or engraved pattern based on the input histogram of oriented gradients feature map; and If the first probability is equal to or higher than a first predetermined threshold, the pixel region is accepted as the region of interest.

2. The computer-implemented method of claim 1, wherein: The generating of the histogram of oriented gradient feature map is performed by stacking convolution filters in a trained convolutional neural network.

3. The computer-implemented method of claim 1 , wherein: The generating of the histogram of oriented gradients feature map is performed by a histogram of oriented gradients generator separate from the trained neural network.

4. The computer-implemented method of claim 1, wherein: The trained neural network includes one or two fully connected layers.

5. The computer-implemented method of claim 2, wherein: The trained convolutional neural network is trained on training data comprising a plurality of histogram of oriented gradients feature maps generated from a plurality of rectified digital images of tire sidewalls.

6. The computer-implemented method of claim 5, wherein: The training data also includes synthetic data.

7. The computer-implemented method according to any one of claims 1 to 6, characterized in that: The method further comprises If the first probability is lower than the first predetermined threshold, the pixel region is rejected as a region of interest.

8. The computer-implemented method according to any one of claims 1 to 6, characterized in that: The method further comprises applying a classifier to the region of interest; wherein the classifier is configured to output a second probability that the region of interest contains the embossed and / or engraved pattern; and If the second probability is lower than a second predetermined threshold, the region of interest is determined to be a false positive.

9. A method for reading an embossed and / or engraved pattern on a tire sidewall, characterized in that: The method includes: providing a digital image of the sidewall of the tire; Correcting the digital image, comprising: Detecting n radius ranges related to tire roundness, where n is multiple; Generate a radius range histogram; identifying a tire inner radius and a tire outer radius from the radius range histogram; generating a region of interest on the rectified digital image; applying a classifier to determine whether the region of interest is a false positive and discarding the region of interest if it is a false positive, or selecting the region of interest if it is not a false positive; selecting a region of the rectified digital image adjacent to the selected region of interest; applying a classifier to said region of said rectified digital image adjacent to said region of interest to read said embossed and / or engraved motif, The generating of the region of interest comprises: Generating a directional gradient histogram feature map of the corrected digital image; Inputting the oriented gradient histogram feature map into a trained neural network, wherein the trained neural network is configured to output a probability that a pixel area of ​​the rectified digital image contains the embossed and / or engraved pattern based on the input oriented gradient histogram feature map; and If the probability is equal to or higher than a predetermined threshold, the pixel region is accepted as the region of interest.

10. The method according to claim 9, characterized in that The generating of the histogram of oriented gradient feature map is performed by stacking convolution filters in a trained convolutional neural network.

11. The method according to claim 9, characterized in that The generating of the histogram of oriented gradients feature map is performed by a histogram of oriented gradients generator separate from the trained neural network.

12. A data processing device comprising means for performing the steps of the method according to any one of claims 1 to 11.

13. The data processing device according to claim 12, characterized in that The steps are executed by a central processing unit (CPU).

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer readable storage medium has stored thereon the computer program according to claim 14.

Images