System for detecting and classifying anomalies in rubber products

Abstract

The invention relates to a system for detecting and classifying anomalies in rubber products. The detection and classification system (100) performs a method for detecting and classifying anomalies based on an image of a tire surface. The invention also relates to a tyre production site comprising the disclosed detection and classification system (100).

Description

Technical Field

[0001] The present invention relates to a vision-based industrial inspection and classification system for inspecting rubber products such as tires. The disclosed inspection and classification system enables a data-based correlation to be established between imposed theoretical quality standards and field reality. Background Art

[0002] In the field of industrial processing, it is difficult to check the surface of tires intended to be installed on wheeled vehicles. Due to the black rubber material forming these tires (because carbon is used to enhance the elastomer mixture), these tires usually appear black. Therefore, it is difficult to achieve automatic detection of abnormalities on the surface of the tire. In addition, the form of the abnormality can be related to its severity, and different abnormalities vary according to different abnormal categories, including but not limited to various stains (including oil stains), bumps, scratches, surface roughness, tread marks, cuts, contrast differences, tread erosion, notches, smooth teeth, missing materials, excess materials, black depth differences, cracks, engraved materials, and local discoloration. As used herein, the term "abnormality" refers to the detected difference (e.g., represented by a deviation from a standard value) without containing any other information (therefore, "abnormality" is not automatically interpreted as a "defect").

[0003] There are some solutions in the prior art to overcome this challenge. For example, the applicant's patent FR3103555 discloses a system for evaluating the condition of a tire by acquiring visual images. Figure 1 (corresponding to patent FR3103555 Figure 5 ), the disclosed system utilizes both a two-dimensional camera (or "sensor") and a three-dimensional camera to acquire images of a tire 10 having an exterior region (including sidewalls 12, shoulders 14, and tread (or "crown") 16) and an interior region (including inner side 18 and inner ring 20). The two-dimensional sensor is used for both the interior and the exterior (in embodiments that include three-dimensional sensors, these sensors are used only for the exterior surface of the tire 10). In the method performed by the disclosed system, the tire 10 enters the system, is blocked in the system, and is set in a rotational state so that the various sensors can capture images of the tire (e.g., by quickly lowering the sensor until it is close to the area where it should be located, and then by switching to a slow speed to accurately adjust the position).

[0004] The latest developments also employ artificial intelligence (or "AI") methods, in particular deep learning methods, which enable the interpretation of images acquired of tires (including the acquisition device adopted in patent FR3103555). Recent advances in deep learning and data analysis techniques (combined with platforms for computing and storing data) have opened the way to the development of new methods in the field of industrial vision. The main purpose is to use the "eyes of the machine" to identify anomalies in images captured by image capture devices (such as one or more sensors and / or cameras) and to input these captured images into vision algorithms for calculation.

[0005] In the field of industrial vision, the prior art proposes some applications for detecting and classifying tire anomalies during the production process. Such applications propose a tire appearance anomaly detection method that combines histogram of oriented gradients (HOG) and local binary pattern (LBP) features (see Liu, Hongbin et al., “Tire appearance defect detection method via combining HOG and LBP features,” Front. Phys., DOI: 10.3389 / fphy.2022.1099261 (January 12, 2023) (“Liu reference”). In this method, a tire image dataset is constructed to provide images of tires with anomalies and images of tires without anomalies. Next, the HOG and LBP features of the tire images are extracted and analyzed respectively to train a support vector machine (SVM). Finally, the classifier of the SVM calculates the prediction score of the test image by combining the HOG and LBP features to determine whether the test image is a tire image without anomalies or a tire image containing anomalies (see Liu reference). Figure 1 ). This application only involves classification and not deep learning or detection.

[0006] In another application, a tire inspection system is disclosed, wherein a data processing method is performed on tire sidewall images obtained from a camera and a laser sensor (see Kuric I. et al., "Analysis of the Possibilities of Tire-Defect Inspection Based on Unsupervised Learning and Deep Learning"). Learning),” Sensors, https: / / doi.org / 10.3390 / s21217073 (October 25, 2021) (“Kuric Reference”). The captured data includes visual and geometric data characterizing the surface of the imaged tire, thereby providing a realistic representation of its sidewall. The inspection system performs an unfolding process (or “polar coordinate transformation”) to continue processing the data obtained by the camera. The principles and automation of polar coordinate transformation based on the concept of polynomial regression (i.e., supervised learning) are proposed. Based on the data from the laser sensor, anomaly detection is performed using an unsupervised clustering method, and then anomaly classification is performed using a VGG-16 neural network (see Figure 23 of Kuric Reference). The inspection system aims to detect both formed and unformed anomalies (i.e., anomalies), rather than using only supervised learning methods. Using compatible data to infer deep anomalies cannot detect anomalies on the imaged tire surface.

[0007] Another application proposes a method of functionally combining a convolutional neural network (or CNN) and an SVM to classify anomalies on the inner surface of a motor vehicle tire (see Tada, H. and Sugiura, A., “Defect Classification on Automobile Tire Inner Surfaces with Functional Classifiers,” Transactions of the Institute of Systems, Control and Information Engineers, Vol. 34, No. 1, pp. 1-10 (2021) (“Tada reference”). In this method, dividing an image into multiple smaller regions increases the number of images, thereby limiting the applicable machine learning methods. Therefore, a CNN is applied to the segmented image of the entire tire, while an SVM is applied to the segmented image within the range defined by the CNN (see FIG. 9 of Tada reference). The reliance on processing the image block by block does not involve any detection network, but only classification means. Focusing only on the inner tread cannot detect anomalies on the surface of the imaged tire.

[0008] In the field of industrial vision, existing technologies have inspired other applications beyond the detection and classification of tire anomalies during production. For example, a proposed system adopts an approach based on photometry and stereo vision that combines photometric stereo and anomaly detection to eliminate highlight and shadow interference on metal surfaces (see Cao, Y. et al., "Photometric-Stereo-Based Defect Detection System for Metal Parts," Sensors, https: / / doi.org / 10.3390 / s22218374 (November 1, 2022) ("Cao Reference"). On this basis, the system takes images captured using multiple directional lights at the input and uses a photometric stereo model to obtain a normal map. The detection model then uses the normal map to locate and classify anomalies (see Figure 8 of Cao Reference). This approach does not involve post-processing using metadata or the concept of severity levels of detected anomalies.

[0009] It has been found that these techniques require large amounts of annotated data to fully exploit their effectiveness. Therefore, their initial implementation cost is non-negligible. Therefore, it is useful to make the deep network "aware" of the quality of its own predictions.

[0010] Among the known deep learning techniques, there are recurrent neural networks (RNNs) that are capable of processing input sequences of variable length. RNN is an artificial neural network with recurrent connections that processes sequences of variable length by having a recurrent hidden state whose activation at any moment depends on the activation at the previous moment (see Chung, J. et al., “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”, NIPS 2014 Deep Learning and Representation Learning Workshop, https: / / doi.org / 10.48550 / arXiv.1412.3555 (December 11, 2014) (“Chung reference”). Since the gradient tends to vanish, it is difficult to train RNN to capture long-term dependencies. Therefore, a gated recurrent network (or gated recurrent unit or GRU network) as a recurrent unit of RNN can be proposed to capture dependencies on various time scales on an adaptive basis (see Chung reference) (cited in Cho, K, Bahdanau, D. and Bengio Y., “Neural machine translation by jointly learning to align and translate”). align and translate),” arXiv:1409.0473(2014)(“Cho reference”).

[0011] The prior art proposes a Mask Notation R-CNN (or Mask R-CNN) that includes a deep learning-based instance segmentation technique. Mask R-CNN is an architecture developed for the task of segmenting instances. The method efficiently generates a bounding box and a segmentation mask for each instance in an image (see He, K. et al., "Mask R-CNN", arXiv:1703.06870) (January 24, 2018) ("He reference"). Reference Figure 2 , Figure 2 A network is shown for directly learning the intersection-over-union (IoU) between the predicted mask and the ground truth mask to describe the quality of instance segmentation (see Huang, Z. et al., "Mask Scoring R-CNN", arXiv:1903.00241v1 (March 1, 2019) ("Z. Huang reference") ( Figure 2 Corresponding to Z. Huang reference Figure 3 ). In the architecture shown, the input image is introduced into the background network to generate the region of interest (RoI) through the region proposal network (RPN), and the RoI features are generated through the RoI alignment layer (RoIAlign), which generates a fixed-size feature map for each rectangular RoI in the input feature map (see https: / / www.mathworks.com / help / vision / ref / nnet.cn.layer.roialignlayer.html). To predict the mask IoU, the predicted mask and RoI features are used as input.

[0012] The Mask R-CNN network performs pixel-wise segmentation of detected objects. Therefore, it is able to support multiple overlapping categories and objects. To take advantage of this, methods based on deep learning using Mask R-CNN are being developed in the field of industrial vision (e.g., see Huang, H. et al., “Deep learning-based instance segmentation of cracks from shield tunnel lining images”, Structure and Infrastructure Engineering, DOI: 10.1080 / 15732479.2020.1838559 (2020) (“H. Huang reference”).

[0013] To improve the field of industrial vision to benefit the manufacturing of tires, where rubber products are difficult to detect and anomalies are numerous and varied, the disclosed invention provides a detection and classification system capable of integrating images and metadata of imaged tires in the algorithm. Thus, the disclosed invention achieves performance gains, enabling the implementation of more complex rules that are close to the quality standards performed to inspect the appearance of tires during the production process. Summary of the invention

[0014] The present invention relates to a detection and classification system that performs a method for detecting and classifying anomalies based on an image of a tire surface, characterized in that the system comprises:

[0015] - one or more tire image acquisition systems for evaluating the condition of the entire surface of the imaged tire;

[0016] - at least one communication network that manages data entering the detection and classification system, wherein each communication network includes one or more communication servers, each communication server includes one or more processors operably connected to a memory, the memory being configured to store an image processing module of the processor to analyze the images obtained from the tire image acquisition system in order to determine the presence and location of each anomaly visible on the surface of the imaged tire identified, wherein each processor includes an analysis application execution module that processes the obtained images and each processor is capable of executing programmed instructions stored in the memory in order to perform the steps of the detection and classification method;

[0017] - at least one database storing images obtained from the tire image acquisition system;

[0018] - at least one database storing metadata describing various aspects of each imaged tire;

[0019] wherein the images obtained from the tire image acquisition system are composed of pixels associated with the location of parameters and anomalies, which are transmitted as captured images and stored in the memory of the detection and classification system;

[0020] Anomaly detection network that utilizes Mask R-CNN with two additional branches:

[0021] -Mask-IOU branch, which is used to predict the intersection over union (IoU) between the predicted mask and the true mask; and

[0022] -Bbox-IOU branch, which is used to predict the IoU between the predicted bounding box (Bbox) and the real bounding box (Bbox);

[0023] in:

[0024] -Pre-train the weights of Mask R-CNN;

[0025] - The neural network is then trained globally using all images acquired at a fixed ratio and scale;

[0026] - Apply zone-wise setting strategy relative to the ratio and size of the original image;

[0027] The system is adapted to the acquired tire image and anomalies are located on the surface of the tire.

[0028] In some embodiments of the detection and classification system of the present invention, the weights of Mask R-CNN are pre-trained on the COCO dataset.

[0029] In some embodiments of the detection and classification system of the present invention, the detection and classification system is configured to extract the following features:

[0030] - coordinates of the anomaly bounding box;

[0031] - height and width of the anomaly bounding box;

[0032] - class probabilities of the classification branch;

[0033] - Prediction of the utility index of the mask and the utility index of the bounding box;

[0034] - the surface area of ​​the anomaly calculated from the anomaly's mask; and

[0035] - Information about metadata of the imaged tire.

[0036] In some embodiments of the detection and classification systems of the present invention, the total loss is a weighted sum of the individual losses and is obtained by employing a mathematical model of the following type:

[0037] [Mathematical formula 1]

[0038]

[0039] in:

[0040] -L rpn cls represents the RPN classification loss, which is the cross entropy loss of anchor classification between background and object;

[0041] -L rpn reg represents the regression loss, which is the regression of anchor positioning;

[0042] -L cls represents the faster classification loss, which is the cross entropy loss for classifying bounding boxes according to their categories;

[0043] -L reg represents a faster regression loss, which is the same as the regression loss, except that the loss is applied to a sampled set of regions of interest;

[0044] -L IoU_bbox represents the mean squared error loss (or “MSE loss”) between the target IoU and the predicted IoU (i.e., between the predicted bounding box and the target bounding box); and

[0045] -LIoU_mask With L IoU_bbox Exactly the same, except the utility index is calculated per pixel within each bounding box.

[0046] In some embodiments of the detection and classification system of the present invention, the detection and classification system comprises a deep learning based severity classification network, wherein the classification network comprises:

[0047] - at least one multi-layer perceptron (MLP) for extracting features of the detected anomalies;

[0048] - Feature extraction method using Resnet18 encoder and Gated Recurrent Unit (GRU); and

[0049] -A network that uses MLP and attention mechanism to fuse all extracted features.

[0050] In some embodiments of the detection and classification system of the present invention, the detection and classification system includes a Resnet18 encoder, the Resnet18 encoder being trained for each imaged tire region;

[0051] The training is performed to classify anomalies according to their severity.

[0052] In some embodiments of the detection and classification system of the present invention, the memory determines one or more parameters of each imaged tire by executing instructions of the image processing module, thereby performing an anomaly detection and classification method comprising the following steps:

[0053] - a step of acquiring data corresponding to the imaged tire;

[0054] - a step of training an object detection network, in which the detection neural network is trained on the images obtained in the previous acquisition step, and the object detection network predicts for each image a box containing a potential abnormal area, as well as an associated abnormality type and an associated confidence level;

[0055] - a step comprising a combination of a step of transforming the output of the object detection network and a step of metadata integration, in which the output of the object detection network is formatted to be compatible with a binary classification machine learning algorithm before the output of the first network is used to train the second network, and metadata is also integrated; and

[0056] - A step of training a second network to predict the severity of the detected anomalies.

[0057] In some embodiments of the detection and classification systems of the present invention, the one or more tire image acquisition systems provide photometric stereo images.

[0058] In some embodiments of the detection and classification system of the present invention, the detection and classification system further includes a detection system, which is used to collect information about the physical environment around the detection and classification system, and the detection system includes an illumination source having one or more illuminations, and the illumination of the illumination source is encoded in the detection system and / or pre-encoded during the neural network training process.

[0059] In addition, according to the present invention, the detection and classification system may further include:

[0060] - an enriched database that lists, for each tire, all detected anomalies and the tire’s overall severity rating,

[0061] - Means for implementing a third algorithm making use of a rich database, said third algorithm enabling the assignment of an overall severity level to the tyre based on the acquisition of an image of the tyre.

[0062] The present invention also relates to a tire production site comprising the disclosed detection and sorting system.

[0063] Other aspects of the invention will become apparent from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The nature and various advantages of the present invention will become more apparent from the following detailed description read in conjunction with the accompanying drawings, in which like reference numerals represent like parts, and in which:

[0065] Figure 1 A prior art system is shown performing a method of acquiring a visual image of a tire surface for each region of the tire for which an image is to be acquired.

[0066] Figure 2 Shown is the overall architecture of the prior art Mask R-CNN network.

[0067] Figure 3 The overall architecture of the detection and classification system of the present invention is shown.

[0068] Figure 4 shows the Figure 3 The detailed architecture of the model that models the severity of the detected anomalies in the detection and classification methods performed by the system.

[0069] Figure 5 A portion of the training process for a zone-wise setting strategy for a tire involving imaging is shown.

[0070] Figure 6Deep learning based classification using a personalized network is shown. DETAILED DESCRIPTION

[0071] Referring now to the drawings, wherein like reference numerals identify like elements, Figure 3 The overall architecture of a detection and classification system (or "system") 100 is shown. The system 100 performs a method for detecting and classifying anomalies on the surface of an imaged tire (or "detection and classification method" or "method"). The system 100 performs the method via a learning process that employs two networks in succession that represent two deep learning architectures: a first object detection network and a second object detection network that utilizes the output of the first network, reprocesses the output, and then combines it with metadata describing various aspects of the imaged tire (e.g., the size of the tire, one or more regions of the imaged tire, etc.) to predict the severity of each anomaly detected by the first network. The metadata representing the profile of the imaged tire may indicate that the file provides images that visually indicate the presence of a particular anomaly (as well as the absence of anomalies on the surface of the imaged tire). This structure makes the system 100 faster and more robust because it knows how to adapt to all sizes and types of tires it sees. It should be understood that the system 100 may form part of a site where tire production occurs.

[0072] The system 100 includes at least one communication network (or "network") that manages data entering the system from various sources. The communication network includes one or more communication servers (or "servers"), each of which includes one or more processors operably connected to a memory (as used herein, the term "server" includes one or more servers, and the term "processor" includes one or more processors). The memory is configured to store an image processing module of the processor to analyze the image of the imaged tire obtained to determine the presence and location of each anomaly visible on the surface of the imaged tire identified (the anomaly is fully or partially visible). The one or more processors include an analysis application execution module that processes the image, and the one or more processors of the analysis application execution module are capable of executing programming instructions stored in the memory to perform the steps of the detection and classification method (described below).

[0073] The term "processor" (or, alternatively, the term "programmable logic circuit") refers to one or more devices that are capable of processing and analyzing data and include one or more software for processing the data (e.g., one or more integrated circuits included in a computer, one or more controllers, one or more microcontrollers, one or more microcomputers, one or more programmable logic controllers (PLCs), one or more application specific integrated circuits, one or more neural networks, and / or one or more other known equivalent programmable circuits known to those skilled in the art). The processor includes one or more software for processing data entering the system 100, as well as one or more software for identifying and locating differences and identifying the source of the differences so as to correct them.

[0074] The memory may include volatile and non-volatile memory devices. Non-volatile memory may include solid-state memory such as NAND flash memory, "keep alive" memory (or KAM) for saving various operating variables when the processor is turned off, magnetic and optical storage media, or any other suitable data storage device that retains data when the system 100 is disabled or loses its power supply. Volatile memory may include static and dynamic RAM that stores program instructions and data (including learning applications).

[0075] The processor manages data entering the system 100 from at least one database 110 of the system 100 storing images of tires (as used herein, the term "database" refers to one or more databases). The images stored in the database 110 are obtained from at least one tire image acquisition system forming part of the system 100. These images are obtained in the step of acquiring images of tires (and therefore corresponding data) performed by the system 100 in the detection and classification method.

[0076] In some embodiments of the system 100, the tire image acquisition system comprises an acquisition system for evaluating the state of the entire surface of the tire by acquiring visual images (e.g., a system of the type disclosed in patent FR3103555 of the applicant). In the image acquisition step, one or more tire image acquisition systems capture one or more images of the entire surface of the identified tire. In one embodiment of the system 100, the acquisition system provides photometric stereo images, which enable the image of the imaged tire to be reconstructed in three dimensions from a single image capture. In order to reconstruct the image, the smallest details of the photometric stereo images (including anomalies present on the imaged tire surface) are identified. Acquiring photometric stereo images enables the collection of photometric data on the surface of the imaged tire and the application of anomaly detection methods based on photometric stereo (see Cao reference).

[0077] One or more sensors of the acquisition system of system 100 detect the presence of a tire in the field of view of the camera, thereby triggering the capture of an image of the surface of the tire. The area (or areas) of the tire imaged can be defined prior to image acquisition. In some embodiments of system 100, the sensor is triggered when the tire enters the field of view of the background of the captured image. The captured image is composed of pixels related to the parameters and positioning of the tire. These images are transmitted and stored in database 110 as captured images.

[0078] The acquisition system of system 100 may include an illumination source (e.g., known programmable LEDs) having one or more illuminations to serve as a light source for the imaged tire. The illuminations may be encoded in the acquisition system or may be pre-encoded during the neural network training process (e.g., using one or more computer programs that include data representing the illuminations and images of the identified tire). Changes in the illumination source captured in the images acquired by the acquisition system (e.g., the acquired images may include indications of reflections due to changes in illumination) are depicted by pixels having different color intensities. Thus, the images captured by the acquisition system show the surface of the imaged tire and anomalies therein, if any. The camera (and / or sensor) and the illumination source may be moved in an alternating or random manner to adjust the objective lens and the illumination, respectively, according to the parameters of the tire located in the capture area of ​​the acquisition system.

[0079] In some embodiments of the detection and classification method performed by the system 100, the processor can configure the detection and classification system 100 (specifically, the tire manufacturing site including the system 100) according to one or more parameters of one or more imaged tires. In these embodiments, it will be understood that one or more reinforcement learning approaches (including deep reinforcement learning) can be employed. The processor can also refer to reference information (e.g., a tire parameter table) to perform a final determination of one or more parameters of the imaged tire. The reference information may include known tire parameters corresponding to a plurality of commercially available tires. For example, after the data processing module has calculated one or more parameters of the identified imaged tire, the processor can compare the detected parameters with the known parameters stored in the reference information.

[0080] The system 100 also includes a database 112 that stores metadata describing various aspects of the imaged tires. The metadata specifies information associated with the corresponding imaged tire and may include some or all of the general information about such tire. The term "general information" as used herein is used to specify data corresponding to the identified imaged tire. Such data may include, but is not limited to, tire size (which may be indicated by the type of tire and / or its nomenclature), tire construction code (e.g., "R" indicates radial construction), tire production source (e.g., the name and / or brand of the tire manufacturer, the date and place of tire manufacturing, distribution and / or storage), tire unique identification number (or "serial number"), tire load index, tire speed symbol (e.g., "A5", which indicates 25 km / h) and / or tire scheduled mileage. For example, for a tire of size 385 / 65R225, the number "385" indicates the nominal cross-sectional width of the tire in millimeters, the number "65" indicates the aspect ratio of the tire, the letter "R" indicates a radial tire, and the number "225" indicates the rim diameter in inches.

[0081] Reference again Figure 3 And also refer to Figure 4 , the data entering the system 100 includes quality annotations that enable the location and classification of anomalies. The annotations may be defined based on pre-established quality standards (e.g., quality standards standardized by tire manufacturers during production and / or quality standards that comply with government standards). As an example, anomaly categories may be defined in the following categories:

[0082] - bump;

[0083] - surface roughness;

[0084] - tread mark;

[0085] - cut;

[0086] - contrast differences;

[0087] - tread erosion;

[0088] - a chip; and

[0089] -smooth teeth.

[0090] It will be appreciated that, depending on the area of ​​the tire's surface that is imaged, there will be more or less anomalies, sometimes with visual differences (that is, the "within class" differences for the same camera area are small, but the "within class" differences between different areas may be higher). For example, one surface roughness on the inside of the tread will be very different from another surface roughness on the outside of the tread.

[0091] It will also be appreciated that the annotations may include information about the severity level associated with each anomaly category. In one embodiment, the system 100 is configured with two severity levels, "critical" or "non-critical," thereby enabling rebalancing of severity categories.

[0092] Reference again Figure 3 and Figure 4 , the system 100 enables detection and segmentation of anomalies relative to their visual system. Due to the large number of anomaly types with the same severity, it is not possible to detect severity directly. Therefore, the system 100 includes a dedicated network for predicting severity.

[0093] For anomaly detection, the architecture of the anomaly detection network shows novelty and creativity by using Mask R-CNN with two additional branches:

[0094] -Mask-IOU branch, which is used to predict the intersection-over-union (IoU) between the predicted mask and its true mask (the architecture of the IoU branch is taken from H. Huang reference). Its purpose is to describe the quality of instance segmentation and eliminate false positives, not only based on the classification score, but also based on the predicted IoU; and

[0095] -Bbox-IOU branch, which is used to predict the IoU between the predicted bounding box and its true bounding box (see Wang, J. et al., “Semi-Supervised Active Learning for Instance Segmentation via Scoring Predictions”, https: / / arxiv.org / pdf / 2012.04829(2020) (“Wang reference”).

[0096] After detection, the system 100 must extract features of each detected anomaly to classify it according to its severity. Typically, for images, anomaly regions have unique features represented by varying intensity values ​​relative to the background (e.g., oil stains are typically darker in color, which is represented by lower intensity values ​​corresponding to the annotation "contrast difference"). The deep learning function and the gradient and gray value features on the anomaly edges motivate the use of deep learning-based Mask R-CNN to segment each anomaly based on the background of the tire image.

[0097] In one embodiment, the system 100 is configured to extract the following features:

[0098] - The coordinates of the bounding box of the anomaly (e.g. the center of the bounding box), since the severity depends on its location;

[0099] - height and width of the bounding box of the anomaly, in order to have a height / width ratio feature;

[0100] - class probabilities of the classification branch;

[0101] - Prediction of utility index of mask and prediction of utility index of bounding box;

[0102] - the surface area of ​​the anomaly calculated from its mask; and

[0103] - Information about metadata (including general information about the imaged tire).

[0104] Reference again Figure 3 and Figure 4 as well as Figure 5 , a definition of the training process is given, where the weights of Mask R-CNN are pre-trained on the COCO dataset, which is known to those skilled in the art (see https: / / cocodataset.org / #home). The neural network is then trained globally using all images obtained at a fixed ratio and scale (1024*1024) (see Figure 4 ). This initial training enables the network to adapt to photometric stereo images and localize anomalies. Afterwards, a zone-wise setting strategy is applied with respect to the ratio and size of the original image. The maximum possible resolution is maintained for the memory provided by the graphics processing unit (GPU) for training.

[0105] The system 100 uses a region proposal network (RPN) that proposes bounding boxes for candidate objects (i.e., candidate anomalies detected on the surface of the imaged tire). The system 100 extracts features from each candidate box and performs classification and regression of the bounding boxes. In some embodiments of the system 100, the system uses region of interest pooling (or RoI pooling or RoIPool) to extract features, which is a well-known method for solving the still image size requirements of object detection networks (see He reference). The features used by these two extractions can be shared for faster reasoning.

[0106] For training a network, the total loss is a weighted sum of the individual losses. The total loss can be thought of as the sum of the previous losses, obtained by adopting the following type of mathematical model:

[0107] [Mathematical formula 1]

[0108]

[0109] in:

[0110] -L rpn cls represents the RPN classification loss, which is the cross entropy loss of anchor classification between background and object;

[0111] -L rpn reg represents the regression loss, which is the regression of anchor point positioning;

[0112] -L cls represents the faster classification loss, which is the cross entropy loss for classifying the bounding box according to its category;

[0113] -L reg represents a faster regression loss, which is the same as the regression loss, except that the loss is applied to a sampled set of regions of interest;

[0114] -L IoU_bbox represents the mean squared error loss (or “MSE loss”) between the target IoU and the predicted IoU (i.e., between the predicted bounding box and the target bounding box); and

[0115] -L IoU_mask With L IoU_bbox Exactly the same, except the utility index is calculated per pixel within each bounding box.

[0116] Reference again Figures 3 to 5 , and also refer to Figure 6 , Figure 6 A deep learning based severity classification using a personalized network is shown. As shown, the system 100 uses a personalized multimodal deep learning neural network to merge the original image with the extracted features. There is one dataset for each imaged tire region, with one anomaly instance detected per row.

[0117] exist Figure 6 In the illustrated embodiment of the personalized network, the severity classification includes three parts:

[0118] - Multilayer Perceptron (MLP) for feature extraction from tabular data;

[0119] - a feature extraction method using a Resnet18 encoder for extracting features from small patches and a Gated Recurrent Unit (GRU) for capturing interactions between small patches; and

[0120] -Network that uses MLP and attention mechanism to fuse visual and tabular features.

[0121] The architecture allows for improved classification results by incorporating metadata. The self-attention module allows the network to exclude irrelevant information such as background. The cross-attention module makes it possible to ensure that each modality guides the other.

[0122] To extract features in each acquired image, the system 100 includes a Resnet18 encoder that is trained for each imaged tire region. For example, the training classifies minor anomalies into three (3) categories (no anomaly, no severe anomaly, severe anomaly) based on their severity. This pre-training makes it possible to ensure that the extracted features are relevant.

[0123] Next, a bidirectional LSTM (or "BiLSTM") approach is used, which effectively increases the amount of information available to the network. This BiLSTM approach uses background modeling to classify small sequences based on high-resolution images. The RNN can be used to process sequences of various lengths, so the area of ​​the anomaly bounding box remains independent of the size of the original image. It is also possible to process images at full resolution without resizing.

[0124] The memory of system 100 performs an anomaly detection and classification method (or "method") by executing instructions of the image processing module to determine one or more parameters of each imaged tire. As used herein, the term "method" or "process" may include one or more steps performed by at least one computer system including one or more processors for executing instructions for running the steps. Unless otherwise specified, any sequence of steps is given by way of example, and the described method is not limited to any particular sequence.

[0125] To start the method, it comprises a step of acquiring data corresponding to the imaged tire. In this step, one or more tire image acquisition systems (for example, systems of the type disclosed in patent FR3103555 of the applicant) capture one or more images of the entire surface of the identified tire.

[0126] In one embodiment of the system 100, the system 100 provides photometric stereo images that enable reconstruction of an image of a tire in three dimensions from a single image capture. To reconstruct the image, the smallest details of the photometric stereo images (including anomalies present on the imaged tire surface) are identified. Acquiring photometric stereo images enables collection of photometric data on the imaged tire surface and application of photometric stereo-based anomaly detection methods (see Cao reference).

[0127] The method comprises a step of training an object detection network. In this step, the detection neural network is trained on images of an irregular envelope annotated with anomaly classes (e.g., photometric stereo images obtained in a previous acquisition step), and also on clean images. In this step, the object detection network predicts for each image a box containing potential anomaly regions, as well as the associated anomaly type and its associated confidence. The work around the confidence of the predictions is done specifically using the trained model's own predictions of the confidence of the quality of its segmentation (i.e., "bounding box IoU" and "segmentation IoU") (see Figure 6 ).

[0128] The method comprises a step comprising a combination of a step of transforming the output of the object detection network and a step of metadata integration. In this step, the output of the object detection network is formatted to be compatible with a binary classification machine learning algorithm before the output of the first network is used to train the second network (as understood, the formatting is performed by means of performing classification encoding of the anomalies, extracting new features of the segmented anomalies (such as its surface area, its length / width ratio, its height, etc.)). Metadata (e.g., measurements of the dimensions of the imaged tire, the camera area involved, etc.) are also integrated into these features.

[0129] The method comprises a step of training a second network to make predictions about the severity of detected anomalies. In this step, all the features integrated in the previous step are retrieved into a new dataset constructed in such a way that a row corresponds to a detected anomaly with the new features. To define the severity of anomalies in this detection, the following rules are applied:

[0130] - If the detection has an intersection with an anomaly annotated as severe, the severity of the target will be "severe";

[0131] - If the detection has an intersection with an anomaly annotated as non-severe, or has no intersection with an anomaly (a "false positive"), the severity of the target will be "non-severe".

[0132] Therefore, a second model may be trained to predict severity based on this second data set in combination with the starting image (obtained in the acquisition step). It will be appreciated that, depending on the type of acquisition system, the model may be specialized for a camera zone or cluster of camera zones, thereby obtaining the best performance for each zone.

[0133] The system 100 of the present invention can easily repeat one or more steps of the detection and classification method in the correct order to achieve industrial vision parameters implemented at the site where the tires are produced.

[0134] The detection and classification method performed by the system 100 can be performed by controlling the PLC and can include pre-programming the production information of the tire. For example, the settings of the inspection process can be related to the parameters of the identified tire so that the production line is configured according to one or more parameters calculated by the module for processing the data obtained by the system 100 from each production line.

[0135] The settings of the detection and classification methods performed by the system 100 can be related to the parameters of the typical physical environment in which the system 100 operates. In some embodiments of the present invention, the system 100 (and / or a tire production site including the detection and classification system 100) can receive voice commands or other audio data indicating, for example, a program or stop in a production line. The request can include a request for the current state of the tire production cycle. The generated response can be represented in an auditory, visual, tactile manner (e.g., using a tactile interface) and / or in a virtual and / or augmented manner. The response can be recorded in one or more neural networks in association with the corresponding data.

[0136] For all embodiments of system 100, a monitoring system can be implemented. At least a portion of the monitoring system can be provided in a portable device, such as a mobile network device (e.g., a mobile phone, a laptop computer, one or more portable devices connected to a network (including "augmented reality" and / or "virtual reality" devices), a wearable garment connected to a network, and / or any combination thereof and / or any equivalent). It is contemplated that the detection and comparison steps can be performed iteratively.

[0137] The terms "at least one" and "one or more" can be used interchangeably. A range presented as "between a and b" includes the values ​​"a" and "b".

[0138] Although specific embodiments of the disclosed system have been shown and described, it will be appreciated that various changes, additions and modifications may be made without departing from the spirit or scope of the invention. Therefore, no limitation should be imposed on the scope of the described invention, except as disclosed in the appended claims.

[0139] refer to Figure 3 The described anomaly detection steps enable:

[0140] - Acquire images of the tire and locate anomalies,

[0141] - For each detected anomaly, extract the characteristics of the identified anomaly (e.g., the surface area of ​​the anomaly, the location of the anomaly on the tire, the tire's logo, etc.)

[0142] - Assign a severity level to the anomalies, classified into at least two categories: first category: no repair, second category: repair.

[0143] An enriched database is built after the anomaly detection step, which lists for each tire all anomalies detected and the overall severity rating of the tire.

[0144] For example, an enriched database may be generated based on some or all of the results of the anomaly detection step. Such an enriched database has the advantage of giving an overall severity rating for the entire tire, which constitutes more useful information and a more useful approach than giving severity ratings for each anomaly.

[0145] A third algorithm is constructed using the enriched database, thereby generating a third algorithm that enables the assignment of an overall severity level to a tire based on the acquisition of an image of the tire.

[0146] The third algorithm implements an artificial intelligence algorithm based on multiple instance learning (referred to as a multiple instance learning algorithm). This algorithm takes into account a variable number of anomalies from one tire to another while still being able to assign a severity level to the tire. The severity level is determined using features for each anomaly detected for the tire, and the number of anomalies is variable from one tire to another.

[0147] Then, the third algorithm is implemented as follows.

[0148] Similar to reference Figure 3 Another anomaly detection step is performed in the described manner:

[0149] - for each tire, obtain an image of the tire and locate anomalies,

[0150] - For each tire, a set of detected anomalies is determined, and features of the determined anomalies are extracted (eg, surface area of ​​the anomalies, location of the anomalies on the tire, identification of the tire, etc.).

[0151] Next, for the tire, the features of the detected anomalies are provided to a third algorithm based on multiple instance learning to assign a severity level. The severity level may be associated with a classification into at least two of the following categories: a first category: no repair, a second category: repair.

Claims

1. A detection and classification system (100) for performing a method for detecting and classifying anomalies based on an image of a tire surface, characterized in that The system comprises: - one or more tire image acquisition systems for evaluating the condition of the entire surface of the imaged tire; - at least one communication network that manages data entering the detection and classification system (100), wherein each communication network includes one or more communication servers, each communication server includes one or more processors operably connected to a memory, the memory being configured to store an image processing module of the processor to analyze the images obtained from the tire image acquisition system in order to determine the presence and location of each anomaly visible on the surface of the imaged tire identified, wherein each processor includes an analysis application execution module that processes the obtained images and each processor is capable of executing programmed instructions stored in the memory in order to perform the steps of the detection and classification method; - at least one database (110) storing images obtained from the tire image acquisition system; - at least one database (112) storing metadata describing various aspects of each imaged tire; wherein the images obtained from the tire image acquisition system are composed of pixels associated with the location of parameters and anomalies, and these images are transmitted as captured images and stored in the memory of the detection and classification system (100); Anomaly detection network that utilizes Mask R-CNN with two additional branches: -Mask-IOU branch, which is used to predict the intersection-over-union ratio between the predicted mask and the true mask; and -Bbox-IOU branch, which is used to predict the intersection-over-union ratio between the predicted bounding box and the true bounding box; in: -Pre-train the weights of Mask R-CNN; - The neural network is then trained globally using all images acquired at a fixed ratio and scale; - Apply partition setting strategy relative to the ratio and size of the original image; The system (100) is adapted to obtain an image of the tire and locate anomalies on the surface of the tire.

2. The detection and classification system (100) according to claim 1, wherein: The weights of Mask R-CNN are pre-trained on the COCO dataset.

3. The detection and classification system (100) according to claim 1 or claim 2, wherein: The detection and classification system (100) is configured to extract the following features: - coordinates of the anomaly bounding box; - height and width of the anomaly bounding box; - class probabilities of the classification branch; - Prediction of utility index of mask and prediction of utility index of bounding box; - the surface area of ​​the anomaly calculated from the anomaly's mask; as well as - Information about metadata of the imaged tire.

4. The detection and classification system (100) according to claim 3, wherein: To train the network, the total loss is a weighted sum of the individual losses and is obtained by adopting the following type of mathematical model: [Mathematical formula 1] in: -L rpncls represents the RPN classification loss, which is the cross entropy loss of anchor classification between background and object; -L rpnreg represents the regression loss, which is the regression of anchor positioning; -L cls represents the faster classification loss, which is the cross entropy loss for classifying bounding boxes according to their categories; -L reg represents a faster regression loss, which is the same as the regression loss, except that the loss is applied to a sampled set of regions of interest; -L IoU_bbox represents the mean square error loss between the target intersection-over-Union ratio and the predicted intersection-over-Union ratio, that is, between the predicted bounding box and the target bounding box; and -L IoU_mask With L IoU_bbox Exactly the same, except the utility index is calculated per pixel within each bounding box.

5. The detection and classification system (100) according to claim 4, wherein: The detection and classification system (100) comprises a deep learning based severity classification network, wherein the classification network comprises: - at least one multi-layer perceptron for extracting features of detected anomalies; - Feature extraction method using Resnet18 encoder and gated recurrent units; and -A network that uses multi-layer perceptrons and attention mechanisms to fuse all extracted features.

6. The detection and classification system (100) according to claim 5, wherein: The detection and classification system (100) includes a Resnet18 encoder, which is trained for each imaged tire region; The training is performed to classify anomalies according to their severity.

7. The detection and classification system (100) according to claim 6, wherein: The memory determines one or more parameters of each imaged tire by executing instructions of the image processing module, thereby performing an anomaly detection and classification method comprising the following steps: - a step of acquiring data corresponding to the imaged tire; - a step of training an object detection network, in which the detection neural network is trained on the images obtained in the previous acquisition step, and the object detection network predicts for each image a box containing a potential abnormal area, as well as an associated abnormality type and an associated confidence level; - a step comprising a combination of a step of transforming the output of the object detection network and a step of metadata integration, in which the output of the object detection network is formatted to be compatible with a binary classification machine learning algorithm before the output of the first network is used to train the second network, and metadata is also integrated; as well as - A step of training a second network to predict the severity of the detected anomalies.

8. The detection and classification system (100) according to any one of claims 1 to 7, wherein: The one or more tire image acquisition systems provide photometric stereo images.

9. The detection and classification system (100) according to any one of claims 1 to 8, wherein: The detection and classification system (100) further comprises a detection system for collecting information about the physical environment surrounding the detection and classification system (100), the detection system comprising an illumination source having one or more illuminations, the illumination of the illumination source being encoded in the detection system and / or pre-encoded during the neural network training process.

10. The detection and classification system (100) according to one of claims 1 to 9, further comprising: - A rich database that lists, for each tire, all detected anomalies and the tire’s overall severity rating, - Means for implementing a third algorithm making use of a rich database, said third algorithm enabling the assignment of an overall severity level to the tyre based on the acquisition of an image of the tyre.

11. A tire production site comprising a detection and classification system (100) according to any one of claims 1 to 10.

Images