System for detecting and classifying anomalies in rubber products

Abstract

The invention relates to a detection and classification system (100) which performs a method of detecting and classifying anomalies from images of tire surfaces. The invention also relates to a tire production site comprising the disclosed system (100). Figure for abstract: Fig. 3

Description

Title of the invention: System for detecting and classifying anomalies in rubber products Technical field

[0001] The invention relates to a detection and classification system based on industrial vision for detecting rubber products such as tires. The disclosed detection and classification system makes it possible to make a link by data between an imposed theoretical quality standard and the reality on the ground. Context

[0002] In the field of industrial processes, the surfaces of tires intended to equip rolling vehicles are difficult to control. These tires generally have a black color due to the black rubber material that constitutes them (due to the use of carbon to reinforce the elastomeric compounds). Thus, the automated detection of tire surface anomalies is very difficult to achieve.Furthermore, the form of an anomaly may be related to its severity with the different anomalies varying according to distinct anomaly classes which include, without limitation, various stains (including grease stains), dents, scratches, surface roughness, indentations, cuts, contrast differences, tread erosions, chips, smooth teeth, material shortages, material excesses, black shade differences, cracks, etched materials and localized discolorations. As used herein, the term "anomaly" refers to a detected difference (represented, for example, by a deviation from a standard value) without other information (thus, an "anomaly" is not automatically interpreted as a "defect").

[0003] There are solutions in the prior art to overcome this challenge. For example, the Applicant's patent FR3103555 discloses a system for assessing the condition of a tire by acquiring a visual image. Referring to [Fig.l] (which corresponds to [Fig.5] of patent FR3103555), the disclosed system acquires images of a tire 10 having external areas (including the sidewalls 12, the shoulders 14 and the tread (or "crown") 16) and internal areas (including the internal sides 18 and the inner crown 20) using both two-dimensional and three-dimensional cameras (or "sensors"). The two-dimensional sensors are used for both the interior and the exterior (in embodiments incorporating the three-dimensional sensors, these sensors are used only for the external surfaces of the tire 10).During a process performed by the disclosed system, the tire 10 enters the system where . it is locked and rotated so that the various sensors can take images of the tire (for example, by quickly lowering the sensor to near the area where it should be located, then switching to slow speed to precisely adjust the position).

[0004] There are also recent developments that employ artificial intelligence (or "AI") methods, and in particular deep learning (or "deep learning") to interpret the images acquired from a tire (including the acquisition means used in patent FR3103555). The recent improvement in deep learning techniques and data analysis, combined with computing and data storage platforms, has opened up avenues for the development of new approaches in the field of machine vision. Its main objective is to use the "eyes of the machine" to recognize anomalies in images captured by image capture devices (e.g., one or more sensors and / or cameras), and to input these captured images into visual algorithms for calculation.

[0005] In the field of machine vision, the prior art proposes some applications for the detection and classification of anomalies in tires during production. Such an application proposes a combination of the histogram of oriented gradients (HOG) and the characteristics of local binary patterns (LBP) in a method for detecting tire appearance anomalies (see Liu, Hongbin, et al., “Tire appearance defect detection method via combining HOG and LBP features,” Front. Phys., DOI: 10.3389 / fphy.2022.1099261 (January 12, 2023)(“the Liu reference”). In this method, a tire image dataset is constructed to provide tire images with and without anomalies.Then, the HOG and LBP features of the tire images are respectively extracted and analyzed to train the support vector machine (SVM). Finally, the SVM classifier calculates the prediction scores of the test images by combining the HOG and LBP features to determine whether the test image is a tire image without anomaly or a tire image with anomaly (see [Fig.l] in Reference Liu). This application is only concerned with classification and not with deep learning or detection.

[0006] In another application, a tire inspection system is disclosed whose data processing methods are performed on the images obtained from the tire sidewalls from a camera and a laser sensor (see Kuric I., et al., “Analysis of the Possibilities of Tire-Defect Inspection Based on Unsupervised Learning and Deep Learning”, Sensors, https: / / doi.org / 10.3390 / s21217073 (October 25 202l)(“the Kuric reference”). The captured data includes visual and geometric data characterizing the surface of the imaged tire, providing a real representation of its sidewall. An unfolding process (or “polar transformation”) is performed by the inspection system to further process the data obtained by the camera. The principles and automation of the designed polar transformation, based on polynomial regression (i.e., supervised learning), are presented. Based on the laser sensor data, anomaly detection is performed using an unsupervised clustering method, followed by anomaly classification using the VGG-16 neural network (see Figure 23 of the Kuric reference). The inspection system aims to detect both trained and untrained anomalies, i.e., anomalies, rather than using only supervised learning methods.Using conformal data to deduce depth anomalies does not allow the detection of surface anomalies of the imaged tire.

[0007] Another application proposes a method that functionally combines a convolutional neural network (or “CNN”) and an SVM to classify anomalies on the inner surface of an automobile tire (see Tada, H. and Sugiura, A., “Defect Classification on Automobile Tire Inner Surfaces with Functional Classifiers,” Transactions of the Institute of Systems, Control and Information Engineers, Vol. 34, No. 1, pp. 1-10 (2021)(“the Tada reference”)). In this method, dividing an image into multiple images of smaller regions increases the number of images, which limits the applicable machine learning methods. Thus, the CNN is applied to the divided images of the entire tire, while the SVM is applied to the divided images within the range delimited by the CNN (see Figure 9 of the Tada reference).The reliance on patch-by-patch image processing does not involve any detection network but only a means of classification. Focusing only on the inner tread does not allow for the detection of surface anomalies of the imaged tire.

[0008] In the field of machine vision, the prior art inspires other applications outside of the detection and classification of anomalies in tires during production. For example, a proposed system employs a photometric and stereoscopic based method that combines photometric stereo and anomaly detection to eliminate the interference of highlights and shadows on metal surfaces (see Cao, Y. et al., “Photometric-Stereo-Based Defect Detection System for Metal Parts,” Sensors, https: / / doi.org / 10.3390 / s22218374 (1 November 2022)(“the Cao reference”). Based on this, the system takes as input images captured with multiple directional lights and obtains the map of normals through the photometric stereo model. The detection model then uses the normal map to locate and classify anomalies (see Figure 8 of the CAD reference). This approach does not involve post-processing with metadata or the notion of severity level of a detected anomaly.

[0009] It is noted that these techniques require large amounts of annotated data to reveal their full effectiveness. As a result, their initial implementation cost is not negligible. It is therefore useful to allow a deep network to be "aware" of the quality of its own predictions.

[0010] Among the known deep learning techniques, there are recurrent neural networks (or "recurrent neural networks" or "RNNs") that are capable of processing a variable-length input sequence. The RNN, an artificial neural network with recurrent connections, processes the variable-length sequence by having a recurrent hidden state whose activation at each instant depends on that of the previous instant (see Chung, J. et al, "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling", NIPS 2014 Deep Learning and Representation Learning Workshop, https: / / doi.org / 10.48550 / arXiv.l 412.3555 (December 11, 2014) ("the Chung reference"). It is difficult to train RNNs to capture long-term dependencies because gradients tend to vanish.Thus, the gated recurrent network (or a "gated recurrent unit" or "GRU"), being a recurrent unit of the RNN, could be proposed to capture dependencies at different time scales in an adaptive manner (see Chung reference)(citing Cho, K, Bahdanau, D. and Bengio Y., "Neural .machine translation by jointly leaming to align and translate", arXiv:1409.0473 (2014)("Cho reference").

[0011] The prior art proposes a mask scoring R-CNN (or “Mask R-CNN”) that includes a deep learning instance segmentation technique. A Mask R-CNN is an architecture developed to address the task of instance segmentation. This method efficiently generates a bounding box and a segmentation mask for each instance in an image”)(see He, K., et al., “Mask R-CNN,” arXiv: 1703.06870)(January 24, 2018)(“the He reference”). Referring to [Fig.2], [Fig.2] represents a network to directly learn the Intersection-over-Union (or “intersection-over-union” or “loU”) between the predicted mask and the terrain mask to describe the quality of instance segmentation (see Huang, Z., et al., “Mask Scoring R-CNN”, arXiv: 1903.00241vl (1 Tuesday 2019)(“reference Z. Huang”))([Fig.2] corresponds to [Fig.3] of reference Z. Huang).In the represented architecture, the input image is fed into a background network to generate regions of interest (or "Roi") via a region proposal network (or "Region Proposal Network" or "RPN") and Roi features. via an ROI alignment layer that produces fixed-size feature maps for each rectangular ROI in an input feature map ("RoIAlign") (see https: / / www.mathworks.com / help / vision / ref / nnet.cnn.layer.roialignlayer.html). To predict the MaskloU, the predicted mask and the Roi feature are used as the input.

[0012] The Mask R-CNN network performs pixel-level segmentation on the detected objects. It can therefore support multiple classes and overlapping objects. To take advantage of this benefit, methods based on deep learning using a Mask R-CNN are being developed in the field of machine vision (e.g., see Huang, H., et al., “Deep learning-based instance segmentation of cracks from shield tunnel lining images,” Structure and Infrastructure Engineering, D01:10.1080 / 15732479.2020.1838559 (2020)(“reference H. Huang”).

[0013] In order to improve the field of industrial vision for the benefit of tire manufacturing (where rubber products are difficult to detect and anomalies are varied), the disclosed invention provides a detection and classification system allowing the integration of images and metadata of the imaged tires into an algorithm. The disclosed invention therefore achieves a performance gain allowing the implementation of more complex rules approaching the quality standards imposed for the appearance control of tires during production. Summary of the invention

[0014] The invention relates to a detection and classification system which carries out a method of detecting and classifying anomalies from images of tire surfaces, characterized in that the system comprises: - one or more tire image acquisition systems enabling the evaluation of the condition of the entire surface of an imaged tire; - at least one communication network that manages incoming data to the system, wherein each communication network comprises one or more communication servers each comprising one or more processors operatively connected to a memory configured to store an image processing module of the processor, which analyzes the images obtained from the image acquisition systems to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire, wherein each processor comprises an analysis application execution module that performs the processing of the obtained images and each processor is capable of executing programmed instructions stored in memory to carry out the steps of the detection and classification process; - at least one database in which the images obtained from the tire image acquisition systems are stored; - at least one database in which metadata is stored that describes aspects of each imaged tire; wherein the images obtained from the tire image acquisition systems are formed of pixels associated with the parameters and positioning of the anomalies, these images being transferred and stored as captured images in the memory of the inspection system; - an anomaly detection network that uses a Mask R-CNN with two additional branches including: - a Mask-IOU branch to predict the Intersection-over-Union (loU) between a predicted mask and its ground truth mask; and - a Bbox-IOU branch to predict the IoU between a predicted bounding box (Bbox) and its surrounding box (Bbox); in which: - the Mask R-CNN weights are pre-trained; - a neural network is then trained globally with all the images obtained at a fixed ratio and scale; and - a zone adjustment strategy is applied with respect to the ratio and size of the original images; so that the system adapts to the images obtained from the tires and locates anomalies on their surfaces.

[0015] In some embodiments of the inspection system of the invention, the weights of the Mask R-CNN are pre-trained on the COCO dataset.

[0016] In some embodiments of the inspection system of the invention, the system is configured to extract the following features: - Coordinates of the anomaly bounding box; - The height and width of the anomaly bounding box; - The class probability of the classification branch; - Predictions of the mask utility index and the bounding box utility index; - The anomaly surface calculated on the anomaly mask; and - Information on the metadata of the imaged tire.

[0017] In certain embodiments of the inspection system of the invention, the total loss is a weighting of the individual losses and is obtained by employing a mathematical model of the type: [Math 1] ^Tota!= + K ' ^rpn„.,+ Xj ' Lds + X4 • Lreg + X5 • + X6 • LjoU^ Or : - L jpn ck represents the RPN classification loss which is a cross-entropy loss for anchor classification between background and objects; - Lrpnreg represents the regression loss which is the regression of the anchor location; - Lck represents the faster classification loss which is a cross-entropy loss for classifying bounding boxes based on their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied on a sampled set of regions of interest; - LloUbbox represents the mean square error loss (or "MSE loss") between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and - LloU mask is exactly the same as LloUbbox except that the utility index is calculated per pixel inside each bounding box.

[0018] In some embodiments of the inspection system of the invention, the system comprises a deep learning-based severity classification network, wherein the classification network comprises: - at least one multi-layer perceptron (MLP) to extract the characteristics of the detected anomalies; - a feature extraction method using a Resnetl8 encoder and a gated recurrent unit (GRU); and - a network using an MLP and an attention mechanism to fuse all the extracted features.

[0019] In certain embodiments of the inspection system of the invention, the system incorporates a Resnetl8 encoder driven by imaged tire area; such that this training performs a classification of anomalies according to their severity.

[0020] In certain embodiments of the inspection system of the invention, the memory, by executing the instructions of the image processing module in order to determine one or more parameters of each imaged tire, executes a method of detecting and classifying anomalies comprising the following steps: - a step of acquiring the data corresponding to the imaged tires; - a step of training a network for object detection, during which a detection neural network is trained on images obtained during the previous acquisition step, and during which the network for object detection predicts, for each image, boxes containing the potential areas of anomalies with the associated type of anomaly and its associated confidence; - a step comprising a combination of a step of transforming the outputs of the network for object detection and a step of integrating the metadata, during which, before training a second network using the outputs of the first network, the outputs of the network for object detection are formatted in order to make them compatible with a binary classification machine learning algorithm, and during which the metadata are also integrated; and - a training step of the second network to predict the severity of a detected anomaly. In certain embodiments of the inspection system of the invention, the tire image acquisition system(s) provide stereo-photometric images.

[0021] In some embodiments of the inspection system of the invention, the system further comprises a detection system for collecting information about the physical environment around the system, the detection system comprising an illumination source having one or more illuminations whose illuminations are encoded in the detection system and / or pre-coded during a training process of a neural network.

[0022] Further according to the invention, the inspection system may further comprise - an enriched database which, for each tire, lists all the anomalies detected, as well as an overall severity rating for the tire, - a device implementing a third algorithm using the enriched database, which makes it possible to assign an overall severity rating for a tire, based on an acquisition of tire images.

[0023] The invention also relates to a tire production site, comprising the disclosed detection and classification system.

[0024] Other aspects of the invention will become apparent from the following detailed description. Brief description of the drawings

[0025] The nature and various advantages of the invention will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numerals designate like parts throughout, and in which: [Fig.l] [Fig.l] represents the different areas of a tire where images are to be acquired during a process carried out by a prior art system allowing the acquisition of a visual image of the tire surfaces. [Fig.2] [Fig.2] represents an overall architecture of a prior art Mask R-CNN type network. [Fig.3] [Fig.3] represents an overall architecture of a detection and classification system of the invention. [Fig.4] [Fig.4] represents a detailed architecture of a model for predicting the severity of anomalies detected during a detection and classification process carried out by the system of [Fig.3]. [Fig.5] [Fig.5] represents part of a training process incorporating a zone-based adjustment strategy for an imaged tire. [Fig.6][Fig.6] represents a classification based on deep learning with a custom network. Detailed description

[0026] Referring now to the figures, in which the same numbers identify identical elements, [Fig. 3] represents an overall architecture of a detection and classification system (or "system") 100. The system 100 performs a method for detecting and classifying surface anomalies of imaged tires (or "detection and classification method" or "method"). The system 100 performs this method via a learning process that employs a succession of two networks representing two deep learning architectures: a first network for object detection, and a second network using the outputs of the first network reworked and then combined with metadata that describe aspects of an imaged tire (for example, the dimensions of the tire, the area(s) of the imaged tire, etc.) in order to predict the severity of each anomaly detected by the first network.The metadata of a file representing an imaged tire may indicate that the file provides images visually indicative of the presence of a particular anomaly (and also the absence of an anomaly in the surface of the imaged tire). This structure makes the system 100 faster and more robust because it can adapt to all sizes and types of tires that it sees. It is understood that the system 100 could be part of a site where tire production is carried out.

[0027] The system 100 includes at least one communication network (or "network") that manages incoming data to the system from various sources. The communication network incorporates one or more communication servers (or "servers") each comprising one or more processors operatively connected to a memory (as used herein, the term "server" incorporates one or more servers, and the term "processor" incorporates one or more processors). The memory is configured to store an image processing module of the processor, which analyzes the obtained images of the imaged tires to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire (the anomaly being visible either in whole or in part). The one or more processors include an analysis application execution module that performs the image processing, wherein the one or more processors are capable of executing programmed instructions stored in the memory to perform the steps of the detection and classification method (as described below).

[0028] The term "processor" (or, alternatively, the term "programmable logic circuit") refers to one or more devices capable of processing and analyzing data and including one or more software programs for processing them (e.g., one or more integrated circuits known to those skilled in the art as being included in a computer, one or more controllers, one or more microcontrollers, one or more microcomputers, one or more programmable logic controllers (or "PLCs"), one or more application-specific integrated circuits, one or more neural networks, and / or one or more other known equivalent programmable circuits). The processor includes one or more software programs for processing the incoming data of the system 100 as well as one or more software programs for identifying and locating variances and identifying their sources to correct them.

[0029] The memory may include both volatile and non-volatile memory devices. The non-volatile memory may include solid-state memories, such as NAND flash memory, keep-alive memory (KAM) for saving various operating variables while the processor is powered off, magnetic and optical storage media, or any other suitable data storage device that retains data when the system 100 is powered off or loses power. The volatile memory may include static and dynamic RAM that stores program instructions and data, including a learning application.

[0030] The processor manages the incoming data to the system 100 from at least one database 110 of the system 100 in which the images of the tires are stored (as used herein, the term "database" refers to one or more databases). The images stored in the database 110 are obtained from at least one tire image acquisition system that is part of the system 100. These images are obtained during a step of acquiring the images (and therefore the corresponding data) of the tires carried out by the system 100 during the detection and classification process.

[0031] In embodiments of the system 100, the tire image acquisition system comprises an acquisition system allowing the evaluation of the condition of the entire surface of a tire by acquiring a visual image (for example, a system of the type disclosed by the Applicant's patent FR3103555). During the image acquisition step, one or more tire image acquisition systems capture one or more images of the entire surface of an identified tire. In one embodiment of the system 100, the acquisition system provides stereo-photometric images allowing the image of the imaged tire to be reconstructed in three dimensions from a single shot. The stereo-photometric images are recognized for the reconstruction of the images in their smallest details (including the anomalies present in the surface of the imaged tire).The acquisition of stereo-photometric images allows the collection of photometric data on the surface of imaged tires and the application of an anomaly detection method based on photometric stereo (see the CAD reference).

[0032] The sensor(s) of the acquisition system of the system 100 detect the presence of a tire in the field of view of the sensor, which triggers the capture of the image of the tire surface. The area (or areas) of the imaged tire could be defined in advance of the acquisition of the images. In some embodiments of the system 100, the sensor is triggered when the tire enters the field of view against the background of the captured image. The captured images are formed of pixels associated with the parameters and positioning of the tires. These images are transferred and stored as captured images in the database 110.

[0033] The acquisition system of the system 100 could incorporate an illumination source having one or more illuminations (e.g., known programmable LEDs) to serve as a source of light onto the imaged tires. The illuminations may be encoded into the acquisition system, or they may be pre-encoded during a training process of a neural network (e.g., using one or more computer programs incorporating data representative of the illuminations and images of the identified tires). Changes in the illumination source, which are captured in images obtained by the acquisition system, are represented by pixels of different color intensity (e.g., the obtained images may contain indications of reflection due to a change in illumination). Thus, the image captured by the acquisition system reveals the surfaces of the imaged tires as well as anomalies therein (if present).The camera (and / or sensor) and the lighting source may move in an alternating or random manner to adjust, respectively, the lens and the lighting. function of the parameters of the tire placed in a capture zone of the acquisition system.

[0034] In embodiments of a detection and classification method performed by the system 100, the processor may configure the inspection system 100 (and in particular a tire manufacturing site incorporating the system 100) on one or more parameters of one or more imaged tires. In these embodiments, it is understood that one or more means of reinforcement learning could be employed (including deep reinforcement learning). The processor may also refer to a reference (for example, a table of tire parameters) to make a final determination of one or more parameters of an imaged tire. The reference may include known tire parameters corresponding to a plurality of commercially available tires.For example, after the data processing module has calculated one or more parameters of an identified imaged tire, the processor may compare the detected parameters with the known parameters stored in the reference.

[0035] The system 100 also includes a database 112 having metadata stored therein that describes aspects of an imaged tire. The metadata specifies information with respect to the corresponding imaged tire, and may include some or all of the general information of such a tire. The term "general information" (singular or plural) is used herein to refer to data corresponding to an identified imaged tire.This data may include, but is not limited to, its size (which may be represented by the tire type and / or its nomenclature), its construction code (e.g., "R" for radial), its production source (e.g., the name and / or brand of the tire producer, its date and location of manufacture, distribution, and / or storage), its unique identification number (or "serial number"), its load index, its speed symbol (e.g., "A5" representing 25 km / h), and / or its expected mileage. For example, for a tire size 385 / 65R225, the number "385" represents the tire's nominal section width in millimeters, the number "65" represents the tire's aspect ratio, the letter "R" represents a radial tire, and the number "225" represents the rim diameter in inches.

[0036] Referring again to [Fig. 3] and further to [Fig. 4], the incoming data to the system 100 includes quality annotations that allow anomalies to be located and classified. The annotations could be defined according to quality criteria established in advance (for example, quality criteria standardized by the manufacturer of the tires being produced and / or quality criteria that comply with government standards). For example, anomaly classes could be defined in the following classes: - the bumps; - surface roughness; - fingerprints; - cuts; - contrast differences; - tread erosion; - the chips; and - the smooth teeth.

[0037] It is understood that, depending on the area of ​​the surface of the tire imaged, certain anomalies will be more or less present with sometimes visual differences (i.e., the “intra-class” differences are low for the same camera area but they could be high between the areas). For example, a surface roughness on the inside tread will be very different from another surface roughness on the outside tread.

[0038] It is also understood that the annotation may include information regarding the severity level associated with each anomaly class. In one embodiment, the system 100 is configured with two severity levels "severe" or "non-severe", which allows the severity classes to be rebalanced.

[0039] Referring again to Figures 3 and 4, the system 100 makes it possible to detect and segment anomalies with respect to their visual families. The severity is not susceptible to direct detection due to the high number of anomaly types having the same severity. Thus, the system 100 incorporates a dedicated network for predicting severity.

[0040] For anomaly detection, the architecture of an anomaly detection network has novelty and inventive activity by using a Mask R-CNN with two additional branches: - A Mask-IOU branch to predict the Intersection-over-Union (loU) between the predicted mask and its ground truth mask (the architecture of the loU branch is taken from H. Huang). The objective is to describe the quality of instance segmentation and eliminate false positives not only based on the classification scores but also based on the predicted IoU; and - A Bbox-IoU branch to predict the IoU between the predicted Bbox and its ground truth Bbox (see Wang, J., et al., “Semi-Supervised Active Learning for Instance Segmentation via Scoring Predictions”, https: / / arxiv.org / pdf / 2012.04829 (2020)(“the Wang reference”).

[0041] After detection, the system 100 must extract features from each detected anomaly in order to classify it according to its severity. In general, for an image, the anomaly areas have unique characteristics that result in varied intensity values ​​compared to the background (e.g., grease stains are generally dark in color which results in lower intensity values ​​corresponding to the annotation "contrast differences"). The deep learning function and the gradient and gray value features on the edges of the anomalies motivate the use of the deep learning-based Mask R-CNN to segment each anomaly from the background of the tire image.

[0042] In one embodiment, the system 100 is configured to extract the following features: - Coordinates of the anomaly's bounding box (e.g., the center of the bounding box) because the severity depends on its location; - The height and width of the bounding box of the anomaly to have the Height / Width ratio characteristic; - The class probability of the classification branch; - Predictions of the mask utility index and the bounding box utility index; - The anomaly surface calculated on the anomaly mask; and - Metadata information (including general information of the imaged tires).

[0043] Referring again to Figures 3 and 4, and further to [Fig.5], a training process is defined whose Mask R-CNN weights are pre-trained on the COCO dataset (or "the COCO dataset") which is known to those skilled in the art (see https: / / cocodataset.Org / #home). The neural network is then trained globally with all the images obtained at a fixed ratio and scale (1024*1024) (see [Fig.4]). This first training allows the network to adapt to the stereo-photometric images and to locate anomalies. Afterwards, a zone-based adjustment strategy is applied with respect to the ratio and size of the original images. The maximum possible resolution is maintained with respect to the memory of which known graphics processing units (or "Graphics Processing Units" or "GPUs") are arranged for training.

[0044] The system 100 uses a region proposal network (or "Region Proposal Network" or "RPN") which proposes bounding boxes of candidate objects (being candidate anomalies detected on the surface of an imaged tire). The system 100 extracts features from each candidate box and performs classification and regression of the bounding boxes. In embodiments of the system 100, the system extracts features using region of interest pooling (or "Roi Pooling" or "RoIPool") which is recognized to solve the problem of fixed image size requirement for the object detection network (see reference He). The features used by these two extractions can be shared for faster inference.

[0045] For network training, the total loss is a weighting of the individual losses. The total loss, which can be considered as the sum of the previous losses, is obtained by using a mathematical model of the type: [Math 1] ^Tota!= ^1 ' + K ' ^rpn„.,+ Xj ' Lds + X4 • Lreg + X5 • + X6 ■ LloU,^ Or : - L rpneis represents the RPN classification loss which is a cross-entropy loss for anchor classification between background and objects; - L^eg represents the regression loss which is the regression of the location of the anchors; - Lck represents the faster classification loss which is a cross-entropy loss for classifying bounding boxes based on their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied on a sampled set of regions of interest; - LloUbbox represents the mean square error loss (or "MSE loss") between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and - LloU mask is exactly the same as LloUbbox except that the utility index is calculated per pixel inside each bounding box.

[0046] Referring again to Figures 3-5, and further to [Fig.6], [Fig.6] depicts a deep learning-based severity classification with a custom network. As depicted, the system 100 uses a custom multimodal deep learning neural network to incorporate the original image with the extracted features. There is one dataset per imaged tire area, with one anomaly instance detected per row.

[0047] In the embodiment of the personalized network shown in [Fig.6], the severity classification is composed of three parts: - a multilayer perceptron (or “multilayer perceptron” or “MLP”) to extract characteristics from tabular data; - a feature extraction method using a Resnetl8 encoder to extract patch features and a gated recurrent unit (GRU) to capture patch interactions; and - a network using an MLP and the attention mechanism to fuse visual and tabular features.

[0048] This architecture takes into account the improvement of classification results through the incorporation of metadata. A self-attention module allows the network to exclude irrelevant information such as the background. A cross-attention module ensures that each of the modalities guides the other.

[0049] To extract features in each image obtained, the system 100 incorporates a Resnetl8 encoder trained per imaged tire area. For example, this training performs a classification of small anomalies with three (3) classes according to their severity (no anomaly, no serious anomaly, serious anomaly). This pre-training makes it possible to ensure that the extracted features are relevant.

[0050] Next, a bidirectional LSTM (or "BiLSTM") approach is used which effectively increases the amount of information available to the network. This BiLSTM approach uses contextual modeling to classify patch sequences from high-resolution images. The RNN is useful for processing sequences of different lengths, so it remains independent of the original image size, the area of ​​the anomaly bounding box. It is also possible to process the image at its full resolution without resizing it.

[0051] The memory of the system 100, in executing the instructions of the image processing module to determine one or more parameters of each imaged tire, executes an anomaly detection and classification method (or “method”). As used herein, the term “method” or “process” may include one or more steps performed by at least one computer system having one or more processors to execute instructions that perform the steps. Unless otherwise indicated, any sequence of steps is given by way of example and does not limit the described methods to any particular sequence.

[0052] To start the method, the method comprises a step of acquiring data corresponding to the imaged tires. During this step, one or more tire image acquisition systems (for example, a system of the type disclosed by the Applicant's patent FR3103555) captures one or more images of the entire surface of an identified tire.

[0053] In one embodiment of the system 100, the system 100 provides stereo-photometric images making it possible to reconstruct the image of the tire in three dimensions from a single shot. The stereo-photometric images are recognized for the reconstruction of images in their smallest details (including anomalies present in the surface of the imaged tire). The acquisition of stereo-photometric images makes it possible to collect photometric data on the surface of imaged tires and apply an anomaly detection method based on photometric stereo (see the CAD reference).

[0054] The method comprises a step of training a network for object detection. During this step, a detection neural network is trained on images (for example, stereo-photometric images obtained during the previous acquisition step) on irregular envelopes with their annotated anomaly category but also with healthy images. During this step, the network for object detection predicts, for each image, boxes containing the potential areas of anomalies with the associated anomaly type and its associated confidence. Work around the confidence of the predictions has been done thanks in particular to the prediction of the trained model's own confidence in the quality of its segmentation (i.e., "Bbox loU" and "Segmentation IoU") (see [Fig.6]).

[0055] The method comprises a step comprising a combination of a step of transforming the outputs of the network for object detection and a step of integrating metadata. During this step, before training a second network using the outputs of the first network, the outputs of the network for object detection are formatted to make them compatible with a binary classification machine learning algorithm (as understood, these formattings are carried out by means of categorical encoding of the anomalies, extractions of new characteristics of the segmented anomaly such as its surface area, its length / width ratio, its height, etc.). The metadata (for example, the dimension measurements of the imaged tire, the camera area concerned, etc.) are also integrated into these characteristics.

[0056] The method comprises a step of training the second network to predict the severity of a detected anomaly. During this step, all the features integrated during the previous step are recovered in a new data set which is constructed in a way where a row corresponds to a detection of the anomaly with its new features. To define the severity of the anomaly of this detection, the following rules are applied: - If the detection has an intersection with an anomaly annotated as serious, the target will be “Serious” severity; - If the detection has an intersection with an anomaly annotated as non-serious, or if there is no intersection with an anomaly (“false positive”), the target will be “Non-Serious” severity.

[0057] Thus, the second model could be trained to predict the severity based on this second data set combined with the initial images (obtained during the acquisition step). It is understood that, depending on the types of acquisition systems, the models could be specialized by the camera zone or by a grouping of camera zones in order to obtain the best performances per zone.

[0058] The system 100 of the invention can easily repeat one or more steps of a detection and classification method in an order to properly achieve the industrial vision parameters set up at a site where the tires are being produced.

[0059] The detection and classification processes performed by the system 100 may be performed by the PLC control and may include pre-programming of the tire production information. For example, a setting of an inspection process may be associated with the parameters of an identified tire so that a production line is configured on one or more parameters calculated by a data processing module obtained by the system 100 from each production line.

[0060] A setting of a detection and classification method performed by the system 100 may be associated with the parameters of the typical physical environments in which the system 100 operates. In embodiments of the invention, the system 100 (and / or a tire production site incorporating the system 100) may receive voice commands or other audio data representing, for example, a step or a stop of the production line. The request may include a request for the current status of a tire production cycle. A generated response may be represented audibly, visually, tactilely (for example, using a haptic interface), and / or virtually and / or augmentedly. This response, associated with the corresponding data, may be recorded in one or more neural networks.

[0061] For all embodiments of the system 100, a monitoring system could be implemented. At least a portion of the monitoring system may be provided in a portable device such as a mobile network device (e.g., a mobile phone, a laptop, one or more network-connected wearable devices (including “augmented reality” and / or “virtual reality” devices), network-connected wearables, and / or any combinations and / or equivalents). It is envisaged that detection and comparison steps may be performed iteratively.

[0062] The terms "at least one" and "one or more" are used interchangeably. Ranges that are presented as being "between a and b" encompass the values ​​"a" and "b".

[0063] Although particular embodiments of the disclosed system have been illustrated and described, it will be understood that various changes, additions, and modifications may be practiced without departing from the spirit and scope of the present disclosure. Accordingly, no limitations should be imposed on the scope of the disclosed invention except those set forth in the appended claims.

[0064] The anomaly detection step, described with reference to [Fig.3], allows - the acquisition of images of the tires and the location of the anomalies, - for each anomaly detected, the extraction of characteristics of the determined anomalies (for example the surface of the anomaly, the position of the anomaly on the tire, the identity of the tire, etc.) - the allocation of a severity rating for the anomaly, with a classification into at least two categories: 1st category: no repair, 2nd category: repair.

[0065] After the anomaly detection step, an enriched database is constructed which, for each tire, lists all the anomalies detected, as well as an overall severity rating for the tire.

[0066] For example, the enriched database can be produced from some or all of the results from the anomaly detection step. This enriched database has the advantage of giving an overall severity score for the entire tire, which constitutes more interesting information and approach than giving a severity score anomaly by anomaly.

[0067] A third algorithm is constructed using the enriched database, in order to produce this third algorithm which makes it possible to assign an overall severity rating for a tire, from an acquisition of images of the tires.

[0068] The third algorithm implements an artificial intelligence algorithm based on multi-instance learning (called “multiple instance learning algorithm” in English). This algorithm takes into account a variable number of anomalies from one tire to another, while being able to assign a severity rating to said tire. The severity rating is determined by using the characteristics of each of the anomalies detected for the tire, and the number of anomalies being variable from one tire to another.

[0069] Then we implement the third algorithm in the following manner.

[0070] An anomaly detection step is again carried out, in a manner similar to that described with reference to [Fig.3]: - for each tire, we acquire images of the tires and locate the anomalies, - for each tire, we determine all the anomalies detected, and we extract characteristics of the determined anomalies (for example the surface of the anomaly, the position of the anomaly on the tire, the identity of the tire, etc.).

[0071] Then, for a tire, the characteristics of the detected anomalies are provided to the third algorithm based on multi-instance learning, so that it can assign a severity rating. The severity rating can be associated with a classification into at least two categories: 1st category: no repair, 2nd category: repair.

Claims

1. Claims A detection and classification system (100) which performs a method of detecting and classifying anomalies from images of tire surfaces, characterized in that the system comprises: - one or more tire image acquisition systems enabling the evaluation of the condition of the entire surface of an imaged tire; - at least one communication network that manages the incoming data to the system (100), wherein each communication network comprises one or more communication servers each comprising one or more processors operatively connected to a memory configured to store an image processing module of the processor, which analyzes the images obtained from the image acquisition systems to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire, wherein each processor comprises an analysis application execution module that performs the processing of the obtained images and each processor is capable of executing programmed instructions stored in the memory to perform the steps of the detection and classification method; - at least one database (110) in which the images obtained from the tire image acquisition systems are stored; - at least one database (112) having metadata stored therein that describes aspects of each imaged tire; wherein the images obtained from the tire image acquisition systems are formed of pixels associated with the parameters and positioning of the anomalies, these images being transferred and stored as captured images in the memory of the inspection system (100); - an anomaly detection network that uses a Mask R-CNN with two additional branches including: - a Mask-IOU branch to predict the Intersection-over-Union (loU) between a predicted mask and its ground truth mask; and - a Bbox-IOU branch to predict the IoU between a predicted bounding box (Bbox) and its bounding box (Bbox); wherein: - the weights of the Mask R-CNN are pre-trained; - a neural network is then globally trained with all the images obtained at a fixed ratio and scale; and - a zone-based adjustment strategy is applied with respect to the ratio and size of the original images; so that the system (100) adapts to the obtained images of the tires and locates anomalies on their surfaces.

2. The system (100) of claim 1, wherein the weights of the Mask R-CNN are pre-trained on the COCO dataset.

3. The system (100) of claim 1 or claim 2, wherein the system (100) is configured to extract the following features: - Coordinates of the anomaly bounding box; - The height and width of the anomaly bounding box; - The class probability of the classification branch; - The predictions of the mask utility index and the bounding box utility index; - The anomaly surface calculated on the anomaly mask; and - The metadata information of the imaged tire.

4. The system (100) of claim 3, wherein, for training the network, the total loss is a weighting of the individual losses and is obtained by employing a mathematical model of the type: [Math 1] \ L ' ^rpn„s+ X3 • Lds+ À4 • £«$ + X5 • \ Where: - L rpneis represents the RPN classification loss which is a cross-entropy loss for the classification of anchors between the background and objects; - Lrpnreg represents the regression loss which is the regression of the anchor location; - Lckrepresents the faster classification loss which is a cross-entropy loss for the classification of bounding boxes according to their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied on a sampled set of regions of interest; - Llou_bbox represents the mean square error loss (or "MSE loss") between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and - LloU mask is exactly the same as for LloU bbox except that the utility index is calculated per pixel inside each bounding box.

5. The system (100) of claim 4, wherein the system (100) comprises a deep learning-based severity classification network, wherein the classification network comprises: - at least one multi-layer perceptron (MLP) for extracting features of detected anomalies; - a feature extraction method using a Resnetl8 encoder and a gated recurrent unit (GRU); and - a network using an MLP and an attention mechanism for fusing all extracted features.

6. The system (100) of claim 5, wherein the system (100) incorporates a Resnetl8 encoder driven per imaged tire area; such that this training performs a classification of anomalies according to their severity.

7. The system (100) of claim 6, wherein the memory, by executing the instructions of the image processing module in order to determine one or more parameters of each imaged tire, executes a method for detecting and classifying anomalies comprising the following steps: - a step of acquiring data corresponding to the imaged tires; - a step of training a network for detecting objects, during which a neural detection network is trained on images obtained during the previous acquisition step, and during which the network for detecting objects predicts, for each image, boxes containing the potential zones of anomalies with the associated type of anomaly and its associated confidence; - a step comprising a combination of a step of transforming the outputs of the network for object detection and a step of integrating the metadata, during which, before training a second network using the outputs of the first network, the outputs of the network for object detection are formatted in order to make them compatible with a binary classification machine learning algorithm, and during which the metadata are also integrated; and - a step of training the second network to predict the severity of a detected anomaly.

8. The system (100) of any one of claims 1 to 7, wherein the tire image acquisition system(s) provide stereo-photometric images.

9. The system (100) of any one of claims 1 to 8, wherein the system (100) further comprises a detection system for collecting information about the physical environment around the system (100), the detection system comprising an illumination source having one or more illuminations whose illuminations are encoded in the detection system and / or pre-coded during a training process of a neural network.

10. The system (100) according to one of claims 1 to 9, further comprises - an enriched database which, for each tire, lists all of the detected anomalies, as well as an overall severity rating for the tire, - a device implementing a third algorithm using the enriched database, which makes it possible to assign an overall severity rating for a tire, from an acquisition of images of the tires.

11. A tire production site, comprising the system (100) of any one of claims 1 to 10.