Anomaly detection and classification system for rubber products

Abstract

The invention relates to a detection and classification system (100) that performs a method for detecting and classifying anomalies from images of tire surfaces. The invention also relates to a tire production site comprising the disclosed system (100). Figure for the abstract: Fig. 3

Description

Title of the invention: System for detecting and classifying anomalies in rubber products. Technical field

[0001] The invention relates to a machine vision-based detection and classification system for detecting rubber products such as tires. The disclosed detection and classification system establishes a data-driven link between a theoretical quality standard and real-world conditions. Context

[0002] In the field of industrial processes, the surfaces of tires intended for use on vehicles are difficult to inspect. These tires are generally black due to the black rubber compound they are made of (resulting from the use of carbon to reinforce the elastomeric compounds). Therefore, automated detection of tire surface defects is very difficult to achieve.Furthermore, the form of an anomaly can be related to its severity, with different anomalies varying according to distinct anomaly classes that include, but are not limited to, various blemishes (including grease stains), dents, scratches, surface roughness, indentations, cuts, contrast differences, tread erosion, chips, smooth teeth, material deficiencies, material excesses, differences in black shades, cracks, etched materials, and localized discolorations. As used here, the term "anomaly" refers to a detected difference (represented, for example, by a deviation from a standard value) without further information (therefore, an "anomaly" is not automatically interpreted as a "defect").

[0003] Prior art solutions exist to overcome this challenge. For example, the Applicant's French patent FR3103555 discloses a system for assessing the condition of a tire by acquiring a visual image. Referring to [Fig. 1] (which corresponds to [Fig. 5] of French patent FR3103555), the disclosed system acquires images of a tire 10 having external areas (including the sidewalls 12, shoulders 14, and tread (or "top") 16) and internal areas (including the inner sides 18 and inner rim 20) using both two-dimensional and three-dimensional cameras (or "sensors"). The two-dimensional sensors are used for both the inside and outside (in embodiments incorporating three-dimensional sensors, these sensors are used only for the external surfaces of the tire 10).During a process carried out by the disclosed system, the pneumatic 10 enters the system where it is blocked and it is rotated so that the different sensors can take images of the tire (for example, by quickly lowering the sensor to near the area where it needs to be located, then slowing down to precisely adjust the position).

[0004] There are also recent developments that employ artificial intelligence (or "AI") methods, and in particular deep learning (or "deep learning"), to interpret images acquired from a tire (including the acquisition method used in French patent FR3103555). Recent improvements in deep learning techniques and data analysis, combined with computing and data storage platforms, have opened avenues for the development of new approaches in the field of machine vision. Its main objective is to use the "eyes of the machine" to recognize anomalies in images captured by image capture devices (e.g., one or more sensors and / or cameras), and to input these captured images into visual algorithms for computation.

[0005] In the field of machine vision, the prior art offers several applications for the detection and classification of anomalies in tires during production. One such application combines the histogram of oriented gradients (HOG) and local binary pattern (LBP) features in a method for detecting tire appearance anomalies (see Liu, Hongbin, et al., "Tire appearance defect detection method via combining HOG and LBP features," Front. Phys., DOI: 10.3389 / fphy.2022.1099261 (January 12, 2023) ("Liu reference")). In this method, a dataset of tire images is constructed to provide images of tires with and without anomalies.Next, the HOG and LBP features of the tire images are extracted and analyzed to train the support vector machine (SVM). Finally, the SVM classifier calculates prediction scores for the test images by combining the HOG and LBP features to determine whether the test image is a tire image without anomalies or a tire image with anomalies (see [Fig. 1] in Liu's reference). This application is limited to classification and does not include deep learning or detection.

[0006] In another application, a tire inspection system is disclosed in which data processing methods are performed on images obtained from tire sidewalls using a camera and a laser sensor (see Kuric I., et al., “Analysis of the Possibilities of Tire-Defect Inspection Based on Unsupervised Leaming and Deep Leaming”, Sensors, https: / / doi.org / 10.3390 / s21217073 (October 25). 202l)(“the Kuric reference”). The captured data includes visual and geometric data characterizing the surface of the imaged tire, providing a true representation of its sidewall. A unfolding process (or “polar transformation”) is performed by the inspection system to further process the data obtained by the camera. The principles and automation of the designed polar transformation, based on polynomial regression (i.e., supervised learning), are presented. Based on the laser sensor data, anomaly detection is performed using an unsupervised clustering method, followed by anomaly classification using the VGG-16 neural network (see Figure 23 of the Kuric reference). The inspection system aims to detect both formed and unformed anomalies, i.e., anomalies, rather than relying solely on supervised learning methods.Using compliant data to infer in-depth anomalies does not allow for the detection of surface anomalies in the imaged tire.

[0007] Another application proposes a method that functionally combines a convolutional neural network (CNN) and a system virtual machine (SVM) to classify defects on the inner surface of an automobile tire (see Tada, H. and Sugiura, A., “Defect Classification on Automobile Tire Inner Surfaces with Functional Classifiers,” Transactions of the Institute of Systems, Control and Information Engineers, Vol. 34, No. 1, pp. 1–10 (2021) (“the Tada reference”)). In this method, dividing an image into multiple images of smaller regions increases the number of images, which limits the applicable machine learning methods. Thus, the CNN is applied to the divided images of the entire tire, while the SVM is applied to the divided images within the range delimited by the CNN (see Figure 9 of the Tada reference).The reliance on patch-by-patch image processing does not affect any detection network but only a classification method. Focusing solely on the inner tread does not allow for the detection of surface anomalies on the imaged tire.

[0008] In the field of machine vision, the prior art inspires other applications beyond the detection and classification of anomalies in tires during production. For example, a proposed system employs a method based on photometry and stereoscopy that combines photometric stereometry and anomaly detection to eliminate the interference of highlights and shadows on metallic surfaces (see Cao, Y. et al., “Photometric-Stereo-Based Defect Detection System for Metal Parts,” Sensors, https: / / doi.org / 10.3390 / s22218374 (November 1, 2022) (“the Cao reference”). On this basis, the system takes as input images captured with multiple directional lights and obtains the normal map using the stereo photometric model. The detection model then uses the normal map to locate and classify anomalies (see Figure 8 in the CAD reference). This approach does not address post-processing with metadata or the concept of the severity level of a detected anomaly.

[0009] It is observed that these techniques require large amounts of annotated data to reveal their full effectiveness. Consequently, their initial implementation cost is not negligible. It is therefore useful to enable a deep network to be "aware" of the quality of its own predictions.

[0010] Among the known deep learning techniques are recurrent neural networks (RNNs) that are capable of processing input sequences of varying lengths. An RNN, an artificial neural network with recurrent connections, processes the variable-length sequence by having a recurrent hidden state whose activation at each time step depends on that of the previous time step (see Chung, J. et al., "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling," NIPS 2014 Deep Learning and Representation Learning Workshop, https: / / doi.org / 10.48550 / arXiv.1412.3555 (December 11, 2014) ("the Chung reference"). It is difficult to train RNNs to capture long-term dependencies because gradients tend to vanish.Thus, the gated recurrent network (or a "gated recurrent unit" or "GRU"), being a recurrent unit of the RNN, could be proposed to capture dependencies at different time scales in an adaptive manner (see the reference Chung)(citing Cho, K, Bahdanau, D. and Bengio Y., "Neural .machine translation by jointly learning to align and translate", arXiv: 1409.0473 (2014)("the reference Cho")).

[0011] Prior art proposes a mask-noting R-CNN (or "Mask R-CNN") which includes a deep learning instance segmentation technique. A Mask R-CNN is an architecture developed to handle the task of instance segmentation. This method efficiently generates a bounding box and a segmentation mask for each instance in an image (see He, K., et al., "Mask R-CNN", arXiv: 1703.06870) (January 24, 2018) (the "He reference"). With reference to [Fig.2], [Fig.2] represents a network for directly learning the Intersection-over-Union (or “intersection-over-union” or “loU”) between the predicted mask and the field mask to describe the quality of instance segmentation (see Huang, Z., et al., “Mask Scoring R-CNN”, arXiv: 1903.00241vl (Tuesday 1, 2019) (“the Z. Huang reference”) ([Fig.2] corresponds to [Fig.3] in the Z. Huang reference).In the architecture shown, the input image is fed into a background network to generate regions of interest (or "ROI") via a region proposal network (or "Region Proposal Network" or "RPN") and ROI features via a . A ROI alignment layer that produces fixed-size feature maps for each rectangular ROI in an input feature map ("RoIAlign") (see https: / / www.mathworks.com / help / vision / ref / nnet.cnn.layer.roialignlayer.html). To predict the MaskloU, the predicted mask and the ROI feature are used as input.

[0012] The Mask R-CNN performs pixel-level segmentation on detected objects. It can therefore support multiple classes and overlapping objects. To take advantage of this benefit, deep learning-based methods using a Mask R-CNN are being developed in the field of machine vision (for example, see Huang, H., et al., “Deep learning-based instance segmentation of cracks from shield tunnel lining images”, Structure and Infrastructure Engineering, D01:10.1080 / 15732479.2020.1838559 (2020) (“reference H. Huang”).

[0013] In order to improve the field of machine vision for the benefit of tire manufacturing (where rubber products are difficult to detect and anomalies are varied), the disclosed invention provides a detection and classification system that allows the integration of images and metadata of the imaged tires into an algorithm. The disclosed invention thus achieves a performance gain, enabling the implementation of more complex rules that approach the quality standards imposed for the visual inspection of tires during production. Summary of the invention

[0014] The invention relates to a detection and classification system that performs a method for detecting and classifying anomalies from images of tire surfaces, characterized in that the system comprises: - one or more tire image acquisition systems allowing the evaluation of the condition of the entire surface of an imaged tire; - at least one communication network which manages the data entering the system, in which each communication network includes one or more communication servers each comprising one or more processors operationally connected to a memory configured to store an image processing module of the processor, which analyzes the images obtained from the image acquisition systems to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire, where each processor includes an analysis application execution module which performs the processing of the images obtained and each processor is capable of executing programmed instructions stored in memory to perform the steps of the detection and classification process; - at least one database in which images obtained from tire image acquisition systems are stored; - at least one database with stored metadata describing aspects of each imaged tire; in which the images obtained from the tire image acquisition systems are formed of pixels associated with the parameters and positioning of anomalies, these images being transferred and stored as captured images in the memory of the inspection system; - an anomaly detection network that uses an R-CNN mask with two additional branches comprising: - a Mask-IOU branch to predict the Intersection-on-Union (loU) between a predicted mask and its ground truth mask; and - a Bbox-IOU branch to predict the IoU between a predicted bounding box (Bbox) and its bounding box (Bbox); in which: - the Mask R-CNN weights are pre-trained; - a neural network is then trained globally with all the images obtained at a fixed ratio and scale; and - a zone-based adjustment strategy is applied in relation to the ratio and size of the original images; so that the system adapts to the images obtained from the tires and locates anomalies on their surfaces.

[0015] In certain embodiments of the inspection system of the invention, the Mask R-CNN weights are pre-trained on the COCO dataset.

[0016] In certain embodiments of the inspection system of the invention, the system is configured to extract the following characteristics: - Coordinates of the bounding box of the anomaly; - The height and width of the bounding box of the anomaly; - The probability of the class of the classification branch; - Predictions of the utility index of the mask and the utility index of the bounding box; - The anomaly surface calculated on the anomaly mask; and - Information on the metadata of the imaged tire.

[0017] In certain embodiments of the inspection system of the invention, the total loss is a weighting of the individual losses and is obtained by employing a mathematical model of the type: [Math 1] ^Total ~ ' ^2 ' Lrpnreg + X3 • Lelx + k4 • Lreg + X5 • L{OU+ ^6 ' LloU^ Or : - L jpn ck represents the RPN classification loss which is a cross entropy loss for anchor classification between background and objects; - Lrpnreg represents the regression loss which is the regression of the anchor location; - Lck represents the faster classification loss which is a cross-entropy loss for the classification of bounding boxes according to their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied to a sampled set of regions of interest; - LloU bbox represents the root mean square error loss (or "MSE loss") between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and - LloU mask is exactly the same as for LloU bbox except that the utility index is calculated per pixel inside each bounding box.

[0018] In certain embodiments of the inspection system of the invention, the system includes a severity classification network based on deep learning, wherein the classification network comprises: - at least one multilayer perceptron (MLP) to extract the characteristics of the detected anomalies; - a feature extraction method using a Resnetl8 encoder and a gated recurrent unit (GRU); and - a network using an MLP and an attention mechanism to merge all extracted features.

[0019] In certain embodiments of the inspection system of the invention, the system incorporates a Resnetl8 encoder driven by an imaged pneumatic zone; so that this training performs a classification of anomalies according to their severity.

[0020] In certain embodiments of the inspection system of the invention, the memory, by executing the instructions of the image processing module in order to determine one or more parameters of each imaged tire, performs a method for detecting and classifying anomalies comprising the following steps: - a step of acquiring the data corresponding to the imaged tires; - a training step for a network for object detection, during which a detection neural network is trained on images obtained during the previous acquisition step, and during which the network for object detection predicts, for each image, boxes containing the potential areas of anomalies with the type of anomaly associated and its associated confidence; - a step comprising a combination of a network output transformation step for object detection and a metadata integration step, during which, before training a second network using the outputs of the first network, the network outputs for object detection are formatted to make them compatible with a binary classification machine learning algorithm, and during which the metadata is also integrated; and - a training step for the second network to predict the severity of a detected anomaly. In certain embodiments of the inspection system of the invention, the tire image acquisition system(s) provide stereo-photometric images.

[0021] In certain embodiments of the inspection system of the invention, the system further comprises a detection system for collecting information on the physical environment around the system, the detection system comprising a light source having one or more lights whose lights are encoded in the detection system and / or pre-coded during a neural network training process.

[0022] The invention also relates to a tire production site, including the disclosed detection and classification system.

[0023] Other aspects of the invention will become evident from the following detailed description. Brief description of the drawings

[0024] The nature and various advantages of the invention will become more evident upon reading the following detailed description, together with the accompanying drawings, in which the same reference numbers designate identical parts throughout, and in which: [Fig.1] Fig.1 represents the different areas of a tire where images must be acquired during a process carried out by a prior art system enabling the acquisition of a visual image of the tire surfaces. [Fig.2] Fig.2 represents an overall architecture of a Mask R-CNN type network from the prior art. [Fig.3] Fig.3 represents an overall architecture of a detection and classification system of the invention. [Fig.4] The [Fig.4] represents a detailed architecture of a model for predicting the severity of anomalies detected during a detection and classification process carried out by the system of the [Fig.3]. [Fig. 5] [Fig. 5] represents part of a training process incorporating a zone-based adjustment strategy for an imaged tire. [Fig.6] [Fig.6] represents a classification based on deep learning with a custom network. Detailed description

[0025] Referring now to the figures, in which the same numbers identify identical elements, [Fig. 3] represents an overall architecture of a detection and classification system (or "system") 100. The system 100 performs a method for detecting and classifying surface anomalies of imaged tires (or "detection and classification method" or "method"). The system 100 performs this method via a learning process that employs a succession of two networks representing two deep learning architectures: a first network for object detection, and a second network using the outputs of the first network, reworked and then combined with metadata that describe aspects of an imaged tire (for example, the dimensions of the tire, the area(s) of the imaged tire, etc.) in order to predict the severity of each anomaly detected by the first network.The metadata of a file representing an imaged tire can indicate that the file provides visually indicative images of the presence of a particular anomaly (and also the absence of an anomaly in the surface of the imaged tire). This structure makes System 100 faster and more robust because it can adapt to all sizes and types of tires it sees. It is understood that System 100 could be part of a site where tire production takes place.

[0026] The system 100 includes at least one communication network (or "network") that manages incoming data to the system from various sources. The communication network incorporates one or more communication servers (or "servers"), each comprising one or more processors operationally connected to a memory (as used here, the term "server" incorporates one or more servers, and the term "processor" incorporates one or more processors). The memory is configured to store an image processing module of the processor, which analyzes the images obtained from the imaged tires to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire (the anomaly being visible either in whole or in part).The processor(s) include an analysis application execution module that performs image processing, and the processor(s) are capable of executing programmed instructions stored in memory to carry out the steps of the detection and classification process (as described below).

[0027] The term "processor" (or, alternatively, the term "logic circuit processor") "grammable" refers to one or more devices capable of processing and analyzing data and comprising one or more software programs for its processing (for example, one or more integrated circuits known to those skilled in the art as being included in a computer, one or more controllers, one or more microcontrollers, one or more microcomputers, one or more programmable logic controllers (PLCs), one or more application-specific integrated circuits, one or more neural networks, and / or one or more other known equivalent programmable circuits). The processor includes one or more software programs for processing the incoming data from the system, as well as one or more software programs for identifying and locating variances and identifying their sources in order to correct them.

[0028] Memory may include both volatile and non-volatile memory devices. Non-volatile memory may include solid-state memories, such as NAND flash memory, keep-alive memory (or KAM) for saving various operating variables while the processor is powered off, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is powered off or loses its power supply. Volatile memory may include static and dynamic RAM that stores program instructions and data, including a learning application.

[0029] The processor manages the incoming data to system 100 from at least one database 112 of system 100 in which the tire images are stored (as used here, the term "database" refers to one or more databases). The images stored in database 112 are obtained from at least one tire image acquisition system that is part of system 100. These images are obtained during a tire image acquisition step (and therefore the corresponding data) performed by system 100 during the detection and classification process.

[0030] In embodiments of system 100, the tire image acquisition system includes an acquisition system for evaluating the condition of the entire surface of a tire by acquiring a visual image (for example, a system of the type disclosed by Applicant's French patent FR3103555). During the image acquisition step, one or more tire image acquisition systems capture one or more images of the entire surface of an identified tire. In one embodiment of system 100, the acquisition system provides stereophotometric images that allow the image of the imaged tire to be reconstructed in three dimensions from a single shot. The stereophotometric images are recognized for reconstructing the images in their finest details (including any anomalies present in the surface of the imaged tire). The acquisition of stereo-photometric images allows the collection of photometric data on the surface of imaged tires and the application of an anomaly detection method based on stereo photometry (see Cao reference).

[0031] The sensor(s) of the acquisition system of system 100 detect the presence of a tire in the sensor's field of view, which triggers the capture of an image of the tire's surface. The area(s) of the imaged tire could be defined prior to image acquisition. In some embodiments of system 100, the sensor is triggered when the tire enters the field of view in the background of the captured image. The captured images are composed of pixels associated with the parameters and positioning of the tires. These images are transferred and stored as captured images in the database 112.

[0032] The acquisition system of system 100 could incorporate a light source having one or more illuminations (for example, known programmable LEDs) to serve as a light source on the imaged tires. The illuminations can be encoded in the acquisition system, or they can be pre-encoded during a neural network training process (for example, using one or more computer programs incorporating representative data of the illuminations and images of the identified tires). Changes in the light source, which are captured in images obtained by the acquisition system, are represented by pixels of different color intensity (for example, the images obtained may contain indications of reflection due to a change in illumination). Thus, the image captured by the acquisition system reveals the surfaces of the imaged tires as well as any anomalies therein (if present).The camera (and / or sensor) and the light source can move in an alternative or random manner to adjust, respectively, the lens and the lighting according to the parameters of the tire placed in a capture zone of the acquisition system.

[0033] In embodiments of a detection and classification process performed by the system 100, the processor can configure the inspection system 100 (and in particular a tire manufacturing site incorporating the system 100) on one or more parameters of one or more imaged tires. In these embodiments, it is understood that one or more reinforcement learning methods could be employed (including deep reinforcement learning). The processor can also refer to a reference (for example, a table of tire parameters) to perform a final determination of one or more parameters of an imaged tire. The reference can include known tire parameters corresponding to a plurality of tires. commercially available. For example, after the data processing module has calculated one or more parameters of an identified imaged tire, the processor can compare the detected parameters with the known parameters stored in the reference.

[0034] The system 100 also includes a database 114 in which metadata is stored describing aspects of an imaged tire. The metadata specifies information relating to the corresponding imaged tire and may include some or all of the general information of such a tire. The term "general information" (singular or plural) is used here to refer to the data corresponding to an identified imaged tire.This data may include, but is not limited to, its size (which can be represented by the tire type and / or its nomenclature), its construction code (e.g., "R" for radial), its source of production (e.g., the name and / or brand of the tire manufacturer, its date and place of manufacture, distribution, and / or storage), its unique identification number (or "serial number"), its load index, its speed symbol (e.g., "A5" representing 25 km / h), and / or its expected mileage. For example, for a tire size 385 / 65R225, the number "385" represents the nominal width of the tire section in millimeters, the number "65" represents the tire's aspect ratio, the letter "R" represents a radial tire, and the number "225" represents the rim diameter in inches.

[0035] With further reference to [Fig. 3] and [Fig. 4], the input data to system 100 includes quality annotations that allow for the location and classification of anomalies. The annotations could be defined according to pre-established quality criteria (for example, quality criteria standardized by the tire manufacturer during production and / or quality criteria that conform to government standards). For example, the classes of anomalies could be defined as follows: - the bumps; - surface roughness; - fingerprints; - the cuts; - differences in contrast; - erosion of the tread; - the splinters; and - the smooth teeth.

[0036] It is understood that, depending on the area of ​​the imaged tire surface, certain anomalies will be more or less present, sometimes with visual differences (i.e., the "intra-class" differences are small for the same camera area, but they (could be high between zones). For example, a surface roughness on the inside of the tread will be very different from another surface roughness on the outside of the tread.

[0037] It is also understood that the annotation may include information concerning the severity level associated with each class of anomaly. In one embodiment, the system 100 is configured with two severity levels, "severe" or "not severe," which allows for a rebalancing of the severity classes.

[0038] Referring again to Figures 3 and 4, the System 100 allows for the detection and segmentation of anomalies according to their visual families. Severity is not susceptible to direct detection due to the large number of anomaly types with the same severity. Thus, the System 100 incorporates a dedicated network to predict severity.

[0039] For anomaly detection, the architecture of an anomaly detection network possesses novelty and inventive step by using an R-CNN Mask with two additional branches: - A Mask-IOU branch to predict the Intersection-on-Union (loU) between the predicted mask and its ground truth mask (the loU branch architecture is taken from the H. Huang reference). The goal is to describe the quality of instance segmentation and eliminate false positives not only based on classification scores but also on the predicted IoU; and - A Bbox-IoU branch to predict the IoU between the predicted Bbox and its ground truth Bbox (see Wang, J., et al., “Semi-Supervised Active Leaming for Instance Segmentation via Scoring Predictions”, https: / / arxiv.org / pdf / 2012.04829 (2020)(“the Wang reference”).

[0040] After detection, the system 100 must extract features from each detected anomaly in order to classify it according to its severity. In general, for an image, anomaly areas exhibit unique characteristics that result in varying intensity values ​​relative to the background (for example, grease stains are generally dark in color, resulting in lower intensity values ​​corresponding to the annotation "contrast differences"). The deep learning function and the gradient and gray-value characteristics at the edges of the anomalies motivate the use of the Mask R-CNN based on deep learning to segment each anomaly from the background of the tire image.

[0041] In one embodiment, the system 100 is configured to extract the following features: - Coordinates of the bounding box of the anomaly (for example, the center of the bounding box) because the severity depends on its location; - The height and width of the bounding box of the anomaly to have the characteristic of Height / Width ratio; - The probability of the class of the classification branch; - Predictions of the utility index of the mask and the utility index of the bounding box; - The area of ​​the anomaly calculated on the anomaly mask; and - Information on the metadata (including general information on the imaged tires).

[0042] Referring again to Figures 3 and 4, and further to [Fig. 5], a training process is defined in which the weights of the Mask R-CNN are pre-trained on the COCO dataset, which is known to those skilled in the art (see https: / / cocodataset.org / #home). The neural network is then trained globally with all the images obtained at a fixed ratio and scale (1024 x 1024) (see [Fig. 4]). This initial training allows the network to adapt to stereophotometric images and to locate anomalies. Subsequently, a zone-based tuning strategy is applied with respect to the ratio and size of the original images. The maximum possible resolution is maintained with respect to the memory, for which known graphics processing units (GPUs) are allocated for training.

[0043] System 100 uses a Region Proposal Network (RPN) that proposes bounding boxes of candidate objects (being candidate anomalies detected on the surface of an imaged tire). System 100 extracts features from each candidate box and performs classification and regression of the bounding boxes. In embodiments of System 100, the system extracts features using Region of Interest Pooling (ROI Pooling or ROIPool), which is known to solve the problem of the fixed image size requirement for the object detection network (see reference H). The features used by these two extractions can be shared for faster inference.

[0044] For network training, the total loss is a weighting of the individual losses. The total loss, which can be considered as the sum of the previous losses, is obtained by employing a mathematical model of the type: [Math 1] ^2 ' X3 • Le[s+ X4 • Lreg + X5 • LjouUxxres + Or: - L ck represents the RPN classification loss which is a cross entropy loss for anchor classification between background and objects; - Lrpnreg represents the regression loss, which is the regression of the localization of anchors; - Lck represents the faster classification loss which is a cross-entropy loss for the classification of bounding boxes according to their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied to a sampled set of regions of interest; - LloUbbox represents the root mean square error loss (or "MSE loss") between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and - LloU mask is exactly the same as for LloUbbox except that the utility index is calculated per pixel inside each bounding box.

[0045] Referring again to Figures 3 to 5, and further to [Fig. 6], [Fig. 6] represents a severity classification based on deep learning with a custom network. As shown, the system uses a custom multimodal deep learning neural network to incorporate the original image with the extracted features. There is one dataset per imaged tire area, with one instance of anomaly detected per line.

[0046] In the embodiment of the custom network shown in [Fig. 6], the severity classification is composed of three parts: - a multilayer perceptron (or "multilayer perceptron" or "MLP") to extract features from tabular data; - a feature extraction method using a Resnetl8 encoder to extract features from patches and a gated recurrent unit (or "GRU") to capture interactions between patches; and - a network using an MLP and the attention mechanism to fuse visual and tabular features.

[0047] This architecture takes into account the improvement of classification results through the incorporation of metadata. A self-attention module allows the network to exclude irrelevant information such as the background. A cross-attention module ensures that each modality guides the other.

[0048] To extract features from each image obtained, the system 100 incorporates a Resnetl8 encoder driven by an imaged pneumatic zone. As an example, this training performs a classification of minor anomalies into three (3) classes based on their severity (no anomaly, no serious anomaly, serious anomaly). This pre-training ensures that the extracted features are relevant.

[0049] Next, a bidirectional LSTM (or "BiLSTM") approach is used, which effectively increases the amount of information available to the network. This The BiLSTM approach uses contextual modeling to classify patch sequences from high-resolution images. The RNN is useful for processing sequences of varying lengths, thus remaining independent of the original image size and the anomaly delimiter box area. It is also possible to process the image at its full resolution without resizing it.

[0050] The memory of system 100, by executing the instructions of the image processing module to determine one or more parameters of each imaged tire, performs an anomaly detection and classification method (or "method"). As used herein, the term "method" or "process" may include one or more steps performed by at least one computer system comprising one or more processors to execute instructions that perform the steps. Unless otherwise specified, any sequence of steps is given by way of example and does not limit the described methods to any particular sequence.

[0051] To start the process, the process includes a step of acquiring the data corresponding to the imaged tires. During this step, one or more tire image acquisition systems (for example, a system of the type disclosed by Applicant's patent FR3103555) captures one or more images of the entire surface of an identified tire.

[0052] In one embodiment of system 100, the system provides stereophotometric images that allow the three-dimensional image of the tire to be reconstructed from a single shot. The stereophotometric images are recognized for their ability to reconstruct images in minute detail (including anomalies present on the surface of the imaged tire). The acquisition of stereophotometric images allows for the collection of photometric data on the surface of imaged tires and the application of a stereophotometric anomaly detection method (see CAD reference).

[0053] The method includes a step of training a network for object detection. During this step, a detection neural network is trained on images (for example, stereophotometric images obtained during the previous acquisition step) of irregular envelopes with their anomaly category annotated, as well as on undamaged images. During this step, the object detection network predicts, for each image, boxes containing the potential areas of anomalies with the associated anomaly type and its associated confidence level. Work on the confidence of the predictions was carried out, notably through the prediction of the trained model's own confidence in the quality of its segmentation (i.e., "Bbox loU" and "Segmentation IoU") (see [Fig. 6]).

[0054] The method includes a step comprising a combination of a network output transformation step for object detection and an integration step metadata. During this step, before training a second network using the outputs of the first network, the outputs of the object detection network are formatted to make them compatible with a binary classification machine learning algorithm (as understood, this formatting is achieved through categorical encoding of anomalies, extraction of new features from the segmented anomaly such as its area, length / width ratio, height, etc.). Metadata (for example, the dimensions of the imaged tire, the camera area involved, etc.) are also integrated into these features.

[0055] The method includes a step of training the second network to predict the severity of a detected anomaly. During this step, all the features integrated during the previous step are retrieved into a new dataset which is constructed in such a way that one row corresponds to a detection of the anomaly with its new features. To define the severity of the anomaly for this detection, the following rules are applied: - If the detection has an intersection with an anomaly annotated as serious, the target will be of "Severe" severity; - If the detection has an intersection with an anomaly annotated as non-severe, or if there is no intersection with an anomaly ("false positive"), the target will be severity "Non-Severe".

[0056] Thus, the second model could be trained to predict severity based on this second dataset combined with the initial images (obtained during the acquisition stage). It is understood that, depending on the types of acquisition systems, the models can be specialized by camera area or by a grouping of camera areas in order to obtain the best performance per area.

[0057] The system 100 of the invention can easily repeat one or more steps of a detection and classification process in an order to properly achieve the industrial vision parameters set up at a site where tires are being produced.

[0058] The detection and classification processes performed by the system 100 can be controlled by the PLC and may include pre-programming of tire production information. For example, a setting for an inspection process can be associated with the parameters of an identified tire so that a production line is configured based on one or more parameters calculated by a data processing module using data obtained by the system 100 from each production line.

[0059] A setting of a detection and classification process performed by the system 100 can be associated with the parameters of typical physical environments in which System 100 is operational. In embodiments of the invention, System 100 (and / or a tire production site incorporating System 100) can receive voice commands or other audio data representing, for example, a step or a stop on the production line. The request may include a request for the current status of a tire production cycle. A generated response can be represented audibly, visually, tactilely (for example, using a haptic interface), and / or virtually and / or augmented. This response, along with the corresponding data, can be stored in one or more neural networks.

[0060] For all embodiments of system 100, a monitoring system could be implemented. At least part of the monitoring system can be provided in a portable device such as a mobile network device (e.g., a mobile phone, a laptop computer, one or more portable network-connected devices (including augmented reality and / or virtual reality devices), network-connected wearable clothing, and / or any combination thereof). It is conceivable that detection and comparison steps could be performed iteratively.

[0061] The terms "at least one" and "one or more" are used interchangeably. The ranges presented as being "between a and b" encompass the values ​​"a" and "b".

[0062] Although particular embodiments of the disclosed system have been illustrated and described, it will be understood that various changes, additions, and modifications can be made without departing from the spirit or scope of this disclosure. Therefore, no limitations should be imposed on the scope of the invention described except those set forth in the appended claims.

Claims

Claims

1. A detection and classification system (100) which carries out a method of detecting and classifying anomalies from images of tire surfaces, characterized in that the system comprises: - one or more tire image acquisition systems allowing the evaluation of a state of the entire surface of an imaged tire; - at least one communication network that manages the incoming data to the system (100), wherein each communication network comprises one or more communication servers each comprising one or more processors operatively connected to a memory configured to store an image processing module of the processor, which analyzes the images obtained from the image acquisition systems to determine the presence and positioning of each visible anomaly on the surface of an identified imaged tire, wherein each processor comprises an analysis application execution module that performs the processing of the obtained images and each processor is capable of executing programmed instructions stored in the memory to perform the steps of the detection and classification method; - at least one database (100) in which the images obtained from the tire image acquisition systems are stored; - at least one database (114) in which metadata is stored that describes aspects of each imaged tire; wherein the images obtained from the tire image acquisition systems are formed of pixels associated with the parameters and positioning of the anomalies, these images being transferred and stored as captured images in the memory of the inspection system (100); - an anomaly detection network that uses a Mask R-CNN with two additional branches including: - a Mask-IOU branch to predict the Intersection-over-Union (loU) between a predicted mask and its ground truth mask; and - a Bbox-IOU branch to predict the IoU between a predicted bounding box (Bbox) and its surrounding box (Bbox); in which: - the Mask R-CNN weights are pre-trained; - a neural network is then globally trained with all the images obtained at a fixed ratio and scale; and - a zone-based adjustment strategy is applied with respect to the ratio and size of the original images; so that the system (100) adapts to the images obtained from the tires and locates anomalies on their surfaces.

2. The system (100) of claim 1, wherein the weights of the Mask R-CNN are pre-trained on the COCO dataset

3. The system (100) of claim 1 or claim 2, wherein the system 100 is configured to extract the following features: - Coordinates of the anomaly bounding box; - The height and width of the anomaly bounding box; - The class probability of the classification branch; - The predictions of the mask utility index and the bounding box utility index; - The anomaly surface calculated on the anomaly mask; and - The metadata information of the imaged tire.

4. The system (100) of claim 3, wherein, for training the network, the total loss is a weighting of the individual losses and is obtained by employing a mathematical model of the type: [Math 1] ^Total~ ^1 ' L-;wæ + X2 ' + À3 ' Ljs+ X4 • Lreg + Xj • L]oUbt,„n..( + X6 ■ Where: - L ck represents the RPN classification loss which is a cross-entropy loss for classifying anchors between background and objects; - Ljpnreg represents the regression loss which is the regression of anchor localization; - Lck represents the faster classification loss which is a cross-entropy loss for classifying bounding boxes based on their classes; - Lreg represents the faster regression loss which is the same as the regression loss except that the loss is applied on a sampled set of regions of interest; - Llou_bbox represents the mean square error loss (or “MSE loss”) between the target IoU (i.e., between the predicted bounding boxes and the target bounding boxes) and the predicted IoU; and. - Liou_mask is exactly the same as for LloU bbox except that the utility index is calculated per pixel inside each bounding box.

5. The system (100) of claim 4, wherein the system (100) comprises a deep learning-based severity classification network, wherein the classification network comprises: - at least one multi-layer perceptron (MLP) for extracting features of detected anomalies; - a feature extraction method using a Resnetl8 encoder and a gated recurrent unit (GRU); and - a network using an MLP and an attention mechanism for fusing all extracted features.

6. The system (100) of claim 5, wherein the system (100) incorporates a Resnetl8 encoder driven per imaged tire area; such that this training performs a classification of anomalies according to their severity.

7. The system (100) of claim 6, wherein the memory, by executing the instructions of the image processing module in order to determine one or more parameters of each imaged tire, executes a method for detecting and classifying anomalies comprising the following steps: - a step of acquiring data corresponding to the imaged tires; - a step of training a network for detecting objects, during which a neural detection network is trained on images obtained during the previous acquisition step, and during which the network for detecting objects predicts, for each image, boxes containing the potential zones of anomalies with the associated type of anomaly and its associated confidence;- a step comprising a combination of a step of transforming the outputs of the network for object detection and a step of integrating the metadata, during which, before training a second network using the outputs of the first network, the outputs of the network for object detection are formatted to make them compatible with a binary classification machine learning algorithm, and during which the metadata are also integrated; and - a step of training the second network to predict the severity; of a detected anomaly.

8. The system (100) of any one of claims 1 to 7, wherein the tire image acquisition system(s) provide stereo-photometric images.

9. The system (100) of any one of claims 1 to 8, wherein the system (100) further comprises a detection system for collecting information about the physical environment around the system (100), the detection system comprising an illumination source having one or more illuminations whose illuminations are encoded in the detection system and / or pre-coded during a training process of a neural network.

10. A tire production site, comprising the system (100) of any one of claims 1 to 9.

Images