Silicon steel sheet rolling surface multi-type defect recognition method and system based on computer vision
By constructing a digital twin model of silicon steel sheets and using self-supervised learning, high-fidelity defect samples are generated. Combining self-supervised and semi-supervised learning, the problems of scarce samples and large labeling errors in the identification of surface defects in silicon steel sheets are solved, thereby improving the model's recognition accuracy and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING YONG ZHAN ELECTRIC TECH
- Filing Date
- 2025-10-16
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, the identification of surface defects in silicon steel sheet rolling suffers from problems such as a scarcity of real defect samples, high cost and large labeling errors in manual annotation, and poor adaptability of synthetic samples to real-world scenarios, resulting in insufficient model recognition accuracy.
A digital twin model of silicon steel sheet was constructed for physical simulation to generate defect samples. By combining self-supervised learning and semi-supervised learning, and through pre-training with unlabeled data and optimization with pseudo-labels, the simulation and real samples were integrated for incremental training to improve the model's generalization ability.
It solves the problems of scarce real defect samples and high cost of manual annotation, improves the model's generalization ability and recognition accuracy, and adapts to changes in different rolling conditions.
Smart Images
Figure CN121302897B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of material surface defect identification, specifically to a computer vision-based method and system for identifying multiple types of defects on the rolled surface of silicon steel sheets. Background Technology
[0002] In the field of surface defect identification in silicon steel sheet rolling, with the increasing demands for product quality in industrial production, accurate identification of surface defects in silicon steel sheet rolling has become increasingly important. Effective defect identification can ensure product quality, reduce the defect rate, improve production efficiency, and lower production costs, thus playing a significant role in promoting the development of the entire steel industry.
[0003] In existing technologies, various methods are commonly used to solve the problem of identifying surface defects in rolled silicon steel sheets. On one hand, defects are labeled manually using equipment such as electron microscopes to train the identification model. This method requires professionals with extensive experience and expertise to carefully observe and analyze the surface of the silicon steel sheet to determine the type and location of defects. On the other hand, simple data augmentation techniques are employed, increasing the number of samples by rotating, scaling, and flipping existing samples. Simultaneously, traditional semi-supervised learning pseudo-label generation mechanisms are widely used, utilizing the model to predict unlabeled samples and generate pseudo-labels for model training.
[0004] However, existing technologies have significant drawbacks. During silicon steel sheet rolling, the defect incidence rate is extremely low, and the types of defects are diverse. Different defects exhibit significant differences in morphology, size, and cause, resulting in extremely long collection cycles for high-quality defect samples in real-world scenarios. Real-world defect samples are scarce, and simple data augmentation techniques are insufficient to generate defect features consistent with real-world industrial scenarios. Defect labeling relies on manual labor, which is not only extremely costly but also prone to significant labeling errors due to subjective judgment bias. Traditional semi-supervised learning pseudo-label generation mechanisms lack consideration for the physical characteristics of defects, easily introducing noisy samples, leading to insufficient model generalization ability, difficulty adapting to the detection needs of different production lines, and increased false negative rates in identifying minute defects, severely impacting product quality control. Summary of the Invention
[0005] To address the issues of scarce real defect samples, high cost and large annotation errors in the identification of surface defects on rolled silicon steel sheets, as well as the poor adaptability of existing synthetic samples to real-world scenarios leading to insufficient model recognition accuracy, this application provides a computer vision-based method and system for identifying multiple types of defects on the surface of rolled silicon steel sheets.
[0006] In a first aspect, this application provides a computer vision-based method for identifying multiple types of defects on the rolled surface of silicon steel sheets, including:
[0007] A digital twin model of silicon steel sheet is constructed and its physical properties are defined. Based on the constructed digital twin model, a physical simulation of silicon steel sheet rolling is performed to generate defect samples. The physical simulation includes: replicating the three-dimensional morphology of real defects to form a parameterized library; simulating the rolling environment lighting conditions by building a dynamic lighting simulation model; adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise, and vibration blur interference; setting rolling condition parameters and simulating changes in rolling conditions to generate defect samples containing defect morphology, lighting conditions, and dynamic interference.
[0008] Collect real normal samples, and use the simulated defective samples and real normal samples as unlabeled training data. Perform self-supervised pre-training using the unlabeled training data to train and obtain the initial defect recognition model.
[0009] Collect real samples and partially label them; use some simulated defect samples and labeled real samples as labeled training data, and optimize the trained initial defect recognition model using the labeled training data to obtain an optimized initial defect recognition model; use the optimized defect recognition model to predict unlabeled real samples and generate pseudo-labels; filter, correct and visually verify the pseudo-labels, and merge the corrected and verified pseudo-label samples with the labeled real samples to form an enhanced training sample set;
[0010] The simulation-generated defect samples and the enhanced training sample set are integrated to obtain hybrid samples. The hybrid samples are used for incremental training and model iteration to obtain the final defect recognition model to complete the recognition of multiple types of defects on the rolled surface of silicon steel sheets.
[0011] By adopting the above scheme, a digital twin model of silicon steel sheet was constructed and physical simulation was performed to generate defect samples, thus solving the problem of the scarcity of real defect samples. Based on self-supervised learning and unlabeled data pre-training, the model's general feature extraction ability was cultivated, reducing the dependence on labeled defect samples. Pseudo-label optimization and cleaning in semi-supervised learning were used to generate a high-quality augmented dataset. The samples were integrated for incremental training and model iteration to obtain the final defect recognition model, improving the model's generalization ability and defect recognition accuracy.
[0012] Preferably, the construction of the silicon steel sheet digital twin model includes:
[0013] A life-size digital twin model of silicon steel sheet was built using the Blender 3D modeling tool;
[0014] Set physical property parameters, including: iron loss value and magnetic permeability; associate physical properties with rolling process parameters to complete the physical property mapping, including: designing iron loss value and magnetic permeability as dynamic parameters, which are adjusted in real time according to rolling conditions;
[0015] A surface reflection model is constructed based on BRDF, including: calculating the surface reflection characteristics using the bidirectional reflection distribution function to reproduce the reflection characteristics in a real rolling scenario.
[0016] By adopting the above approach, using Blender to build a scaled model, set and map physical properties, and construct a surface reflection model, the real scene of silicon steel sheet rolling can be reproduced more accurately, improving the fit between the synthetic sample and the real scene, and providing a more reliable data foundation for subsequent model training.
[0017] Preferably, the step of adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise, and vibration blurring interference includes:
[0018] A simulation model of the oil contamination layer was constructed based on a fluid dynamics model, including: establishing a multi-factor mapping of oil contamination viscosity based on the Arrhenius equation to obtain the correlation between viscosity and temperature and composition; considering the combined effect of rolling speed and pressure, calculating the resultant force on oil contamination particles; discretizing the oil contamination into several particles and updating their positions using the Euler method; and adjusting particle behavior for different defect types.
[0019] A surface texture noise model is constructed based on the composite texture model of rolling roll features, including: obtaining the surface features of the rolling roll through 3D scanning to obtain the basic texture; simulating the texture superposition effect of multi-pass rolling and obtaining the composite texture by weighted summation of each pass; considering the silicon steel sheet material, adjusting the indentation depth and grayscale fluctuation of the composite texture; and performing time-domain analysis and time-frequency analysis on the adjusted composite texture to construct the surface texture noise model.
[0020] The vibration fuzzy model is constructed based on the dynamic fuzzy kernel model of time-varying vibration parameters, including: obtaining time-varying parameters through Fourier transform analysis of vibration sensor data; directly associating the fuzzy kernel shape with the time-varying parameters; and adjusting the fuzzy kernel for different defect types.
[0021] The coupling relationship between the interferences of the constructed oil layer simulation model, surface texture noise model, and vibration fuzzy model is statistically analyzed; the defect morphology parameters are initialized, the rolling condition parameters are determined and the rolling condition changes are simulated; the oil layer thickness distribution, texture noise and vibration fuzzy interference are calculated and output based on the constructed oil layer simulation model, surface texture noise model and vibration fuzzy model, and the coupling rendering is superimposed based on the coupling relationship between the interferences.
[0022] By adopting the above scheme, the oil stain layer, texture noise and vibration fuzzing interference in the rolling scenario are accurately simulated. The influence of multiple factors and different defect types on the interference is considered, and the coupling relationship between the interferences is handled. High-fidelity synthetic samples that are more in line with the actual rolling scenario are generated, which improves the recognition accuracy of subsequent models for multiple types of defects on the surface of silicon steel sheets.
[0023] Preferably, the self-supervised pre-training using unlabeled training data includes:
[0024] A self-supervised pre-training task set is constructed, including: a defect region jigsaw puzzle task, a rotation angle prediction task, and a texture feature reconstruction task. The defect region jigsaw puzzle task involves randomly cropping a normal sample into multiple sub-blocks and replacing one of these sub-blocks with a defective sub-block, training to predict the location and type of the defective sub-block. The rotation angle prediction task involves randomly rotating the sample at different angles, training to recognize the rotation angle. The texture feature reconstruction task uses an encoder-decoder structure to reconstruct a clear surface texture from noisy samples.
[0025] Select a basic backbone network and pre-train it on a self-supervised pre-training task set. Use a cosine annealing strategy to decay the learning rate until the reconstruction error on the validation set reaches a preset standard.
[0026] By adopting the above scheme, a self-supervised pre-training task set is constructed for tasks including defect region puzzle, rotation angle prediction and texture feature reconstruction. This enables the model to learn the general visual features and structural patterns of silicon steel sheet surfaces from a large number of unlabeled samples, reducing the dependence on labeled defect samples and laying the foundation for subsequent accurate defect identification.
[0027] Preferably, the self-supervised pre-training using unlabeled training data further includes:
[0028] Pre-training will be performed on a set of self-supervised pre-training tasks as a preliminary training process. Based on the pre-trained model, multi-stage tasks will be designed to determine the data generation parameters, self-supervised tasks, and stage objectives for each stage task, thus completing the self-supervised pre-training based on multi-stage tasks. The multi-stage task learning includes: interference-free ideal stage tasks, single-interference stage tasks, multi-variable interference collaborative stage tasks, and real sample collaborative stage tasks.
[0029] By adopting the above scheme, multi-stage self-supervised pre-training of tasks is carried out on the basis of the pre-trained model, allowing the model to gradually learn the characteristics and patterns of silicon steel sheet surface under various conditions such as no interference, single interference, multivariate interference, and collaboration with real samples. This avoids training collapse due to excessive sample differences, effectively achieving a seamless connection from single simulation pre-training to collaborative training with real data, and ultimately improving the accuracy and robustness of silicon steel sheet rolling surface defect identification.
[0030] Preferably, the filtering, correction, and visual verification of pseudo-labels includes:
[0031] The process of filtering and correcting pseudo-labels includes: filtering pseudo-label samples with a confidence level not lower than a preset confidence threshold based on the predicted confidence level, identifying them as high-confidence pseudo-label samples, and obtaining the remaining medium- and low-confidence pseudo-label samples accordingly; and using the K-means clustering algorithm to perform feature clustering on the medium- and low-confidence pseudo-label samples and correcting the label type.
[0032] The steps for visually verifying pseudo-labels include: introducing Grad-CAM visualization technology to visualize the decision-making basis of the screened pseudo-label samples and locating the key areas identified by the model; comparing the spatial overlap rate between the key areas identified by the model and the actual suspected defect areas of the samples, and removing labels with a spatial overlap rate lower than a preset threshold; the actual suspected defect areas of the samples refer to the expected defect areas based on historical rolling process experience for unlabeled real samples.
[0033] By adopting the above scheme, feature clustering is performed on pseudo-label samples with medium and low confidence, and the label type is corrected to improve the reliability of labels for samples with medium and low confidence. The spatial overlap rate between the key area identified by the model and the actual suspected defect area of the sample is compared and labels below the preset threshold are removed to reduce the interference of noise pseudo-labels, improve the quality of samples used for model training, and thus improve the recognition accuracy and generalization ability of the final defect recognition model.
[0034] Preferably, the step of integrating the defect samples generated by the simulation and the enhanced training sample set to obtain hybrid samples, and using the hybrid samples for incremental training and model iteration, includes:
[0035] The simulated defect samples and the enhanced training sample set are integrated proportionally to obtain a hybrid sample; the hybrid sample is processed a second time using enhancement techniques; incremental training and model iteration are carried out using the hybrid sample machine after secondary processing; a dynamic learning rate is set during training, the learning rate is reduced every few rounds, and a verification is performed every few rounds. The verification set uses independently collected real defect samples. If the accuracy improvement of two consecutive verifications does not reach the preset standard, training is stopped.
[0036] By adopting the above scheme, the sample diversity is expanded, enabling the model to dynamically adjust the learning rate and stop training in a timely manner based on the validation results during the training process, thus ensuring the model training effect and improving the model's recognition accuracy and stability for various types of defects on the rolled surface of silicon steel sheets.
[0037] Secondly, this application provides a computer vision-based system for identifying multiple types of defects on the rolled surface of silicon steel sheets, including:
[0038] A silicon steel sheet rolling simulation module is used to construct a digital twin model of silicon steel sheets and define their physical properties; based on the constructed digital twin model, it performs physical simulation of silicon steel sheet rolling and generates defect samples; the physical simulation includes: replicating the three-dimensional morphology of real defects to form a parameterized library; simulating the rolling environment lighting conditions by building a dynamic lighting simulation model; adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise, and vibration blur interference; setting rolling condition parameters and simulating changes in rolling conditions to generate defect samples containing defect morphology, lighting conditions, and dynamic interference;
[0039] The defect self-supervised learning module is used to collect real normal samples. The simulated defect samples and real normal samples are used as unlabeled training data. Self-supervised pre-training is performed using the unlabeled training data to train and obtain the initial defect recognition model.
[0040] The semi-self-supervised learning module for defects is used to collect real samples for partial annotation. Partially simulated defect samples and annotated real samples are used as labeled training data. The trained initial defect recognition model is optimized using this labeled training data to obtain an optimized initial defect recognition model. The optimized defect recognition model is then used to predict unlabeled real samples, generating pseudo-labels. The pseudo-labels are then filtered, corrected, and visually verified. Finally, the corrected and verified pseudo-label samples are merged with the labeled real samples to form an enhanced training sample set.
[0041] The silicon steel sheet surface defect identification module is used to integrate the defect samples generated by simulation and the enhanced training sample set to obtain mixed samples. The mixed samples are used for incremental training and model iteration to obtain the final defect identification model to complete the identification of multiple types of defects on the rolled surface of silicon steel sheets.
[0042] By adopting the above scheme, the problems of scarce real defect samples, high cost of manual annotation and large annotation error in the identification of surface defects in silicon steel sheet rolling are effectively solved, the model's adaptability to different rolling conditions and environments is enhanced, and the model's recognition accuracy and generalization ability are further improved.
[0043] Thirdly, this application provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the method described above.
[0044] Fourthly, this application provides a computer device, the computer device including a memory, a processor and a program stored in the memory and executable thereon, the program being executed by the processor to implement the steps of the method described above.
[0045] In summary, this application has the following beneficial effects:
[0046] 1. By constructing a digital twin model of silicon steel sheets and generating defect samples through physical simulation, the problem of scarce real defect samples is solved. In addition, by combining illumination simulation and dynamic interference simulation of rolling scene, high-fidelity simulated defect samples containing defect morphology, illumination conditions and environmental interference are generated, improving the adaptability of simulated defect samples to real scene. Pseudo-label optimization and cleaning are carried out by self-supervised pre-training and semi-supervised learning, which reduces the dependence on labeled defect samples, reduces the cost of manual labeling and pseudo-label error, and enhances the generalization ability and recognition accuracy of the model. Incremental training and model iteration with mixed samples are carried out to ensure that the model can continuously adapt to the differences in defect features brought about by process changes.
[0047] 2. Construct a self-supervised pre-training task set for tasks including defect region puzzle-making, rotation angle prediction, and texture feature reconstruction. Utilize the self-supervised pre-training task set to learn the general visual features and structural patterns of silicon steel sheet surfaces, reducing reliance on labeled defect samples. The designed multi-stage tasks, through a progressive adaptation framework, enable the model to gradually transition from learning ideal simulation features to learning real complex features, avoiding poor generalization ability and training instability caused by direct transfer.
[0048] 3. Based on the predicted confidence level, high-confidence pseudo-label samples are selected. The K-means clustering algorithm is used to perform feature clustering on medium- and low-confidence pseudo-label samples and correct the label type, thereby improving the reliability of labels for medium- and low-confidence samples. Grad-CAM visualization technology is introduced to visualize the decision basis of the selected pseudo-label samples, locate the key areas identified by the model, compare the spatial overlap rate between the key areas identified by the model and the actual suspected defect areas of the samples, and remove labels below the preset threshold, effectively removing noisy pseudo-labels and improving the reliability and quality of pseudo-labels. Attached Figure Description
[0049] Figure 1 This is a flowchart of the computer vision-based method for identifying multiple types of defects on the rolled surface of silicon steel sheets, as described in a specific embodiment.
[0050] Figure 2 This is a schematic diagram of the structure of the computer vision-based silicon steel sheet rolling surface multi-type defect recognition system described in a specific embodiment. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0052] like Figure 1As shown in the embodiments of this application, a method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision is disclosed. The method includes steps such as constructing a digital twin model of silicon steel sheets, generating defect samples through physical simulation, self-supervised pre-training, semi-supervised learning optimization, and model iteration. Through the cooperation of these steps, the problems of scarce samples, large annotation errors, and insufficient model recognition accuracy in the identification of defects on the rolled surface of silicon steel sheets are solved. Each step will be described in detail below.
[0053] S1. Construct a digital twin model of silicon steel sheet and define its physical properties; perform physical simulation of silicon steel sheet rolling based on the constructed digital twin model of silicon steel sheet and generate defect samples.
[0054] When constructing the digital twin model of silicon steel sheets, the Blender 3D modeling tool was used to build a scaled-down digital twin model of the silicon steel sheets, including geometric parameters such as thickness, width, and surface curvature. During model construction, physical property parameters, such as iron loss and magnetic permeability, needed to be set. To more realistically simulate the rolling process, these physical properties were correlated with rolling process parameters to complete the physical property mapping. For example, iron loss and magnetic permeability could be designed as dynamic parameters, adjusted in real time according to rolling conditions. Rolling conditions include factors such as rolling speed and rolling temperature; different rolling conditions will affect the physical properties of the silicon steel sheets, and these can be adjusted in real time by analyzing the correlation curves between actual rolling conditions and physical properties.
[0055] Meanwhile, to facilitate more realistic physical simulations, a surface reflection model was constructed based on the BRDF (Bidirectional Reflectance Function), and the surface reflectivity was calculated using the bidirectional reflectance distribution function to reproduce the reflectivity in a real rolling scenario. Specifically, the silicon steel sheet surface is metallic and exhibits features such as rolling textures, oil stains, and oxide layers. Therefore, a BRDF model capable of capturing microsurface scattering and the Fresnel effect is required, which can be decomposed into diffuse reflection and specular reflection terms, including: , , In the formula, The BRDF value quantifies the reflection efficiency in different directions. The incident direction, The direction of reflection; The diffuse reflection term is used to simulate energy scattering on a rough surface. The diffuse reflection coefficient is set according to the surface material of the silicon steel sheet; For specular reflection, we simulate specular scattering from a smooth surface. Specular reflection is formed by the superposition of specular reflections from surface micro-protrusions and is influenced by the distribution of micro-surface normals, geometric occlusion, and the Fresnel effect. = The unit vector is the angle bisector of the incident and reflected directions, used to describe the orientation of the micro-surface normal. The normal distribution function of the micro-surface; It is a geometric decay function. For Fresnel functions, Let be the refractive index of the silicon steel sheet surface material; therefore, the final pixel reflected light intensity is calculated using the formula: ;in, The integral is based on all incident directions in space, taking the base of the integral and combining the incident light intensity. The reflected emissivity was calculated.
[0056] Physical simulations of silicon steel sheet rolling are performed based on a constructed digital twin model of the silicon steel sheet to generate defect samples. Specifically, the physical simulation includes several aspects. First, a parameterized library is formed by replicating the 3D morphology of real defects. For example, for typical defects such as scratches, holes, oxide scale, and roll marks, 3D morphological data of real defects are obtained through industrial CT scanning and converted into model files that can be imported into Blender, achieving accurate replication of defect morphology. Simultaneously, multi-dimensional parameter variables are set for each defect, forming a parameterized library containing various defect variants. For example, for scratch defects, parameters such as the length, width, and depth of the scratch can be set; for hole defects, parameters such as the diameter and depth of the hole can be set. Furthermore, considering different rolling conditions, the parameterized library is expanded to include various defect variants, thereby generating diverse defect samples. Secondly, a dynamic lighting simulation model was built to simulate the lighting conditions of the rolling environment. Specifically, this included simulating typical lighting scenarios such as welding arc light and workshop light fluctuations based on measured lighting distribution data from the hot rolling workshop. The model supports dynamic adjustment of lighting intensity and rotation of the lighting angle, and incorporates a temperature influence factor to simulate the shift in the reflective angle of the silicon steel sheet surface under different rolling temperatures. Then, dynamic interference simulation of the rolling scene was added, superimposed with oil contamination layers, texture noise, and vibration blur. Specifically, this included an oil contamination layer generated based on a fluid dynamics model, surface texture noise replicating the characteristics of the rolling roll pattern, and vibration blur simulating production line speeds. Finally, rolling condition parameters were set and rolling condition changes were simulated, generating highly realistic defect samples containing defect morphology and dynamic interference from lighting conditions. The sample resolution was consistent with the resolution acquired by the industrial camera.
[0057] S2. Pre-training with unlabeled data based on self-supervised learning to obtain the initial defect recognition model.
[0058] To reduce reliance on labeled defect samples and overcome the limitations of high cost and large labeling error of manual annotation, a self-supervised learning approach was chosen. By inputting simulated defect samples and a large number of unlabeled normal samples, the model's general feature extraction capability was trained, laying the foundation for accurate defect identification in the future.
[0059] Specifically, real, normal samples are collected, and the simulated defective samples and real, normal samples are used as unlabeled training data. Self-supervised pre-training is then performed using this unlabeled training data to obtain an initial defect recognition model. This self-supervised pre-training using unlabeled training data includes:
[0060] First, a self-supervised pre-training task set is constructed to enable the model to learn the general visual features and structural patterns of silicon steel sheet surfaces from a large number of unlabeled samples. Specifically, this includes: a defect region jigsaw puzzle task, a rotation angle prediction task, and a texture feature reconstruction task. The defect region jigsaw puzzle task involves randomly cropping normal samples into multiple sub-blocks and replacing one of these sub-blocks with a defective one, training the model to predict the location and type of the defective sub-block, thereby learning the relationship between local and global aspects of the image. The rotation angle prediction task involves randomly rotating samples at different angles and training the model to recognize these rotation angles, enabling the model to learn the global structure of the image and laying the foundation for subsequent defect localization. The texture feature reconstruction task uses an encoder-decoder structure to reconstruct clear surface textures from noisy samples, allowing the model to grasp the distribution patterns of normal textures and providing a benchmark for distinguishing between defects and normal textures.
[0061] Secondly, a basic backbone network, such as ResNet-50, is selected and pre-trained on a self-supervised pre-training task set. During pre-training, multi-task joint training is adopted, sharing a basic feature extractor (such as the first n stages of ResNet-50), with each task having an independent prediction head. The loss function is weighted and fused. A cosine annealing strategy is used to decay the learning rate until the reconstruction error on the validation set (such as using PSNR and SSIM to evaluate the texture reconstruction quality) reaches the preset standard. For example, the initial learning rate is set to 0.001, and the cosine annealing strategy is used to periodically adjust it (the number of periods is 50).
[0062] S3. Train and optimize unlabeled data based on semi-supervised learning to obtain an initial defect identification optimization model; use the obtained initial defect identification optimization model to predict pseudo-labels and optimize and clean the generated pseudo-labels to construct an enhanced training sample set.
[0063] Specifically, to further ensure the accuracy of the initial defect identification model, a semi-supervised learning approach is adopted to optimize the model while reducing the cost of manual annotation. First, real samples are collected and partially annotated to generate a high-quality augmented dataset. Then, some simulated defect samples and some real samples are used as labeled training data. The trained initial defect identification model is optimized using this labeled training data to obtain an optimized initial defect identification model. During further training using the labeled training data, a weighted sum of cross-entropy loss and Focal loss functions is used as the loss function to address the imbalanced sample problem.
[0064] Secondly, using a defect identification optimization model, predictions are made for unlabeled real samples to generate pseudo-labels containing defect location, type, and confidence level. The pseudo-labels are then screened, corrected, and visually verified. Simultaneously, samples are randomly selected for manual verification, and the types and locations of manually corrected label errors are recorded. An error correction database is then constructed to optimize the subsequent pseudo-label generation logic.
[0065] The process of filtering, correcting, and visually verifying pseudo-labels includes: filtering and correcting pseudo-label samples with a predicted confidence level not lower than a preset confidence threshold, identifying them as high-confidence pseudo-label samples; and identifying pseudo-label samples with a confidence level lower than the preset threshold as medium-to-low-confidence pseudo-label samples. To further improve the label reliability of medium-to-low-confidence samples, a K-means clustering algorithm (or an HDBSCAN algorithm) is used to perform feature clustering and correct the label types of the medium-to-low-confidence pseudo-label samples. Specifically, for the selected valid clusters, the majority voting method is used to correct the pseudo-labels of all samples within the cluster. The original pseudo-label types of all samples within the cluster are counted, and the label type with the most occurrences is selected as the consensus label for the cluster. The pseudo-labels of all samples within the cluster are then uniformly corrected to the consensus label. The confidence level of the corrected label is updated as: number of samples within the cluster / total number of samples. For samples marked as outliers after clustering, the labels are not corrected temporarily; they are marked as samples awaiting review and can be processed later with manual annotation.
[0066] The steps for visually verifying pseudo-labels include: introducing Grad-CAM visualization technology to visualize the decision-making basis of the screened pseudo-label samples and locate the key areas identified by the model; comparing the spatial overlap rate between the key areas identified by the model and the actual suspected defect areas of the samples, and removing labels with a spatial overlap rate lower than a preset threshold; the actual suspected defect areas of the samples refer to the expected defect areas based on historical rolling process experience for unlabeled real samples.
[0067] Finally, the corrected and verified pseudo-labeled samples will be merged with the labeled real samples to form an enhanced training sample set.
[0068] S4. Integrate the simulation-generated defect samples and the enhanced training sample set to obtain mixed samples. Use the mixed samples for incremental training and model iteration to obtain the final defect recognition model to complete the recognition of multiple types of defects on the rolled surface of silicon steel sheets.
[0069] Specifically, the simulated defect samples and the enhanced training sample set are integrated proportionally to obtain hybrid samples, completing the final optimization and continuous adaptation of the defect recognition model. Enhancement techniques are used to perform secondary processing on the hybrid samples, such as random erasure, color jittering, and Gaussian noise addition, to further expand sample diversity. The hybrid samples after secondary processing are used for incremental training and model iteration. YOLOv8x can be selected as the target detection model, with incremental training using hybrid samples. A dynamic learning rate is set during training, decreasing every few rounds, and validation is performed every few rounds. The validation set uses independently collected real defect samples. If the accuracy improvement after two consecutive validations does not reach the preset standard, training is stopped. Furthermore, an online model iteration mechanism is established, collecting false positives and false negatives from production line inspection daily, manually labeling them, and adding them to the training dataset. Regular incremental fine-tuning is performed to ensure the model can continuously adapt to the differences in defect characteristics caused by changes in production line processes.
[0070] By adopting the above method, the problems of scarce samples, large labeling errors, and insufficient model recognition accuracy in the identification of surface defects in silicon steel sheet rolling can be effectively solved, thereby improving the product quality control effect.
[0071] In a specific embodiment, to more accurately simulate dynamic disturbances in rolling scenarios, make the generated defect samples closer to real rolling scenarios, improve the realism and diversity of the samples, and thus enhance the model's ability to identify defects under different rolling scenarios, this method further includes:
[0072] Specifically, an oil sludge simulation model is constructed based on a fluid dynamics model, including: establishing a multi-factor mapping of oil sludge viscosity based on the Arrhenius equation, and obtaining the correlation between viscosity and temperature and composition; the formula is: In the formula, This represents a multi-factor mapping of oil viscosity; T is the rolling temperature, and c is the additive concentration. The viscosity of pure rolling oil at 25℃ is... Based on experimental fitting, a value of 18000 J / mol was used; the viscosity of the oil contaminant could be calculated under the current rolling conditions. Considering the combined effects of rolling speed and pressure, the resultant force on the oil contaminant particles, including the shear force along the rolling direction, was calculated. ( For rolling speed, For oil particle velocity, (Initial thickness of oil contamination), pressure gradient force along the perpendicular rolling direction ( For rolling pressure, (where the area is the oil stain particle area); the oil stain is discretized into several particles, and the position is updated by the Euler method; the particle behavior is adjusted for different defect types, including: for dent defects: surface tension causes the particles to gather towards the center of the dent, and the aggregation density is proportional to the depth, such as when the depth is 0.5mm, the density increases by 50%; for scratch defects: the particles are stretched along the scratch direction, and the length is increased by 20% compared with the substrate.
[0073] A surface texture noise model is constructed based on the composite texture model of rolling roll features, including: obtaining the surface features of the rolling roll through three-dimensional scanning to obtain the basic texture; such as: the spacing, depth and grayscale fluctuation between parallel stripes of the new roll can be simulated by a sine function; the stripe depth attenuation of the old roll compared to the new roll. The textural superposition effect of multi-pass rolling is simulated by obtaining the composite texture through weighted summation of each pass. Considering the texture variation caused by the inhomogeneity of silicon steel sheet material, the indentation depth and grayscale fluctuation of the composite texture are adjusted. For example, in high-hardness areas (HV>200): the texture indentation is shallow (depth reduced by 20%), and the grayscale fluctuation is ±5; in low-hardness areas (HV<180): the texture indentation is deep (depth increased by 20%), and the grayscale fluctuation is ±12. The adjusted composite texture is subjected to time-domain analysis (Fourier transform of the composite texture to extract frequency domain features and construct a frequency domain noise model) and time-frequency analysis (Short-Time Fourier Transform (STFT) or wavelet transform to analyze the time-varying characteristics of texture noise) to construct a surface texture noise model (by adjusting the frequency domain parameters and time-varying characteristics, the noise distribution under different working conditions is simulated, and the dynamic changes of noise during the rolling process are simulated).
[0074] A vibration fuzzy model is constructed based on a dynamic fuzzy kernel model of time-varying vibration parameters, including: obtaining time-varying parameters, including amplitude, through Fourier transform analysis of vibration sensor data. ,frequency And direction; directly link the fuzzy kernel shape with time-varying parameters, such as: the kernel function formula is: In the formula, ; When the blurring occurs along the x-axis, the blur kernel extends along the x-axis; the blur kernel is adjusted for different defect types, such as increasing the length of the edge blur kernel for defects like depressions or scabs. For crack defect types, the width of the fuzzy kernel increases by 20% along the crack direction.
[0075] The statistical analysis revealed the inter-interference coupling relationships among the constructed oil stain layer simulation model, surface texture noise model, and vibration blur model. These included: the coupling relationship between oil stains and texture: the texture grayscale of the oil stain-covered area decreased by a%, and the angle between the oil stain flow direction and the texture direction determined the occlusion mode; the coupling relationship between oil stains and vibration blur: vibration blur acted on both the oil stain and the substrate, and the blur degree of the oil stain edge was b% higher than that of the substrate. In addition, the coupling between illumination and all interference models could be further considered to determine the coupling relationship between illumination and each interference model. For example, under strong light (>8000 lux), the contrast between oil stains and texture decreased by c%, and the edge diffusion of vibration blur was more obvious, i.e., the grayscale transition area became wider by d.
[0076] Initialize defect morphology parameters, determine rolling condition parameters and their changes, and calculate and output oil layer thickness distribution, texture noise and vibration fuzz interference based on the constructed oil layer simulation model, surface texture noise model and vibration fuzz model. Coupled rendering and superposition are performed based on the coupling relationship between interferences (combined with the simulated light intensity) to add texture noise similar to the real silicon steel sheet surface, such as scratches and oxide spots, to the simulation image. This simulates the distribution and penetration effect of different types and concentrations of oil on the silicon steel sheet surface, including the viscosity, transparency and reflectivity of the oil. It can also perform time-series evolution and calibration according to changes in working conditions (comparing with real samples and adjusting the parameters of each model).
[0077] In a specific embodiment, to better cope with the complex and ever-changing actual production environment, thereby further improving the model's recognition accuracy and performance, and better solving the problem of identifying surface defects in silicon steel sheet rolling; the method further includes: the self-supervised pre-training using unlabeled training data also includes:
[0078] Pre-training will be performed on a set of self-supervised pre-training tasks as a preliminary training process. Based on the pre-trained model, multi-stage tasks will be designed to determine the data generation parameters, self-supervised tasks, and stage objectives for each stage task. This will complete the self-supervised pre-training based on multi-stage tasks, thereby reducing the impact of differences in sample defect distribution, improving model stability, and achieving a smooth transition.
[0079] Specifically, the multi-stage task learning includes: an interference-free ideal stage task, a single-interference stage task, a multi-variable interference collaborative stage task, and a real-sample collaborative stage task.
[0080] For the interference-free ideal stage task, a training phase is completed under ideal conditions without interference. Data generation parameters include: single-type defects generated through simulation, and lighting parameters without interference; self-supervised tasks include: defect region mask prediction (predicting whether a random mask region is a defect) and defect type contrastive learning (generating positive sample pairs for the same defect type and negative samples for other types); stage objective: accurately distinguishing defects from normal textures under interference-free conditions, with the defect region prediction confidence exceeding a pre-set confidence threshold; the loss function is set as contrastive loss.
[0081] For the single-perturbation stage task, learn the invariance of defect features under a single perturbation. Data parameters include: a single type of defect generated through simulation, lighting parameters, and a single perturbation (e.g., oil stains); self-supervised tasks include: perturbation robust contrastive learning (generating two samples for the same defect: one without perturbation and one with single perturbation, which must be identified as the same defect), and defect edge regression (predicting the precise edge coordinates of the defect); the stage objective is: under single perturbation, the defect type identification accuracy should only decrease by a preset accuracy percentage compared to the previous stage, such as 5%; the loss function is set to MSE loss.
[0082] For the multi-variable interference collaborative stage task, defect learning under multi-interference collaborative conditions is completed. Data parameters include: multiple types of defects generated through simulation, illumination parameters that conform to interference (such as oil stains, rolling direction vibration blur); self-supervised tasks include: defect component segmentation and cross-scale feature consistency; stage objectives include: the confidence level of subtype segmentation of composite defects is greater than a preset confidence threshold, and the defect type identification accuracy only decreases by a preset accuracy ratio compared to the previous stage; the loss function is multi-scale contrastive loss.
[0083] For the real-sample collaboration stage, defect learning is achieved through simulation and real-sample collaboration. Data parameters include: real sample parameters and a mixture of parameters generated from simulation in the previous stage; self-supervised tasks include: mixed data contrastive learning (cross-domain comparison of mixed data, aligning simulation and real image features for the same defect type) and pseudo-label self-training (generating pseudo-labels for real data, selecting samples with confidence scores greater than a pre-set confidence threshold for training); stage objective: fully adapting to real data features, with pre-trained features directly usable for optimizing real-sample defect identification; loss functions are cross-domain contrastive loss and pseudo-label classification loss.
[0084] like Figure 2 As shown, this embodiment discloses a computer vision-based system for identifying multiple types of defects on the rolled surface of silicon steel sheets, specifically including:
[0085] The silicon steel sheet rolling simulation module 101 is used to construct a digital twin model of the silicon steel sheet and define its physical properties; perform physical simulation of silicon steel sheet rolling based on the constructed digital twin model, and generate defect samples; the physical simulation includes: replicating the three-dimensional morphology of real defects to form a parameterized library; simulating the rolling environment lighting conditions by building a dynamic lighting simulation model; adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise and vibration blur interference; setting rolling condition parameters and simulating changes in rolling conditions to generate defect samples containing defect morphology, lighting conditions and dynamic interference;
[0086] The defect self-supervised learning module 102 is used to collect real normal samples, and uses the simulated defect samples and real normal samples as unlabeled training data. Self-supervised pre-training is performed using the unlabeled training data to train and obtain the initial defect recognition model.
[0087] The semi-self-supervised learning module 103 for defects is used to collect real samples for partial annotation; the partially simulated defect samples and the partially annotated real samples are used as labeled training data, and the trained initial defect recognition model is optimized using the labeled training data to obtain an optimized initial defect recognition model; the optimized defect recognition model is used to predict the unlabeled real samples to generate pseudo-labels; the pseudo-labels are screened, corrected and visualized for verification, and the corrected and verified pseudo-label samples are merged with the labeled real samples to form an enhanced training sample set;
[0088] The silicon steel sheet surface defect recognition module 104 is used to integrate the defect samples generated by simulation and the enhanced training sample set to obtain mixed samples. The mixed samples are used for incremental training and model iteration to obtain the final defect recognition model to complete the recognition of multiple types of defects on the rolled surface of silicon steel sheets.
[0089] This application also discloses a computer-readable storage medium.
[0090] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the computer vision-based method for identifying multiple types of defects on the rolled surface of silicon steel sheets described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0091] This application also discloses a computer device.
[0092] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executed to perform the aforementioned computer vision-based method for identifying multiple types of defects on the rolled surface of silicon steel sheets.
[0093] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.
Claims
1. A computer vision-based method for identifying multiple types of defects on the rolled surface of silicon steel sheets, characterized in that, include: Construct a digital twin model of silicon steel sheet and define its physical properties; Physical simulation of silicon steel sheet rolling is performed based on the constructed digital twin model of silicon steel sheet, and defect samples are generated. The physical simulation includes: replicating the three-dimensional morphology of real defects to form a parameterized library; simulating the lighting conditions of the rolling environment by building a dynamic lighting simulation model; and adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise, and vibration fuzzing interference. Specifically, an oil stain simulation model is constructed based on a fluid dynamics model, a surface texture noise model is constructed based on a composite texture model based on the characteristics of the rolling roll, and a vibration fuzzing model is constructed based on a dynamic fuzzy kernel model based on time-varying vibration parameters. The thickness distribution of the oil stains, texture noise, and vibration fuzzing interference are calculated, and coupled rendering is performed based on the coupling relationship between the interferences. Rolling condition parameters are set and rolling condition changes are simulated to generate defect samples containing defect morphology, lighting conditions, and dynamic interference. Collect real normal samples, and use the simulated defective samples and real normal samples as unlabeled training data. Perform self-supervised pre-training using the unlabeled training data to train and obtain the initial defect recognition model. Real samples are collected and partially labeled. Some simulated defect samples and labeled real samples are used as labeled training data. The trained initial defect recognition model is optimized using this labeled training data to obtain an optimized initial defect recognition model. The optimized model is then used to predict unlabeled real samples, generating pseudo-labels. The pseudo-labels are then screened, corrected, and visually verified. Specifically, high-confidence pseudo-labels are screened based on a confidence threshold, and medium- and low-confidence pseudo-labels are clustered to correct their types. Grad-CAM is then used to visually verify the screened pseudo-label samples, eliminating those with insufficient spatial overlap. The corrected and verified pseudo-label samples are then merged with the labeled real samples to form an enhanced training sample set. The simulation-generated defect samples and the enhanced training sample set are integrated to obtain hybrid samples. The hybrid samples are used for incremental training and model iteration to obtain the final defect recognition model to complete the recognition of multiple types of defects on the rolled surface of silicon steel sheets.
2. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 1, characterized in that, The construction of the silicon steel sheet digital twin model includes: A life-size digital twin model of silicon steel sheet was built using the Blender 3D modeling tool; Set physical property parameters, including: iron loss value and magnetic permeability; associate physical properties with rolling process parameters to complete the physical property mapping, including: designing iron loss value and magnetic permeability as dynamic parameters, which are adjusted in real time according to rolling conditions; A surface reflection model is constructed based on BRDF, including: calculating the surface reflection characteristics using the bidirectional reflection distribution function to reproduce the reflection characteristics in a real rolling scenario.
3. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 1, characterized in that, The method of adding dynamic interference simulation of the rolling scene, superimposed with oil stains, texture noise and vibration blur interference, includes: A simulation model of the oil contamination layer was constructed based on a fluid dynamics model, including: establishing a multi-factor mapping of oil contamination viscosity based on the Arrhenius equation to obtain the correlation between viscosity and temperature and composition; considering the combined effect of rolling speed and pressure, calculating the resultant force on oil contamination particles; discretizing the oil contamination into several particles and updating their positions using the Euler method; and adjusting particle behavior for different defect types. A surface texture noise model is constructed based on the composite texture model of rolling roll features, including: obtaining the surface features of the rolling roll through 3D scanning to obtain the basic texture; simulating the texture superposition effect of multi-pass rolling and obtaining the composite texture by weighted summation of each pass; considering the silicon steel sheet material, adjusting the indentation depth and grayscale fluctuation of the composite texture; and performing time-domain analysis and time-frequency analysis on the adjusted composite texture to construct the surface texture noise model. The vibration fuzzy model is constructed based on the dynamic fuzzy kernel model of time-varying vibration parameters, including: obtaining time-varying parameters through Fourier transform analysis of vibration sensor data; directly associating the fuzzy kernel shape with the time-varying parameters; and adjusting the fuzzy kernel for different defect types. The coupling relationship between the interferences of the constructed oil layer simulation model, surface texture noise model, and vibration fuzzy model is statistically analyzed; the defect morphology parameters are initialized, the rolling condition parameters are determined and the rolling condition changes are simulated; the oil layer thickness distribution, texture noise and vibration fuzzy interference are calculated and output based on the constructed oil layer simulation model, surface texture noise model and vibration fuzzy model, and the coupling rendering is superimposed based on the coupling relationship between the interferences.
4. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 1, characterized in that, The self-supervised pre-training using unlabeled training data includes: A self-supervised pre-training task set is constructed, including: a defect region jigsaw puzzle task, a rotation angle prediction task, and a texture feature reconstruction task. The defect region jigsaw puzzle task involves randomly cropping a normal sample into multiple sub-blocks and replacing one of these sub-blocks with a defective sub-block, training to predict the location and type of the defective sub-block. The rotation angle prediction task involves randomly rotating the sample at different angles, training to recognize the rotation angle. The texture feature reconstruction task uses an encoder-decoder structure to reconstruct a clear surface texture from noisy samples. Select a basic backbone network and pre-train it on a self-supervised pre-training task set. Use a cosine annealing strategy to decay the learning rate until the reconstruction error on the validation set reaches a preset standard.
5. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 4, characterized in that, The self-supervised pre-training using unlabeled training data also includes: Pre-training will be performed on a set of self-supervised pre-training tasks as a preliminary training process. Based on the pre-trained model, multi-stage tasks will be designed to determine the data generation parameters, self-supervised tasks, and stage objectives for each stage task, thus completing the self-supervised pre-training based on multi-stage tasks. The multi-stage tasks include: interference-free ideal stage task, single-interference stage task, multi-variable interference collaborative stage task, and real sample collaborative stage task.
6. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 1, characterized in that, The process of filtering, correcting, and visually verifying pseudo-labels includes: The process of filtering and correcting pseudo-labels includes: filtering pseudo-label samples with a confidence level not lower than a preset confidence threshold based on the predicted confidence level, identifying them as high-confidence pseudo-label samples, and obtaining the remaining medium- and low-confidence pseudo-label samples accordingly; and using the K-means clustering algorithm to perform feature clustering on the medium- and low-confidence pseudo-label samples and correcting the label type. The steps for visually verifying pseudo-labels include: introducing Grad-CAM visualization technology to visualize the decision-making basis of the screened pseudo-label samples and locating the key areas identified by the model; comparing the spatial overlap rate between the key areas identified by the model and the actual suspected defect areas of the samples, and removing labels with a spatial overlap rate lower than a preset threshold; the actual suspected defect areas of the samples refer to the expected defect areas based on historical rolling process experience for unlabeled real samples.
7. The method for identifying multiple types of defects on the rolled surface of silicon steel sheets based on computer vision according to claim 1, characterized in that, The process of integrating the defect samples generated by the simulation and the enhanced training sample set to obtain hybrid samples, and using the hybrid samples for incremental training and model iteration, includes: The simulated defect samples and the enhanced training sample set are integrated proportionally to obtain a hybrid sample; the hybrid sample is processed a second time using enhancement techniques; incremental training and model iteration are carried out using the hybrid sample machine after secondary processing; a dynamic learning rate is set during training, the learning rate is reduced every few rounds, and a verification is performed every few rounds. The verification set uses independently collected real defect samples. If the accuracy improvement of two consecutive verifications does not reach the preset standard, training is stopped.
8. A computer vision-based system for identifying multiple types of defects on the rolled surface of silicon steel sheets, characterized in that, include: The silicon steel sheet rolling simulation module is used to build a digital twin model of silicon steel sheets and define their physical properties; Physical simulation of silicon steel sheet rolling is performed based on the constructed digital twin model of silicon steel sheet to generate defect samples. The physical simulation includes: replicating the three-dimensional morphology of real defects to form a parameterized library; simulating the rolling environment lighting conditions by building a dynamic lighting simulation model; and adding dynamic interference simulation of the rolling scene, superimposing oil stains, texture noise, and vibration fuzzing interference. Specifically, an oil stain simulation model is constructed based on a fluid dynamics model, a surface texture noise model is constructed based on a composite texture model based on the characteristics of the rolling roll, and a vibration fuzzing model is constructed based on a dynamic fuzzy kernel model based on time-varying vibration parameters. The thickness distribution of the oil stains, texture noise, and vibration fuzzing interference are calculated, and coupled rendering is performed based on the coupling relationship between the interferences. Rolling condition parameters are set and rolling condition changes are simulated to generate defect samples containing defect morphology, lighting conditions, and dynamic interference. The defect self-supervised learning module is used to collect real normal samples. The simulated defect samples and real normal samples are used as unlabeled training data. Self-supervised pre-training is performed using the unlabeled training data to train and obtain the initial defect recognition model. The semi-self-supervised learning module for defects is used to collect real samples for partial annotation. Partially simulated defect samples and annotated real samples are used as labeled training data. The trained initial defect recognition model is optimized using this labeled training data to obtain an optimized initial defect recognition model. The optimized model is then used to predict unlabeled real samples, generating pseudo-labels. The pseudo-labels are then filtered, corrected, and visually verified. Specifically, high-confidence pseudo-labels are filtered based on a confidence threshold, and medium- and low-confidence pseudo-labels are clustered to correct their types. Grad-CAM is then used to visually verify the filtered pseudo-label samples, eliminating those with insufficient spatial overlap. Finally, the corrected and verified pseudo-label samples are merged with the labeled real samples to form an enhanced training sample set. The silicon steel sheet surface defect identification module is used to integrate the defect samples generated by simulation and the enhanced training sample set to obtain mixed samples. The mixed samples are used for incremental training and model iteration to obtain the final defect identification model to complete the identification of multiple types of defects on the rolled surface of silicon steel sheets.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method as described in any one of claims 1 to 7.
10. A computer device, characterized in that, The computer device includes a memory, a processor, and a program stored in and executable on the memory, the program being executed by the processor to implement the steps of the method as described in any one of claims 1 to 7.