Method for detecting quality of hardware sheet production process based on image processing

Abstract

The application aims to provide a hardware sheet material production process quality detection method based on image processing, which comprises the following steps: obtaining sensor data generated by a digital twin workshop; obtaining sheet material image data; inputting the sensor data and the sheet material image data into a sheet material quality detection model; and outputting a sheet material quality judgment result by the sheet material quality detection model. On the basis of machine vision for sheet material quality detection, the application creatively introduces a digital twin workshop to simulate sensor data, which is more convenient than installing sensors on a real production line to obtain data. Then, by constructing a sheet material quality detection model, the image data and sensor data from the sheet material are fused to comprehensively judge the sheet material quality, so that the detection accuracy is high and the detection efficiency is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a quality inspection method for the production process of hardware sheet metal based on image processing. Background Technology

[0002] Sheet metal requires real-time monitoring and inspection during production to ensure that any abnormalities on the production line are detected and resolved immediately. If sheet metal is not inspected during production, the yield rate will drop significantly, potentially leading to safety hazards and endangering consumer safety and property. Sheet metal quality inspection involves a series of scientific tests to ensure the quality and safety of the sheet metal, protecting human health and safety. These tests include, but are not limited to, the following: Appearance inspection: Visually inspecting the surface for defects such as cracks, scars, bubbles, and rust, as well as ensuring the surface is smooth and the color is uniform, to determine the appearance quality. Dimensional inspection: Measuring the length, width, thickness, and other dimensional parameters of the sheet metal to ensure it conforms to standard dimensions and avoids issues affecting performance due to non-compliance. Chemical composition analysis: Analyzing the content of various elements in the sheet metal to determine if its chemical composition meets standard requirements, which is crucial for the sheet metal's physical properties and service life. Mechanical performance testing: Testing mechanical properties such as tensile strength, yield strength, impact toughness, and hardness to ensure the sheet metal's strength, toughness, and hardness meet standard requirements, guaranteeing performance and safety. Surface quality testing: This involves inspecting the surface smoothness, flatness, roughness, and coating adhesion of the boards to ensure that the surface quality meets standard requirements, affecting the appearance and lifespan of the boards. In addition, for certain types of boards, such as engineered wood and wood-based products, further testing is required, such as formaldehyde emission testing, VOC testing, heavy metal testing, and flame retardant performance testing, to ensure that the products meet national standards and environmental protection requirements. These tests are not only related to the quality and safety of the boards but are also important measures to protect consumer rights.

[0003] Currently, the commonly used sheet metal quality inspection is production line inspection based on machine vision. This involves taking pictures of the sheet metal on the production line and then judging the quality of the sheet metal based on the acquired images. However, because the production speed on the sheet metal production line is very fast and some sheet metal is very small, it is difficult to accurately judge whether there are quality problems with the sheet metal using only machine vision. This can easily lead to false positives and false negatives, resulting in low accuracy of sheet metal quality inspection. Summary of the Invention

[0004] The purpose of this invention is to provide a quality inspection method for the production process of metal sheet metal based on image processing. This method creatively introduces a digital twin workshop to simulate sensor data on the basis of machine vision for sheet metal quality inspection. This is more convenient than installing sensors on the actual production line to obtain data. Then, by constructing a sheet metal quality inspection model, the image data and sensor data from the sheet metal are integrated to make a comprehensive judgment on the quality of the sheet metal. The detection accuracy is high and the detection efficiency can be greatly improved.

[0005] Image processing-based quality inspection methods for hardware sheet metal production processes include:

[0006] Acquire sensor data generated in the digital twin workshop;

[0007] Acquire sheet metal image data;

[0008] Input the sensor data and the sheet metal image data into the sheet metal quality inspection model;

[0009] The sheet metal quality inspection model outputs sheet metal quality judgment results.

[0010] Preferably, before inputting the sensor data and the image data into the sheet metal quality inspection model, the method further includes constructing the sheet metal quality inspection model, specifically:

[0011] Construct the input layer, hidden layer, feedback support layer, and output layer of the sheet metal quality inspection model;

[0012] Set the connection weights between the input layer and the hidden layer;

[0013] Set the hidden layer node bias and activation function;

[0014] Set the trend characteristic factor of the feedback receiving layer;

[0015] Set the output weights between each feedback layer and the hidden layer.

[0016] Preferably, the trend characteristic factors for setting the feedback receiving layer include:

[0017] The trend feature factor of the feedback connection layer is expressed as:

[0018]

[0019] in, Let n be the trend feature factor of the i-th feedback connection layer, and n be the number of attributes of the current feedback connection layer.

[0020]

[0021] Where t represents the change per unit time, P tis the output at the current moment, g is the memory factor of the sheet metal quality inspection model, and k is the memory sample of the sheet metal quality inspection model.

[0022] Preferably, the setting of the output weights between each feedback receiving layer and the hidden layer includes:

[0023] The influence factor of the output weights between the feedback receiving layer and the hidden layer in each layer ranges from 0 to 1, and is expressed as follows:

[0024]

[0025] The influence factor of the output weight between each feedback receiving layer and the hidden layer is corrected using a trend characteristic factor:

[0026] W′=W μ

[0027] The output weights of the feedback layer are represented as follows:

[0028]

[0029] Preferably, after setting the output weights between each feedback receiving layer and the hidden layer, the method further includes correcting the output of the hidden layer, specifically:

[0030] The feedback layer is linearly added to the current input, i.e., H = H(k) + H′(k); the corrected hidden layer output matrix is ​​expressed as:

[0031]

[0032] Where P1 is the input of the sheet metal quality inspection model at the current moment, H′(1) is the output weight of the feedback receiving layer at the current moment, and H′(k) is the output weight of the feedback receiving layer at the current moment.

[0033] Preferably, after acquiring the image data of the production workshop, the method further includes correcting the sheet metal image data, specifically:

[0034] Distortion correction is performed on the image data to obtain the image point coordinates after radial distortion correction, as shown below:

[0035]

[0036] Where, x ′ y' is the abscissa after radial correction, y' is the ordinate after radial correction, x is the abscissa before correction, y is the ordinate before correction, and β is the radial distortion coefficient.

[0037] Distortion correction is performed on the image data to obtain the image point coordinates after tangential distortion correction, as shown below:

[0038]

[0039] Where, x ″ y'' is the abscissa after tangential correction, y'' is the ordinate after tangential correction, and γ is the tangential distortion coefficient.

[0040] Preferably, after acquiring the image data of the production workshop, the method further includes denoising the sheet metal image data, specifically:

[0041] For pixel f(x, y) in the sheet metal image, filter the image using a 3×3 template to obtain the filtered current pixel f′(x, y), which is represented as:

[0042] f′(x,y)=median((x+1,y)+(x,y+1)+(x+1,y+1))

[0043] The processed f′(x, y) is expanded and sorted in ascending order of pixel values ​​to generate a monotonically increasing two-dimensional sequence. Let θ be the two-dimensional template, and the median filter output is:

[0044] F(x,y)=med(θ*f′(x+1,y+1)).

[0045] Preferably, before inputting the sensor data and the sheet metal image data into the sheet metal quality inspection model, the method further includes training the sheet metal quality inspection model, specifically:

[0046] The adaptive dictionary for sample data is extracted using the K-SVD method, including: setting the K-SVD algorithm parameters, the number of iterations, and the sparsity of the OMP algorithm during the iteration process, initializing the adaptive dictionary and the number of atoms in the adaptive dictionary;

[0047] The coefficient matrix is ​​solved using the OMP algorithm;

[0048] Randomly arrange the atomic numbers of the dictionary;

[0049] Find the non-zero elements in the coefficient matrix and record their positions;

[0050] If there are no non-zero elements, calculate the error matrix and record the column containing the maximum error;

[0051] Normalize the column containing the maximum error and use it as the atom of the adaptive dictionary;

[0052] Iterate and update until the preset requirements are met;

[0053] Based on the adaptive dictionary, orthogonal matching pursuit is used to sparsely encode the sample data, and the sparse encoding is used as the input to the sheet metal quality inspection model.

[0054] Image processing-based quality inspection system for hardware sheet metal production process includes:

[0055] The sensor data acquisition module is used to acquire sensor data generated in the digital twin workshop;

[0056] Image data acquisition module, used to acquire sheet metal image data;

[0057] The data transmission module is used to input the sensor data and the sheet metal image data into the sheet metal quality inspection model;

[0058] The data processing module is used to output the sheet metal quality judgment result from the sheet metal quality inspection model.

[0059] An electronic device is characterized by comprising: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein when the processor executes the computer instructions, the electronic device executes a hardware sheet production and quality inspection method based on image processing.

[0060] The beneficial effects of this invention are as follows: 1. This invention combines image data acquired by machine vision with sensor data simulated by a digital twin workshop to detect the quality of sheet metal from multiple aspects, which can greatly improve the accuracy of sheet metal quality detection; 2. The sensor data acquired by this invention comes from the simulation of the digital twin workshop rather than from sensors installed in the real workshop, which can greatly improve the efficiency and accuracy of data acquisition, save detection costs, and improve detection efficiency; 3. This invention uses a sheet metal quality detection model to process sheet metal image data and sensor data, which has a higher accuracy rate than ordinary manual judgment and simple artificial intelligence judgment models, and the model can be continuously improved through trend feature factors, making it suitable for various sheet metal quality detection scenarios. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0063] Figure 1 This is a schematic diagram of the quality inspection method for hardware sheet metal production process based on image processing according to the present invention;

[0064] Figure 2 This is a schematic diagram of the sheet metal quality inspection model construction process of the present invention;

[0065] Figure 3 This is a schematic diagram of the adaptive dictionary process for extracting sample data using the K-SVD method of the present invention;

[0066] Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to the present invention;

[0067] Figure 5 This is a schematic diagram of a sheet metal quality inspection model according to the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0069] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0070] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0071] Currently, the commonly used sheet metal quality inspection is production line inspection based on machine vision. This involves taking pictures of the sheet metal on the production line and then judging the quality of the sheet metal based on the acquired images. However, because the production speed on the sheet metal production line is very fast and some sheet metal is very small, it is difficult to accurately judge whether there are quality problems with the sheet metal using only machine vision. This can easily lead to false positives and false negatives, resulting in low accuracy of sheet metal quality inspection.

[0072] This invention combines image data acquired through machine vision with sensor data simulated in a digital twin workshop to inspect sheet metal quality from multiple perspectives, significantly improving the accuracy of sheet metal quality inspection. The sensor data acquired in this invention comes from the digital twin workshop simulation, rather than from sensors installed in a real workshop, greatly improving data acquisition efficiency and accuracy, saving inspection costs, and increasing inspection efficiency. Furthermore, this invention employs a sheet metal quality inspection model to process sheet metal image data and sensor data, achieving higher accuracy compared to ordinary manual judgment and simple artificial intelligence judgment models. The model can be continuously improved through trend feature factors, making it suitable for various sheet metal quality inspection scenarios.

[0073] A quality inspection method for hardware sheet metal production process based on image processing, reference Figure 1 The steps include:

[0074] Step S100: Acquire sensor data generated by the digital twin workshop;

[0075] Digital twin workshops are an important application of cyber-physical systems and a new model for future workshop operations. It is a system-level cyber-physical system organically combined with hardware, software, and a network, simulating the operational state and processes of a real workshop through digital technology. The concept of a digital twin workshop involves a workshop's reference architecture, operating mode, components, and key technologies, aiming to improve production efficiency and product quality and optimize production processes through digital technology. The core of a digital twin workshop lies in the fusion of data, including data related to the physical space (PS), virtual space (VS), and service systems (SSS), domain knowledge, and a collection of derived data generated through data fusion. The fusion and optimization of this data drive the operation and interaction of the digital twin workshop. By aggregating data from the physical and cyberspaces through specific rules, the digital twin workshop enables intelligent management and optimization of the workshop, thereby improving production efficiency and product quality.

[0076] Digital twins, sometimes used to refer to creating a digital model of a factory's buildings and production lines before construction, allows for simulation and modeling of the factory in a virtual cyberspace, transmitting real-world parameters to the actual factory construction. After the factory and production lines are built, they continue to interact during daily operations. In practical applications, digital twin workshops have been applied in multiple fields, such as the washing machine drum production line using digital twin technology in Haier's Shanghai washing machine interconnected factory, and the "digital twin factory" established in Changzhou. These application cases demonstrate the enormous potential of digital twin workshops in improving production efficiency and optimizing production processes.

[0077] The benefits of digital twin workshops include increased production efficiency, improved product quality, reduced costs, enhanced safety, and the promotion of technological and process innovation and continuous improvement. Digital twin workshops use digital technology to simulate and optimize workshop production processes, helping companies quickly identify potential problems and make predictions and optimizations, thereby improving production efficiency. The core of this solution is to establish a digital twin model that realistically and accurately replicates the actual workshop production process. Sensors and data acquisition devices are used to acquire various data in real time, such as temperature, humidity, and machine status, and different production scenarios are simulated to assess production efficiency under various conditions. Furthermore, the workshop digital twin solution can further improve production efficiency by optimizing production planning and resource allocation, avoiding resource idleness and production blockages. Digital twin workshops can also help companies improve safety levels. Through the modeling and simulation of digital twin factories, companies can predict potential hazards and risks in the production process and take timely measures to prevent accidents. Digital twin factories enable visualization and real-time monitoring of the production process, helping companies better understand the safety status of the production process, identify and solve problems in a timely manner, and improve safety levels. Furthermore, with the continuous development of digital twin technology, it is playing an increasingly important role in industrial manufacturing, bringing higher efficiency and competitiveness to enterprises. Digital twin workshops combine the actual factory with the virtual world through digital modeling and simulation technology, enabling comprehensive monitoring and management of factory data, thereby helping actual factories to allocate resources and achieve intelligent production.

[0078] Step S200: Obtain sheet metal image data;

[0079] Acquiring product images from the production line primarily relies on high-precision image acquisition equipment and advanced image processing algorithms. In a production line environment, image acquisition is a crucial step, directly impacting the accuracy and efficiency of product quality inspection. To achieve efficient quality control and production line automation, high-precision image acquisition equipment and corresponding image processing algorithms are necessary. Specific steps and methods for acquiring product images from the production line include: Using high-precision image acquisition equipment: On the production line, high-definition cameras or other image acquisition devices are installed to capture product image information in real time. These devices need to have high resolution and fast response capabilities to ensure that the acquired images clearly reflect product details. Applying advanced image processing algorithms: Image processing algorithms are used to process and analyze the acquired images, extracting key information such as the product's position, shape, and size. These algorithms can identify product features, calculate the accurate location of the gripping point, and control the gripping mechanism for precise gripping. Solving technical challenges: When products are moving at high speeds, the vision system needs to be able to stably and accurately acquire product position information. This requires the use of high-speed image processing technology and real-time tracking algorithms to ensure that the time difference between image acquisition, processing, analysis, and control is as small as possible, in order to achieve real-time and accurate positioning and gripping. Adapting to Diverse Product Lines: Due to the wide variety of products on the production line, varying in shape and size, the visual positioning and grasping system needs to possess high flexibility and adaptability. This requires the system to automatically adjust and optimize according to different product characteristics, ensuring stable and reliable positioning and grasping under various conditions. Considering Practical Factors: In practical applications, the impact of factors such as lighting conditions, product surface reflections, and occlusions on the quality and accuracy of image acquisition must be considered. Appropriate adjustments and optimizations should be made according to specific circumstances to ensure the stability and reliability of the system. Acquiring product images from the production line is a complex process involving hardware equipment, software algorithms, and practical application adjustments. It requires comprehensive consideration of multiple factors to ensure that the acquired image information accurately reflects product quality and supports the automation and intelligent management of the production line.

[0080] Step S300: Input sensor data and sheet metal image data into the sheet metal quality inspection model;

[0081] Sheet metal quality inspection models are artificial intelligence algorithms that utilize big data and deep learning technologies to simulate human thinking and creativity. They can process multimodal data, such as images, text, and audio, to predict future events or generate new content. These models typically have a large number of parameters and can demonstrate intelligence in different domains and tasks, such as generating high-quality text, images, and audio-visual content. Through pre-training and the resulting knowledge base, sheet metal quality inspection models can produce valuable answers that meet actual user needs, improving the naturalness and fluency of dialogue. Furthermore, these models can learn autonomously without human annotation, making them an important research direction for future predictive models. With the increasing scale of data, improved computing power, and algorithmic innovation, sheet metal quality inspection models will face challenges in areas such as large-scale data processing, multimodal data fusion, interpretable predictive models, autonomous learning, and ethical and privacy considerations. The applications of sheet metal quality inspection models are wide-ranging; they can process large amounts of data from various data sources through big data technologies and algorithms, thereby more comprehensively capturing market trends and corporate performance. These models can identify and analyze complex relationships and patterns, providing more accurate predictions and avoiding biases and misjudgments inherent in human intuition. Furthermore, sheet metal quality inspection models typically possess good interpretability, helping users better understand the prediction results. Sheet metal quality inspection models are algorithms that utilize big data and deep learning technologies to simulate human thinking and creativity, demonstrating intelligence across different domains and tasks, and providing powerful tools for future prediction and generation.

[0082] Step S400: The sheet metal quality inspection model outputs the sheet metal quality judgment result.

[0083] The quality judgment standards for sheet metal mainly include the following aspects: Dimensional and geometric requirements: The length, width, thickness, and other dimensional parameters of the sheet metal should meet standard requirements to ensure the accuracy and consistency of its geometric shape. Physical properties: These include the density, strength, hardness, bending performance, impact performance, and modulus of elasticity of the sheet metal, evaluated through experimental tests such as tensile tests, impact tests, and hardness tests. Chemical properties: The chemical composition, solubility, and corrosion resistance of the sheet metal are evaluated through water quality tests, corrosion tests, and solubility tests. Surface quality: This includes the flatness, smoothness, unevenness, defects (such as cracks, damage, bubbles, etc.), and color difference of the sheet metal surface, evaluated through visual observation, measurement with measuring instruments, and optical microscopy. Environmental requirements: The environmental indicators of the sheet metal, such as formaldehyde emission, volatile organic compound content, and heavy metal content, are evaluated through indoor air quality testing and chemical analysis. Fire performance: The combustion performance and flame retardant performance of the sheet metal are evaluated to ensure its performance in fire conditions, through combustion tests and flame retardant tests. Surface treatment and coating evaluation: For coated or surface-treated boards, the adhesion, corrosion resistance, wear resistance, and appearance quality of the coating are evaluated through coating peeling tests, salt spray tests, and visual inspections.

[0084] The sheet metal quality assessment standard aims to ensure that its physical, chemical, environmental, and safety properties meet the requirements to satisfy the needs of different application fields. Dimensional and geometric requirements are mainly judged based on data from sheet metal images, while physical properties, chemical properties, surface quality, environmental protection requirements, fire resistance, and surface treatment and coating evaluation are mainly judged based on sensor data from a digital twin workshop simulation. This invention integrates sheet metal image data and sensor data from a digital twin workshop to comprehensively assess sheet metal quality, which can greatly improve the accuracy of the assessment. Furthermore, obtaining sensor data from a digital twin workshop is more convenient than obtaining data by placing sensors in a real workshop, and it is less susceptible to the inaccuracy of sensor data caused by external environmental influences, further improving the accuracy and efficiency of sheet metal quality inspection.

[0085] Preferably, refer to Figure 2 and Figure 5 Before inputting the sensor data and image data into the sheet metal quality inspection model in step S300, step S230 is also included to construct the sheet metal quality inspection model, specifically as follows:

[0086] Step S231: Construct the input layer, hidden layer, feedback receiving layer, and output layer of the sheet metal quality inspection model;

[0087] Input Layer: In a computer control system, the input layer is the interface between the computer control system and the external world, responsible for receiving signals from sensors, operators, or other input devices. These signals may be continuous (e.g., temperature, pressure) or discrete (e.g., switch states). The main task of the input layer is to convert these raw signals into digital signals that the computer can process, typically achieved through an analog-to-digital converter (ADC). In a neural network, the input layer is the first layer, and its function is to convert the input data into a format that the neural network can process internally. The number of neurons in the input layer depends on the number of features in the input data. In this embodiment of the invention, the input layer only memorizes the input samples; therefore, the number of nodes in the input layer is the same as the number of attributes of the input samples.

[0088] In neural networks, hidden layers are one or more layers located between the input and output layers. Their function is to transform input data into a higher-level feature representation. The number of neurons and connections in each hidden layer vary depending on the specific neural network architecture. The number of hidden layers is an important parameter in neural network architecture design, directly affecting the performance and training time of the neural network. In this embodiment of the invention, hidden layers are used to activate the input, and the output of the hidden layers is fed to the feedback relay layer and the output layer.

[0089] The feedback layer adjusts the control strategy based on the information fed back from the output results to achieve more precise control. This feedback mechanism is a key component in many control systems, ensuring that the system can make corresponding adjustments based on actual operating conditions. In this embodiment of the invention, the feedback layer stores the output of the hidden layer, and the output layer obtains the final calculation result of the network.

[0090] Step S232: Set the connection weights between the input layer and the hidden layer;

[0091] Each connection between the input layer and the hidden layer corresponds to a real number called the connection weight. The role of the connection weight is to add a bias term to the input signal of the neuron, thereby enhancing or inhibiting the intensity of the input signal. By adjusting the weights, the activation mode and response level of the neuron can be changed. In addition, the connection weights can also act as a feature extractor, improving the model's performance and generalization ability by learning features and patterns in the data.

[0092] Step S233: Set the hidden layer node bias and activation function;

[0093] An activation function is a function that operates on neurons in an artificial neural network, responsible for mapping the neuron's input to its output. Activation functions introduce nonlinearity into neurons, allowing the neural network to approximate any nonlinear function, thus enabling its application in numerous nonlinear models.

[0094] Step S234: Set the trend characteristic factor of the feedback receiving layer;

[0095] In this embodiment of the invention, the trend feature factor represents the rate of change of data per unit time, which can help the feedback layer improve its memory and forgetting functions.

[0096] Step S235: Set the output weights between each feedback receiving layer and the hidden layer.

[0097] The hidden layer uses a linear or nonlinear function as its transfer function, the output layer acts as a linear weighting layer, the feedback receiving layer connects the feedback within or between layers to achieve the purpose of memory, and the output weights are used to optimize the relationship between each feedback receiving layer and the hidden layer, thereby improving the accuracy of the sheet metal quality inspection model.

[0098] Preferably, step S234, setting the trend characteristic factor of the feedback receiving layer includes:

[0099] The trend feature factor of the feedback connection layer is expressed as:

[0100]

[0101] in, Let n be the trend feature factor of the i-th feedback connection layer, and n be the number of attributes of the current feedback connection layer.

[0102]

[0103] Where t represents the change per unit time, P t is the output at the current moment, g is the memory factor of the sheet metal quality inspection model, and k is the memory sample of the sheet metal quality inspection model.

[0104] This invention employs a sliding window for data feature extraction. Sliding window data extraction is a flow control technique primarily used to improve data transmission efficiency in network communication. The sliding window technique processes data by maintaining a fixed-size window and continuously sliding it. It defines two pointers, one pointing to the start position and the other to the end position of the window, and then continuously moves these two pointers, reading the window's contents and performing necessary operations. This technique is suitable for solving problems requiring searching or calculation within continuous substrings or subarrays, such as finding the longest non-repeating substring or finding the smallest window containing a target substring within a string. The application of sliding windows is not limited to the data link layer and transport layer; it is also widely used in data analysis, time series data processing, and other fields. In data analysis, the sliding window technique can perform window operations on time series data to obtain various statistical indicators, thereby better understanding the trends and characteristics of the data. This technique observes the data within a fixed-size window by sliding it across the data sequence. The window moves sequentially, each time moving a fixed step size, until it covers the entire data sequence. Sliding window data extraction is a technique that uses a fixed-size window to process data. It involves continuously moving the window and reading its contents to perform data analysis or processing. It is applicable to various scenarios, including but not limited to network communication, string processing, and data analysis.

[0105] Preferably, step S235, setting the output weights between each feedback receiving layer and the hidden layer includes:

[0106] The influence factor of the output weights between the feedback receiving layer and the hidden layer in each layer ranges from 0 to 1, and is expressed as follows:

[0107]

[0108] The influence factor of the output weight between each feedback receiving layer and the hidden layer is corrected using a trend characteristic factor:

[0109] W′=W μ

[0110] The output weights of the feedback layer are represented as follows:

[0111]

[0112] Preferably, after setting the output weights between each feedback receiving layer and the hidden layer in step S235, the method further includes step S236, which corrects the output of the hidden layer, specifically as follows:

[0113] The feedback receiving layer is linearly added to the current input, i.e., H = H(k) + H′(k);

[0114] The corrected hidden layer output matrix is ​​represented as follows:

[0115]

[0116] Where P1 is the input of the sheet metal quality inspection model at the current moment, H′(1) is the output weight of the feedback receiving layer at the current moment, and H′(k) is the output weight of the feedback receiving layer at the current moment.

[0117] Preferably, after acquiring the image data of the production workshop in step S200, the method further includes step S210, which corrects the sheet metal image data, specifically as follows:

[0118] Distortion correction is performed on the image data, and the coordinates of the image points after radial distortion correction are represented as follows:

[0119]

[0120] Where x′ is the abscissa after radial correction, y′ is the ordinate after radial correction, x is the abscissa before correction, y is the ordinate before correction, and β is the radial distortion coefficient.

[0121] Radial distortion correction (RCC) involves a series of image processing steps on distorted images acquired by a camera to remove the distortions present in the image. This typically involves using an appropriate parametric model to simulate image distortion, calculating the model's parameters, and then using that model to remove the distortions generated during camera imaging. The purpose of RCC is to facilitate subsequent processing in computer vision fields such as spatial localization, object detection, and tracking algorithms. This technology has extremely wide applications in many computer vision-related fields, including video surveillance, virtual reality, robot navigation, military targeting, television editing, and medical image analysis.

[0122] Radial distortion primarily includes two forms: barrel distortion and pincushion distortion. Barrel distortion is characterized by pixels diverging towards the image center, while pincushion distortion, conversely, causes the image edges to contract inwards. Both types of distortion are caused by light rays bending more far from the lens center than closer to it. Radial distortion correction calculates and adjusts image parameters to make the corrected image more realistic, improving image quality and accuracy.

[0123] After performing distortion correction on the image data, the coordinates of the image points after tangential distortion correction are represented as follows:

[0124]

[0125] Where, x ″ Let y be the x-coordinate after tangential correction. ″ y is the ordinate after tangential correction, and γ is the tangential distortion coefficient.

[0126] Tangential distortion correction is a step-by-step calibration method that considers distortion factors and uses optimization algorithms to further improve calibration accuracy. This method first solves for camera parameters using direct linear transformation or perspective transformation matrix methods. Then, using the obtained parameters as initial values, it further considers distortion factors, especially tangential distortion, and optimizes these parameters using nonlinear optimization algorithms to achieve the correction goal. Tangential distortion correction mainly addresses the problem of inaccurate angles between the lens and the imaging plane, i.e., the lens is not perfectly parallel to the image plane, which may occur during camera assembly. This distortion leads to a decrease in image quality, therefore, specific algorithms and techniques are needed to correct this distortion to improve image accuracy and sharpness.

[0127] Preferably, after acquiring the image data of the production workshop in step S200, the method further includes step S220, which involves denoising the sheet metal image data, specifically as follows:

[0128] For pixel f(x, y) in the sheet metal image, filter the image using a 3×3 template to obtain the filtered current pixel f′(x, y), which is represented as:

[0129] f′(x,y)=median((x+1,y)+(x,y+1)+(x+1,y+1))

[0130] The processed f′(x, y) is expanded and sorted in ascending order of pixel values ​​to generate a monotonically increasing two-dimensional sequence. Let θ be the two-dimensional template, and the median filter output is:

[0131] F(x,y)=med(θ*f′(x+1,y+1)).

[0132] Image denoising refers to the process of reducing noise in digital images, aiming to recover the original clear image from a noisy image. This process involves identifying and eliminating noise in the image, which can be caused by various factors such as optical, atmospheric, human, and technical factors. The goal of image denoising is to improve image quality while preserving as many important features of the original image as possible. In image processing, denoising is an extremely important step, encompassing two main methods: spatial domain denoising and transform domain denoising. Spatial domain denoising directly operates on the image's gray levels, while transform domain denoising corrects the transform coefficients of the image within a specific transform domain. Spatial domain denoising methods mainly include mean filtering, median filtering, and Wiener filtering, which exhibit different performance characteristics in removing different types of noise (such as Gaussian noise, salt-and-pepper noise, and Poisson noise). For example, median filtering is effective at filtering out impulse noise (such as salt-and-pepper noise), which manifests as random white or black dots in the image. In addition, denoising methods include bilateral filtering, which aims to eliminate noise in the image while avoiding the introduction of new noise or causing edge blurring. Denoising techniques have a wide range of applications, including but not limited to improving image quality, enhancing image readability and visibility, and improving the accuracy of subsequent image analysis.

[0133] The main purpose of image filtering and denoising is to reduce or eliminate noise in an image by applying specific filtering techniques, such as Gaussian filtering and median filtering, while preserving as much of the original image's features and details, such as edges and textures, as possible. This makes the processed image more suitable for further analysis or display. Gaussian filtering partially overcomes the shortcomings of noise suppression, but it cannot completely eliminate them because it does not consider the differences in pixel values ​​and has relatively low weighting for edge information. Median filtering is a non-linear noise removal method that can both remove noise and preserve image edges. Its principle is to replace the value of a point in the digital image with the median value of all points in a region containing that point, thereby removing noise while preserving the image's edge information. Image filtering and denoising techniques, by applying different filtering methods, reduce image noise while preserving important image features as much as possible, providing a better foundation for subsequent image processing and analysis.

[0134] Preferably, refer to Figure 3 Before inputting the sensor data and sheet metal image data into the sheet metal quality inspection model in step S300, step S240 is also included to train the sheet metal quality inspection model, specifically as follows:

[0135] Step S241, extracting an adaptive dictionary from the sample data using the K-SVD method, including: Step S2411, setting the K-SVD algorithm parameters, the number of iterations and the sparsity of the OMP algorithm during the iteration process, and initializing the adaptive dictionary and the number of atoms in the adaptive dictionary;

[0136] Step S2412: Solve for the coefficient matrix using the OMP algorithm;

[0137] The OMP algorithm, short for Orthogonal Matching Pursuit, is a widely used algorithm in signal processing and compressed sensing. It is particularly suitable for recovering information from sparse signals. The working principle of the OMP algorithm mainly includes: Initialization: Initially, the residual is set to the difference between the measurement vector and the projection, and the solution vector is initialized to empty. Iteration process: In each iteration, the algorithm calculates the inner product of the current residual and the column vectors in the dictionary, and selects the corresponding atom with the largest inner product. Then, the index of this atom is added to the set of selected atoms. Solution vector update: The solution vector is updated by calculating the least squares solution using the atoms in the selected atom set. Residual update: The residual is updated using the sub-dictionary composed of the selected atoms. Stopping criterion: If the stopping criterion is met (e.g., the residual energy is below a certain threshold or a predetermined number of iterations is reached), the iteration ends; otherwise, it returns to the second step to continue iterating. In this way, the OMP algorithm can effectively reconstruct the original sparse signal from a small number of observations.

[0138] Step S2413: Randomly arrange the atomic indices of the dictionary, denoted as reperm, and initialize the counter j = 1;

[0139] Step S2414: Let h = rperm(j), find the non-zero elements in the coefficient matrix, record the positions of the non-zero elements, and use them as the h-th atom of the randomized dictionary;

[0140] Step S2415: If there are no non-zero elements, calculate the error matrix and record the column containing the maximum error;

[0141] Step S2416: Normalize the column containing the maximum error as the atom of the adaptive dictionary;

[0142] Step S2417: Iteratively update the counter until the preset requirement is met;

[0143] Iterative updates involve continuously adjusting the parameters in the network to minimize prediction error or maximize prediction accuracy, thereby enabling the model to better fit the training data. This update process is achieved through optimization algorithms such as gradient descent. Each update is based on the current model's prediction error, calculating the gradient using the backpropagation algorithm, and then adjusting the network parameters according to the calculated gradient, aiming to improve the model's performance in the next iteration. The benefits of iterative updates include: improved model performance: By continuously adjusting the network parameters, the sheet metal detection model can gradually learn more accurate feature representations, thereby improving the model's prediction performance. Each iterative update helps the model better adapt to the training data, thus improving the model's generalization ability. Adapting to data changes: In practical applications, the dataset may change, such as the addition of new samples or changes in sample distribution. Through iterative updates, the sheet metal detection model can adapt to these changes, maintaining the model's performance unaffected, further improving the learning efficiency and performance of the sheet metal detection model. Iterative updates of the sheet metal detection model, by continuously adjusting the network parameters to improve model performance and adapt to data changes, are an indispensable step in training deep learning models.

[0144] Step S242: Perform sparse coding on the sample data using orthogonal matching pursuit based on the adaptive dictionary, and use the sparse coding as the input to the sheet metal quality inspection model.

[0145] The dictionary learned using the K-SVD learning algorithm exhibits better sparsity than other dictionaries, accurately representing the sparse characteristics of data with fewer coefficients, and effectively compressing massive operational monitoring data. Compared to time-domain sample signals, frequency-domain signals can effectively eliminate the influence of fault impact time shift between samples of the same type, resulting in better robustness in fault mode recognition. Therefore, frequency-domain data is used for processing. The process of obtaining the input for the sheet metal quality inspection model using the K-SVD algorithm is as follows: All data in the database is frequency-domain transformed to obtain its frequency-domain data; the frequency-domain database is divided into dataset 1 and dataset 2, ensuring that both datasets uniformly contain each fault type; dataset 1 is used as sample data for the K-SVD algorithm to learn the dictionary; based on this dictionary, the OMP algorithm is used to obtain the sparse coefficient matrices of dataset 1 and dataset 2 as training and test sets, respectively, and labels are assigned according to the fault type to obtain the input for the sheet metal quality inspection model.

[0146] Example 2

[0147] Image processing-based quality inspection system for hardware sheet metal production process includes:

[0148] The sensor data acquisition module is used to acquire sensor data generated in the digital twin workshop;

[0149] Image data acquisition module, used to acquire sheet metal image data;

[0150] The data transmission module is used to input sensor data and sheet metal image data into the sheet metal quality inspection model;

[0151] The data processing module is used to output the sheet metal quality judgment results from the sheet metal quality inspection model.

[0152] Example 3

[0153] An electronic device is characterized by comprising: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein when the processor executes the computer instructions, the electronic device executes a quality inspection method for the production process of metal sheet metal based on image processing.

[0154] refer to Figure 4 The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which may include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment of the invention. It should be understood that in the various embodiments of the invention, coupling refers to mutual connection through a specific method, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0155] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., and this embodiment of the invention is not limited thereto.

[0156] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the present invention. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0157] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Output device 24 and input device 23 can be independent devices or an integrated device.

[0158] This invention combines image data acquired through machine vision with sensor data simulated in a digital twin workshop to inspect sheet metal quality from multiple perspectives, significantly improving the accuracy of sheet metal quality inspection. The sensor data acquired in this invention comes from the digital twin workshop simulation, rather than from sensors installed in a real workshop, greatly improving data acquisition efficiency and accuracy, saving inspection costs, and increasing inspection efficiency. Furthermore, this invention employs a sheet metal quality inspection model to process sheet metal image data and sensor data, achieving higher accuracy compared to ordinary manual judgment and simple artificial intelligence judgment models. The model can be continuously improved through trend feature factors, making it suitable for various sheet metal quality inspection scenarios.

[0159] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A quality inspection method for hardware sheet metal production process based on image processing, characterized in that, include: Acquire sensor data generated by the digital twin workshop of the hardware sheet metal production process, including: establishing a digital twin model through digital modeling and simulation technology to accurately replicate the actual workshop production process; acquiring various data of the workshop in real time through sensors and data acquisition equipment, including temperature, humidity and machine status; and simulating different production scenarios to evaluate production efficiency under various conditions and optimize production plans and resource allocation. Acquiring sheet metal image data involves installing high-definition cameras or other image acquisition devices on the production line in the workshop to capture product image information in real time; using image processing algorithms to process and analyze the acquired images to extract key product information, including location, shape, and size; in practical applications, it is also necessary to consider the impact of lighting conditions, product surface reflection, and occlusion on the quality and accuracy of image acquisition. The sensor data and the sheet metal image data are input into the sheet metal quality inspection model; the sheet metal quality inspection model uses big data and deep learning technology to simulate human thinking and creativity, and processes multimodal data, including images, text and audio, to predict future events or generate new content. The sheet metal quality inspection model outputs sheet metal quality judgment results, including judgment of size and geometric requirements based on data from sheet metal images, and judgment of physical properties, chemical properties, surface quality, environmental protection requirements, fire performance, and surface treatment and coating evaluation based on sensor data from digital twin workshop simulation. Before inputting the sensor data and the image data into the sheet metal quality inspection model, the process further includes constructing the sheet metal quality inspection model and training the sheet metal quality inspection model. Specifically, constructing the sheet metal quality inspection model involves: Construct the input layer, hidden layer, feedback support layer, and output layer of the sheet metal quality inspection model; Set the connection weights between the input layer and the hidden layer; Set the hidden layer node bias and activation function; Set the trend characteristic factor of the feedback layer to reflect the impact of historical output on the current output; Set the output weights between each feedback receiving layer and the hidden layer; The sheet metal quality inspection model is trained as follows: An adaptive dictionary is extracted from the sample data using the K-SVD method; Based on the adaptive dictionary, orthogonal matching pursuit is used to sparsely encode the sample data, and the sparse encoding is used as the input to the sheet metal quality inspection model; This includes using image processing algorithms to process and analyze the acquired images, including denoising the sheet metal image data, specifically: Median filtering is applied to the pixels in the sheet metal image using a 3×3 filter template. The filtered pixel values ​​are arranged in ascending order to generate a monotonically increasing two-dimensional sequence. Median filtering output based on two-dimensional template.

2. The method for quality inspection of hardware sheet metal production process based on image processing according to claim 1, characterized in that, The setting of output weights between each feedback receiving layer and the hidden layer includes: The influence factor of the output weight between the feedback receiving layer and the hidden layer of each layer is between 0 and 1; The influence factor of the output weight between each feedback receiving layer and the hidden layer is corrected by using trend characteristic factors.

3. The method for quality inspection of hardware sheet metal production process based on image processing according to claim 2, characterized in that, After setting the output weights between each feedback receiving layer and the hidden layer, the method further includes correcting the output of the hidden layer, specifically: The feedback layer is linearly added to the current input.

4. The method for quality inspection of hardware sheet metal production process based on image processing according to claim 1, characterized in that, After acquiring the image data of the production workshop, the method further includes correcting the sheet metal image data, specifically: The image data is subjected to distortion correction to obtain the coordinates of image points after radial distortion correction; Distortion correction is performed on the image data to obtain the coordinates of image points after tangential distortion correction.

5. The method for quality inspection of hardware sheet metal production process based on image processing according to claim 1, characterized in that, Before inputting the sensor data and the sheet metal image data into the sheet metal quality inspection model, the method further includes training the sheet metal quality inspection model, specifically: The adaptive dictionary for sample data is extracted using the K-SVD method, including: setting the K-SVD algorithm parameters, the number of iterations, and the sparsity of the OMP algorithm during the iteration process, initializing the adaptive dictionary and the number of atoms in the adaptive dictionary; The coefficient matrix is ​​solved using the OMP algorithm; Randomly arrange the atomic numbers of the dictionary; Find the non-zero elements in the coefficient matrix and record their positions; If there are no non-zero elements, calculate the error matrix and record the column containing the maximum error; Normalize the column containing the maximum error and use it as the atom of the adaptive dictionary; Iterate and update until the preset requirements are met; Based on the adaptive dictionary, orthogonal matching pursuit is used to sparsely encode the sample data, and the sparse encoding is used as the input to the sheet metal quality inspection model.

6. A quality inspection system for hardware sheet metal production process based on image processing, characterized in that, The system employs the image processing-based quality inspection method for hardware sheet metal production processes as described in any one of claims 1-5, wherein the system comprises: The sensor data acquisition module is used to acquire sensor data generated in the digital twin workshop; Image data acquisition module, used to acquire sheet metal image data; The data transmission module is used to input the sensor data and the sheet metal image data into the sheet metal quality inspection model; The data processing module is used to output the sheet metal quality judgment result from the sheet metal quality inspection model.

7. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein when the processor executes the computer instructions, the electronic device performs the image processing-based quality inspection method for metal sheet production as described in any one of claims 1 to 5.

Images