A product quality inspection system based on deep learning and machine vision

Abstract

The present invention relates to the technical field of product quality inspection, and in particular to a product quality detection system based on deep learning and machine vision, including a multi-modal image acquisition module, a deep learning defect detection module, and a dynamic resource scheduling module. The multi-modal image acquisition module uses a high-resolution camera array and an adaptive light source to comprehensively acquire product images; the deep learning defect detection module accurately identifies defects through double-path residual denoising, multi-scale context awareness, and graph topology defect classification; the dynamic resource scheduling module realizes reasonable scheduling of personnel and materials and production line balance based on the detection results by means of a skill map, an ant colony algorithm, etc.; the augmented reality interaction module intuitively displays defects and provides repair navigation through holographic projection and multi-modal interaction; the self-evolving detection model can update weights based on new samples, and the multi-physics field simulation module simulates the influence of defects on the mechanical properties of products. The present invention has high detection accuracy and fast efficiency, and significantly improves the product quality detection and production management levels.

Description

Technical Field

[0001] The present invention relates to the technical field of product quality inspection, and specifically to a product quality detection system based on deep learning and machine vision. Background Art

[0002] In the field of industrial manufacturing, product quality is the lifeline of an enterprise. With the rapid development of the manufacturing industry, the market's requirements for product quality are becoming increasingly stringent. Traditional product quality inspection methods have gradually revealed many problems and are difficult to meet the needs of modern production. The present invention has emerged under such a background.

[0003] Disadvantages of traditional personnel scheduling methods: In the past, after product defects were found during the production process, personnel scheduling often relied on manual experience judgment. Managers arranged maintenance personnel based on their general understanding of employees' skills and subjective perception of the defect situation. This method lacks systematicness and accuracy, and it is easy to have a situation where the skills of the maintenance personnel do not match the type of defect. For example, in the manufacturing of electronic devices, when a welding defect is detected on a circuit board, personnel who are not good at electronic welding repair may be arranged to handle it, resulting in low repair efficiency, difficult-to-guarantee repair quality, and even possible product scrapping due to improper repair, increasing production costs.

[0004] Dual challenges of production efficiency and quality: With the accelerating production rhythm, rapid response and efficient handling of product defects have become crucial. However, traditional personnel scheduling methods cannot quickly make the optimal arrangement based on the defect detection results. On large-scale production lines, if the appropriate personnel cannot be promptly allocated to the defect handling positions, it will lead to an extended production stagnation time, affecting the overall production efficiency. At the same time, due to the mismatch of the skills of the maintenance personnel, the defect problems may not be completely solved, causing problems to reappear in subsequent processes, reducing product quality, and damaging the enterprise's reputation.

[0005] Increasing product diversity and defect complexity: Nowadays, there are a wide variety of products and complex production processes, and the corresponding defect types have become more diverse and complex. Defects in different products may involve different professional knowledge and skill areas. For example, in automobile manufacturing, body painting defects and engine component defects require personnel with different professional backgrounds and skill levels to handle. Traditional personnel scheduling methods are difficult to cope with such complex situations and cannot quickly and accurately select the most suitable candidates from among many maintenance personnel, resulting in an extended defect handling cycle and serious impacts on production efficiency and quality.

[0006] Therefore, in view of the above problems, a product quality detection system based on deep learning and machine vision is proposed. Summary of the Invention

[0007] The object of the present invention is to provide a product quality inspection system based on deep learning and machine vision to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A product quality inspection system based on deep learning and machine vision, comprising:

[0010] A multi-modal image acquisition module: composed of a high-resolution industrial camera array, an adaptive light source system and an image acquisition card; the industrial camera array is annularly distributed to cover the product surface in all directions, realizing a 360° detection angle of view; the adaptive light source system dynamically adjusts the multi-band light combination according to the surface reflectivity of the product, and the spectral range covers 400 - 1100nm; the image acquisition card has the ability to parallel process at least 8 camera signals.

[0011] A deep learning defect detection module: used to realize the end-to-end analysis from image to defect decision, specifically including:

[0012] A dual-path residual denoising unit: adopting a dual-branch network structure to respectively extract the low-frequency structure information and high-frequency detail features of the image, and fusing the denoising results through residual connection.

[0013] A multi-scale context awareness unit: by means of a dilated convolutional pyramid network DCPN, synchronously obtaining local defect details and global context correlation features.

[0014] A graph topology defect classification unit: constructing a spatial relationship graph of the defect area, and using a graph convolutional network GCN to iteratively update the node features, and finally outputting the joint probability of the defect type and severity.

[0015] A dynamic resource scheduling module: generating a multi-objective optimization scheduling scheme based on the defect detection results, covering:

[0016] A personnel scheduling unit driven by a skill map: according to the defect type, matching the skill map database of maintenance personnel, and generating a priority queue by calculating the skill matching degree.

[0017] A spatio-temporal constraint material scheduling unit: combining the defect position coordinates and the production line layout topology map, and using the ant colony optimization algorithm to plan the material transportation path.

[0018] A production line throughput balancing unit: adopting a Markov decision process to dynamically adjust the resource allocation ratio between the detection station and the repair station.

[0019] An augmented reality interaction module: including a holographic projection device and a multi-modal interaction terminal, and real-time superimposing a three-dimensional heat map of defects, a repair path navigation and process parameter correction suggestions.

[0020] As a preferred solution, the network structure of the dual-path residual denoising unit is as follows:

[0021] Low-frequency branch: A 5×5 large convolutional kernel is used to extract the macroscopic structural features of the image, and the number of output channels is 64;

[0022] High-frequency branch: A 3×3 small convolutional kernel is used, and a dilated convolutional layer with a dilation rate of 2 is combined to extract the microscopic texture features, and the number of output channels is 128;

[0023] Feature fusion layer: The outputs of the two branches are weighted and fused through a cross-channel attention mechanism, and the calculation formula is:

[0024] , where, , is a learnable weight matrix used to perform a weighted transformation on the input features; is the Sigmoid function that maps the weighted result to between 0 and 1 to obtain the attention weight ; represents the features output by the low-frequency branch, represents the features output by the high-frequency branch; represents element-wise multiplication, and after multiplying the attention weight with the corresponding branch features and adding them together, the fused features are obtained.

[0025] As a preferred solution, the dilated convolutional pyramid network of the multi-scale context awareness unit includes:

[0026] Base feature layer: A bottom-layer feature map with a resolution of 256×256 is extracted through a 3×3 standard convolution;

[0027] Multi-scale context layer: Dilated convolutional layers with dilation rates of 2, 4, and 8 are deployed in parallel, and the resolutions of the output feature maps are 128×128, 64×64, and 32×32 respectively;

[0028] Feature refinement module: A two-way feature pyramid structure is used to fuse multi-scale features, and the calculation formula is:

[0029] ; where, represents the transposed convolution operation that restores the feature maps obtained by dilated convolutions with different dilation rates to the same resolution as the base feature map, , corresponding to the feature maps under different dilation rates respectively; is element-wise multiplication, which multiplies the transposed convolution feature maps with the base feature map processed by the channel attention module element-wise; It is a channel attention module, which is used to calculate the importance weights of each channel of the basic feature map, highlight key information, and finally add the three to obtain the refined features. .

[0030] As a preferred solution, the working process of the graph topology defect classification unit includes:

[0031] Node feature initialization: Using the detected candidate defect region ROI feature vector as the graph node, with a feature dimension of 512;

[0032] Dynamic edge weight calculation: Based on the spatial distance between defect regions and feature similarity Generate an adjacency matrix:

[0033] , where, is the distance attenuation coefficient; is the temperature parameter; represents the th and the th spatial distance between defect regions, represents the th and the th feature similarity between defect regions; is an element of the adjacency matrix, representing the edge weight between the th and the th nodes; k is an index variable used to traverse all possible nodes.

[0034] Multi-hop graph convolution: Iteratively update the node representation through a 3-layer graph convolutional network, with the output dimension of each layer being 256, 128, and 64 respectively.

[0035] As a preferred solution, the optimization objective function of the spatio-temporal constrained material scheduling unit is:

[0036] , where, is the transportation time of the th AGV; is the energy consumption cost of the th AGV; , are the weight coefficients of the transportation time and the energy consumption cost respectively; is a weight coefficient used to adjust the influence of the maximum transportation time on the optimization objective; is the number of AGVs, and the Pareto optimal solution set is solved by improving the ant colony algorithm, respectively represent the transportation times of the

[0037] As a preferred solution, it further includes:

[0038] Self-evolving detection model: Adopt an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, specifically including:

[0039] Graph convolutional layer weight matrix: Update the learnable parameter matrix for each layer of graph convolutional operation , where and are respectively the feature dimensions of the input and output of the th layer;

[0040] Dynamic edge weight generation parameters: Update the distance attenuation coefficient and temperature parameter in the calculation of the adjacency matrix to optimize the spatial relationship modeling between defect regions;

[0041] Attention mechanism parameters: If the attention mechanism is introduced in the classification unit, update the attention weight matrix and linear transformation parameter , ; and respectively represent the input and output channel dimensions in the attention mechanism;

[0042] The weight update is achieved through the following formula:

[0043] , where is the current batch of data, is the sample in the historical memory bank, ; is the gradient of the loss function of the th sample in the historical memory bank with respect to the network weight ; is the learning rate parameter, controlling the step size of updating the weight based on the current batch of data; is the learning rate parameter, controlling the step size of updating the weight based on the samples in the historical memory bank, so as to obtain the network weight at time ;

[0044] Multi-physical field simulation module: Integrate a finite element analysis engine to simulate the influence of defects on the mechanical properties of products and generate a contour map of residual stress distribution.

[0045] As can be seen from the technical solution provided by the present invention above, the beneficial effects of a product quality detection system based on deep learning and machine vision provided by the present invention are:

[0046] 1. The dynamic resource scheduling module arranges maintenance personnel reasonably by using the skill graph-driven personnel scheduling unit according to the defect detection results, the spatio-temporal constrained material scheduling unit plans the optimal material transportation path, and the production line throughput balancing unit dynamically adjusts the resource allocation of the detection and maintenance workstations, improving the maintenance efficiency, reducing the logistics cost, avoiding the idleness or congestion of production line resources, and enabling the entire production line to operate efficiently.

[0047] 2. The multi-modal image acquisition module uses the annular distribution of the high-resolution industrial camera array to achieve 360° non-blind detection of the product surface. It cooperates with the adaptive light source system to dynamically adjust the illumination according to the surface reflectivity of the product, ensuring that the acquired images are clear and comprehensive, providing a high-quality data basis for subsequent defect detection, and effectively avoiding missed detections caused by limited detection perspectives and poor image quality.

[0048] 3. The deep learning defect detection module uses a dual-path residual denoising unit to remove image noise, a multi-scale context awareness unit to capture local and global features, and a graph topology defect classification unit to accurately classify defect types and evaluate severity, significantly improving the accuracy of defect detection. Brief Description of the Drawings

[0049] Figure 1 It is a schematic diagram of the overall structure of a product quality detection system based on deep learning and machine vision according to the present invention. Detailed Embodiments

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the drawings of the specification and the specific embodiments.

[0052] As Figure 1 shown, an embodiment of the present invention provides a product quality detection system based on deep learning and machine vision, including a multi-modal image acquisition module, a deep learning defect detection module, a dynamic resource scheduling module, and an augmented reality interaction module.

[0053] In this embodiment, the multi-modal image acquisition module consists of a high-resolution industrial camera array, an adaptive light source system, and an image acquisition card; the industrial camera array is annularly distributed, covering the product surface in all directions to achieve a 360° detection perspective; the adaptive light source system dynamically adjusts the multi-band light combination according to the surface reflectivity of the product, and the spectral range covers 400 - 1100nm; the image acquisition card has the ability to process at least 8-channel camera signals in parallel.

[0054] Furthermore, the multi-modal image acquisition module aims to obtain product surface image information comprehensively and with high precision, providing a rich and accurate data basis for subsequent product quality inspection; its core functions include:

[0055] 360° Omnidirectional Image Acquisition: Through an array of high-resolution industrial cameras distributed in a ring, it realizes a 360° non-blind-zone shooting of the product surface, ensuring that the surface conditions of all parts of the product can be clearly recorded;

[0056] Adapting to Different Product Characteristics: With the help of an adaptive light source system, it dynamically adjusts the multi-band light combination according to the surface reflectivity of the product, effectively meeting the imaging requirements of products with different materials and surface roughnesses, and ensuring the clarity and contrast of the images;

[0057] High-Speed Data Acquisition and Transmission: The image acquisition card supports parallel processing of at least 8 camera signals, quickly acquires the image data captured by the cameras, and efficiently transmits it to the subsequent processing module to meet the requirements of real-time detection;

[0058] Its structural framework includes:

[0059] High-Resolution Industrial Camera Array: Composed of multiple high-resolution industrial cameras, placed around the product in a ring layout; these cameras have the characteristics of high pixels and high frame rates, and can capture detailed information such as fine textures and defects on the product surface, providing high-precision image materials for subsequent defect detection;

[0060] Adaptive Light Source System: Includes a light source controller and a multi-band light source component; the light source controller monitors the surface reflectivity of the product in real time, calculates the optimal light combination parameters according to the preset algorithm, and controls the multi-band light source component to output the corresponding light; its spectral range covers 400 - 1100nm, which can cover the visible light and part of the near-infrared light bands, adapting to the light absorption and reflection characteristics of different material products;

[0061] Image Acquisition Card: As a bridge connecting the camera array and the subsequent processing system, the image acquisition card is responsible for receiving signals from at least 8 cameras; it has strong parallel processing capabilities, can process multi-channel camera data simultaneously, convert the analog image signals captured by the cameras into digital signals, and perform preliminary data format sorting and preprocessing so that the subsequent modules can quickly read and process the image data;

[0062] Its operation steps include:

[0063] Initialization of the Camera Array: When the system starts, the high-resolution industrial camera array conducts self-check and parameter initialization, including adjusting parameters such as camera focal length, aperture, and exposure time to ensure that the cameras are in the best shooting state;

[0064] Adaptive Light Source Adjustment: When the adaptive light source system is turned on, the light source controller emits initial light to the product surface. At the same time, it monitors the intensity and spectral distribution of the reflected light on the product surface in real time and calculates the reflectivity of the product surface. Based on the reflectivity data, the light source controller adjusts the light intensity and combination of the multi-band light source component to obtain the most suitable lighting conditions on the product surface;

[0065] Image Acquisition: After the light source adjustment is completed, the high-resolution industrial camera array takes 360° circumferential shots of the product according to the preset shooting frequency and angle. Each camera transmits the captured image signal to the image acquisition card;

[0066] Data Transmission and Preprocessing: The image acquisition card receives multiple camera signals in parallel, converts the analog signals into digital signals, and performs preliminary preprocessing operations such as image denoising and contrast enhancement. The processed image data is transmitted to the deep learning defect detection module for subsequent analysis according to a specific data format and transmission protocol;

[0067] Its beneficial effects include:

[0068] Improved Detection Comprehensiveness: The 360° annular camera array layout avoids detection blind spots, greatly improves the reliability of the overall quality detection of products, and reduces the risk of missed detection. For example, when detecting complex-shaped mechanical parts, it can comprehensively capture potential defects on all sides of the parts;

[0069] Optimized Image Quality: The adaptive light source system dynamically adjusts the lighting according to the product surface characteristics, significantly improving the clarity and contrast of the captured images. For mirror-finish metal products, it can effectively avoid image blurring caused by excessive reflection. For plastic products with rough surfaces, it can enhance image details and make it easier to detect minor defects;

[0070] Improved Detection Efficiency: The parallel processing ability of the image acquisition card greatly shortens the image acquisition and transmission time, meeting the demand for real-time product quality detection on high-speed production lines. On the electronic product production line, it can quickly complete the image acquisition of products, ensuring the efficient operation of the production line.

[0071] In this embodiment, the deep learning defect detection module is used to implement end-to-end analysis from images to defect decisions, specifically including:

[0072] Dual-Path Residual Denoising Unit: Adopting a dual-branch network structure, it extracts the low-frequency structure information and high-frequency detail features of the image respectively, and fuses the denoising results through residual connections;

[0073] Multi-Scale Context Awareness Unit: With the help of the Dilated Convolution Pyramid Network (DCPN), it synchronously obtains local defect details and global context correlation features;

[0074] Graph Topological Defect Classification Unit: Construct a spatial relationship graph of the defect area, and use the Graph Convolutional Network (GCN) to iteratively update the node features, and finally output the joint probability of the defect type and severity;

[0075] Furthermore, the deep learning defect detection module is the core of the product quality detection system, and its main functions are as follows:

[0076] Image Denoising and Feature Extraction: The images collected in industrial production often contain noise. The dual-path residual denoising unit can remove noise interference, and at the same time extract the low-frequency structure information of the image with different convolutional kernels, such as the overall contour and general shape of the product, as well as high-frequency detail features, such as surface texture and minute defects, providing clear and valuable data for subsequent analysis;

[0077] Multi-scale Context Awareness: Judging product defects requires integrating local details and global information; the multi-scale context awareness unit uses the Dilated Convolution Pyramid Network (DCPN) to simultaneously capture local defect details and global context-related features at different scales; for example, when detecting a circuit board, it can not only focus on the minute solder joint voids of components, but also consider the overall circuit layout to avoid misjudgment caused by local analysis;

[0078] Defect Classification and Severity Assessment: The graph topological defect classification unit constructs a spatial relationship graph of the defect area, regards the defect as a graph node, determines the edge weights between nodes according to the spatial distance and feature similarity, and uses the Graph Convolutional Network (GCN) to iteratively update the node features, and finally outputs the joint probability of the defect type, such as scratches, cracks, holes, etc., and the severity, realizing the quantitative evaluation of the defect. Its specific structural framework includes:

[0079] The network structure of the dual-path residual denoising unit is:

[0080] Low-frequency Branch: Adopt a 5×5 large convolutional kernel to extract the macroscopic structure features of the image, and the number of output channels is 64;

[0081] High-frequency Branch: Adopt a 3×3 small convolutional kernel and combine it with a dilated convolutional layer with a dilation rate of 2 to extract microscopic texture features, and the number of output channels is 128;

[0082] Feature Fusion Layer: Perform weighted fusion on the outputs of the two branches through a cross-channel attention mechanism. The calculation formula is:

[0083] , where , is a learnable weight matrix used to perform weighted transformation on the input features; is the Sigmoid function, which maps the weighted result to between 0 and 1 to obtain the attention weight ; represents the features output by the low-frequency branch, Represents the features of the high-frequency branch output; Represents element-wise multiplication, multiplying the attention weights with the corresponding branch features and then summing them to obtain the fused features ;

[0084] The dilated convolutional pyramid network of the multi-scale context-aware unit includes:

[0085] Base feature layer: Extracts the underlying feature map with a resolution of 256×256 through a 3×3 standard convolution;

[0086] Multi-scale context layer: Parallelly deploys dilated convolutional layers with dilation rates of 2, 4, and 8, and the output feature map resolutions are 128×128, 64×64, and 32×32 respectively;

[0087] Feature refinement module: Adopts a bidirectional feature pyramid structure to fuse multi-scale features, and the calculation formula is:

[0088] ; where, Represents the transposed convolution operation, restoring the feature maps obtained by dilated convolutions with different dilation rates to the same resolution as the base feature map, which respectively correspond to the feature maps under different dilation rates; is element-wise multiplication, multiplying the transposed convolution feature map with the base feature map processed by the channel attention module element-wise; is the channel attention module, used to calculate the importance weights of each channel of the base feature map, highlight key information, and finally sum the three to obtain the refined feature ;

[0089] The working process of the graph topology defect classification unit includes:

[0090] Node feature initialization: Using the feature vector of the detected candidate defect region ROI as the graph node, with a feature dimension of 512;

[0091] Dynamic edge weight calculation: Based on the spatial distance and feature similarity to generate the adjacency matrix:

[0092] , where, is the distance attenuation coefficient; is the temperature parameter; represents the th and the th spatial distance between the defect regions, represents the the feature similarity between the first and the th defect regions; is an element of the adjacency matrix, representing the edge weight between the th and the

[0093] Multi-hop graph convolution: The node representations are iteratively updated through a 3-layer graph convolutional network, with the output dimensions of each layer being 256, 128, and 64 respectively; the graph convolutional network aggregates the information of neighboring nodes, gradually explores the internal connections of defect features, and realizes the classification and evaluation of defect types and severities.

[0094] The operation steps of the deep learning defect detection module are as follows:

[0095] Image input and denoising: The image acquired by the multi-modal image acquisition module is input into the dual-path residual denoising unit. The low-frequency and high-frequency branches work simultaneously, extract features respectively, and perform weighted fusion at the feature fusion layer to remove noise and obtain a feature-rich image representation.

[0096] Multi-scale feature extraction and fusion: The denoised image representation enters the multi-scale context-aware unit. First, the underlying features are extracted at the basic feature layer, then the multi-scale context features are obtained through dilated convolutional layers with different dilation rates, and finally, they are fused and refined in the feature refinement module to obtain a feature representation containing rich context information.

[0097] Defect classification and evaluation: The above features are input into the graph topology defect classification unit. First, the node features are initialized, then the dynamic edge weights are calculated to generate the adjacency matrix, and finally, the node features are iteratively updated through multi-hop graph convolution, and the joint probability of defect types and severities is output to complete defect classification and severity evaluation.

[0098] The deep learning defect detection module can achieve:

[0099] High-accuracy detection: The multi-scale context-aware and graph topology defect classification technologies enable the module to accurately identify various defects. Compared with traditional detection methods, the accuracy is significantly improved, the false detection and missed detection rates are reduced, and the product quality is enhanced.

[0100] Strong robustness: The dual-path residual denoising unit effectively improves the model's ability to process noisy images, enabling the model to operate stably in complex industrial environments and ensuring reliable detection results.

[0101] Wide adaptability: This module can handle defect detection tasks for products of different shapes and materials, with strong versatility and adaptability, and can be widely applied to various industrial production scenarios.

[0102] In this embodiment, the dynamic resource scheduling module generates a multi-objective optimization scheduling plan based on the defect detection results. The dynamic resource scheduling module is a key part for the efficient operation of the product quality detection system, and mainly has the following functions:

[0103] Optimize personnel scheduling: According to the defect types given by the deep learning defect detection module, accurately match the skill map database of maintenance personnel; by calculating the skill matching degree, reasonably arrange maintenance personnel so that each defect can be handled by the most suitable personnel, improving the maintenance efficiency and quality;

[0104] Plan material scheduling: Combine the defect location coordinates and the production line layout topology map, comprehensively consider the transportation time and energy consumption, and use the ant colony optimization algorithm to plan the optimal path for material transportation; ensure that the maintenance materials are delivered in a timely and efficient manner while reducing the logistics cost;

[0105] Balance the production line throughput: Based on the Markov decision process, dynamically adjust the resource allocation ratio between the detection station and the maintenance station; according to the actual situation of each link on the production line, ensure the coordination of the production rhythm of the production line, avoid the idleness or congestion of station resources, and maximize the overall throughput of the production line;

[0106] Furthermore, the structural framework of the dynamic resource scheduling module includes:

[0107] The personnel scheduling unit driven by the skill map: It contains the skill map database of maintenance personnel, which stores information such as the skills of maintenance personnel, work experience, and the defect types they are good at handling. Through the matching algorithm between the defect type and the skill map, quickly calculate the skill matching degree between the maintenance personnel and the current defect handling task, and generate a priority queue to provide a basis for personnel scheduling, where:

[0108] Calculation of skill matching degree: Let the skill vector of the maintenance personnel be , where represents the ability level of the th maintenance personnel in the th skill item, ; the skill requirement vector corresponding to the defect type is , represents the requirement degree of the th skill item required to handle this defect type; use the cosine similarity algorithm to calculate the skill matching degree , and the formula is:

[0109] , where the numerator is the dot product of the two vectors, measuring the total matching degree of the maintenance personnel's skills and the skills required by the defect in each dimension; the denominators are the modulus lengths of the maintenance personnel's skill vector and the defect skill requirement vector respectively, used to normalize the numerator so that The value is between and the larger the value, the higher the skill matching degree;

[0110] Space-time constrained material scheduling unit: Based on the production line layout topology map and defect location coordinates as basic data, an ant colony optimization algorithm module is built in; The algorithm module simulates the foraging behavior of ants to search for the best path for material transportation according to optimization objectives such as transportation time and energy consumption cost, and at the same time considers time and space constraints to ensure reasonable and efficient material transportation;

[0111] Optimization objective function: The optimization objective is to minimize the weighted sum of transportation time and energy consumption cost, and at the same time consider the impact of the maximum transportation time on the whole. The objective function is:

[0112] , where is the transportation time of the th AGV; is the energy consumption cost of the th AGV; , are the weight coefficients of transportation time and energy consumption cost respectively; is the weight coefficient used to adjust the impact of the maximum transportation time on the optimization objective; is the number of AGVs;

[0113] Solve the Pareto optimal solution set by improving the ant colony algorithm. In each iteration, the probability that the ant transfers from node to node is calculated as:

[0114] ;

[0115] Among them, is the pheromone concentration on the edge . The higher the pheromone concentration, the more favored this path is by ants; is the heuristic information of the edge , usually inversely proportional to the distance between two points. The shorter the distance, the greater the heuristic information; and represent the relative importance of pheromone and heuristic information respectively, the larger, the more inclined the ant is to choose the path with high pheromone concentration, the larger, the more inclined the ant is to choose the path with short distance (large heuristic information); represents the set of nodes that the ant is allowed to choose next, that is, the nodes that the ant can directly reach from the current position;

[0116] Production line throughput balancing unit: Using the Markov decision model, it monitors in real time information such as the working status and task volume of the inspection station and the maintenance station; this information will be integrated and used to define the state of the system at a certain moment; the set of system states is represented by , where each specific state is a comprehensive description of the task volume, equipment status, personnel allocation, etc. of the inspection station and the maintenance station, that is ; the set of actions is denoted as , and the action represents an adjustment operation for the resource allocation of the inspection station and the maintenance station, such as increasing the running time of inspection equipment, deploying maintenance personnel, etc.; the probability of taking action in state and transferring to state is , and the immediate reward is , which represents the immediate benefit obtained when taking action in state and transferring to state , for example, the increase in throughput of the production line under this state transition;

[0117] Solve the optimal policy through the Bellman equation;

[0118] ;

[0119] Among them, is the optimal value function of state , which represents the maximum cumulative reward that can be obtained starting from state and following the optimal policy ; is the discount factor, and its value range is between , indicating the importance of future rewards. The closer it is to , the more attention is paid to future rewards, and vice versa, more attention is paid to immediate rewards;

[0120] Furthermore, the operation steps of the dynamic resource scheduling module include:

[0121] Execute personnel scheduling: After the deep learning defect detection module identifies the product defect and determines its type, the skill graph-driven personnel scheduling unit is activated; first, read the skill graph information of all maintenance personnel from the database, match the defect type with the skill graph one by one, and calculate the skill matching degree according to the above skill matching degree formula; generate a priority queue from high to low according to the matching degree, and the system arranges maintenance personnel to handle the defect according to the queue order;

[0122] Implement material scheduling: After determining the defect location, the spatio-temporal constrained material scheduling unit starts to work. First, obtain the topological map of the production line layout to clarify the positional relationship between each work station and the material storage points. Combine the defect location coordinates, and use the above ant colony optimization algorithm for path planning with the transportation time and energy consumption cost as the optimization objectives. The algorithm iteratively finds the optimal material transportation path, and then controls the transportation equipment to transport materials according to the planned path.

[0123] Adjust the production line throughput balance: The production line throughput balance unit continuously collects real-time data from the inspection work station and the maintenance work station, including the number of completed tasks, the current progress of task processing, the equipment idle time, etc. Determine the current state of the system based on this data. According to the Markov decision process, use the Bellman equation to calculate the value function after taking different actions (such as adjusting the equipment operation time, allocating manpower, etc.) in the current state, and select the action with the maximum value as the optimal resource allocation plan to achieve the dynamic balance of the production line throughput.

[0124] The beneficial effects of the dynamic resource scheduling module include:

[0125] Improve the maintenance efficiency: Precise personnel scheduling ensures that each defect can be quickly processed by maintenance personnel with corresponding skills, reduces the maintenance waiting time, improves the maintenance accuracy and efficiency, and reduces the production delay caused by untimely maintenance of products.

[0126] Reduce the logistics cost: The optimized material transportation path reduces the transportation time and energy consumption, reduces the logistics cost, improves the timeliness of material distribution at the same time, and ensures the continuity of production.

[0127] Enhance the overall efficiency of the production line: Dynamically balance the resource allocation between the inspection work station and the maintenance work station, avoid resource waste and production bottlenecks, significantly improve the production efficiency of the entire production line, and improve the production benefits of the enterprise.

[0128] In this embodiment, the augmented reality interaction module includes a holographic projection device and a multimodal interaction terminal, and superimposes the three-dimensional thermal defect map, the maintenance path navigation and the process parameter correction suggestions in real time.

[0129] Furthermore, the augmented reality interaction module is described in detail below:

[0130] The augmented reality interaction module builds an efficient communication bridge between people, inspection data and equipment in the product quality inspection system, and mainly has the following functions:

[0131] Defect Visual Display: Through a holographic projection device, the defects detected in the product are visually presented in the form of a three-dimensional heat map, enabling operators to quickly and accurately locate the defect positions and understand the severity distribution of the defects; different colored and bright areas represent different degrees of defects. For example, the red area indicates severe defects, the yellow area indicates moderate defects, and the green area indicates minor defects.

[0132] Maintenance Navigation Assistance: In combination with a multi-modal interaction terminal, it provides maintenance path navigation for maintenance personnel, guiding them to quickly reach the defect location, and providing detailed maintenance operation step guidelines according to the defect type and product structure, reducing maintenance time and incorrect operations.

[0133] Real-time Parameter Adjustment: Based on the detection results and maintenance requirements, it displays real-time process parameter correction suggestions on the multi-modal interaction terminal. Operators can directly adjust the parameters on the terminal to achieve immediate optimization of the production process and improve product quality.

[0134] Holographic Projection Device: Composed of a projector, an optical lens group, an image processor, etc.; the image processor receives defect data from the deep learning defect detection module, converts it into a three-dimensional image format, and then projects the three-dimensional heat map onto the product or the relevant operation space through the projector and the optical lens group to achieve three-dimensional display of the defects.

[0135] Multi-modal Interaction Terminal: Integrated with a touch display screen, a voice interaction module, a gesture recognition sensor, etc.; the touch display screen is used to display information such as maintenance paths and process parameter suggestions. Operators can confirm and adjust through touch operations; the voice interaction module supports voice command input and voice prompt output, facilitating interaction when the operator's hands are busy; the gesture recognition sensor captures the operator's gesture actions to achieve non-contact interaction, improving the convenience and efficiency of operations.

[0136] Data Reception and Processing: The augmented reality interaction module obtains product defect information from the deep learning defect detection module, including defect positions, types, severity, etc.; the image processor of the holographic projection device converts this data into image data of a three-dimensional heat map, and the multi-modal interaction terminal sorts out the received data such as maintenance paths and process parameter correction suggestions.

[0137] Defect Display and Interaction Guidance: The holographic projection device projects the generated three-dimensional heat map onto the corresponding position of the product to clearly display the defect distribution; at the same time, the multi-modal interaction terminal displays the maintenance path and operation steps on the touch display screen, and the voice interaction module gives voice prompts to guide the operator to perform the next operation.

[0138] Interaction operations and feedback: Operators interact with the multimodal interaction terminal according to the displayed information through touch screen, voice commands or gesture actions; for example, click to confirm repair steps, input new process parameters, zoom in or out the defect display area through gestures, etc.; the interaction terminal feeds back the operation information to the system, and the system adjusts the display content or executes corresponding control commands according to the feedback.

[0139] Beneficial effects

[0140] Improve detection and repair efficiency: Intuitive defect display and repair navigation enable operators to quickly locate and handle defects, greatly shortening the detection and repair time and improving production efficiency;

[0141] Reduce operation difficulty: Multimodal interaction methods adapt to different operation scenarios and operator habits, reducing the skill requirements for operators and the probability of incorrect operations;

[0142] Optimize the production process: Real-time process parameter correction suggestions help operators adjust the production process in a timely manner, improve product quality and reduce the defective rate.

[0143] In this embodiment, it further includes:

[0144] Self-evolving detection model: Adopt an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, specifically including:

[0145] Graph convolutional layer weight matrix: Update the learnable parameter matrix of each layer of graph convolutional operation , where and are the feature dimensions of the input and output of the th layer respectively;

[0146] Dynamic edge weight generation parameters: Update the distance attenuation coefficient and temperature parameter in the calculation of the adjacency matrix to optimize the spatial relationship modeling between defect regions;

[0147] Attention mechanism parameters: If the attention mechanism is introduced in the classification unit, update the attention weight matrix and linear transformation parameter , ; where and represent the input and output channel dimensions in the attention mechanism respectively;

[0148] The weight update is achieved through the following formula:

[0149] , where is the current batch of data, is a sample from the historical memory bank, ; is the gradient of the loss function of the -th sample in the historical memory bank with respect to the network weights ; is the learning rate parameter that controls the step size for updating the weights based on the current batch of data; is the learning rate parameter that controls the step size for updating the weights based on the samples in the historical memory bank, so as to obtain the network weights at time ;

[0150] Multi-physics field simulation module: Integrates a finite element analysis engine to simulate the impact of defects on the mechanical properties of products and generate a contour map of the residual stress distribution;

[0151] Furthermore, the specific operation steps of the self-evolving detection model are as follows:

[0152] Data preparation: Collect new detection sample data, organize it into a format that meets the model input requirements, and form the current batch of data ; At the same time, extract samples from the historical memory bank , , These historical samples record various data during previous detection processes and are used to assist in the update of the model, enabling it to be optimized by combining past experience and new data;

[0153] Calculate the gradient of the current batch of data: Input the current batch of data into the network of the deep learning defect detection module, and calculate the gradient of the loss function of the current batch of data with respect to the network weights according to the loss function set by the model ; This gradient reflects the direction and degree of adjustment of the current batch of data to the network weights and can be efficiently calculated through the backpropagation algorithm;

[0154] Calculate the gradient of the historical memory bank samples: For each sample in the historical memory bank, also input it into the network and calculate the gradient of its loss function with respect to the network weights ; These gradients reflect the impact of historical samples on weight adjustment. Considering the gradients of historical samples comprehensively can prevent the model from forgetting useful features learned previously;

[0155] Update the network weights: According to the calculated gradients of the current batch of data and the historical memory bank samples, update the network weights according to the update formula , where is the current batch of data, ​is a sample in the historical memory bank, ; is the gradient of the loss function for the th sample in the historical memory bank with respect to the network weights ; is the learning rate parameter that controls the step size for updating the weights based on the current batch of data; is the learning rate parameter that controls the step size for updating the weights based on the samples in the historical memory bank, so as to obtain the network weights at time , enabling the model to continuously adapt to new detection tasks and data changes, and improving the accuracy and adaptability of defect detection;

[0156] Furthermore, the specific operation steps of the multi-physics field simulation module are as follows:

[0157] Pre-data collection and preparation: Obtain the defect information of the product from the deep learning defect detection module, including key data such as the location, size, and shape of the defects; at the same time, collect the material property data of the product, such as elastic modulus, Poisson's ratio, yield strength, etc. These material parameters are crucial for accurately simulating the mechanical properties of the product; in addition, it is necessary to determine the geometric model of the product, which can be the original model designed by CAD software or the three-dimensional model reconstructed based on the actual product scan;

[0158] Establish a multi-physics field simulation model: Use the finite element analysis engine to mesh the geometric model of the product, discretize the continuous solid model into a combination of a finite number of elements for subsequent numerical calculations; during meshing, it is necessary to reasonably adjust the mesh density according to the product structure and defect location, and use finer meshes near the defects and in the stress concentration areas to improve the calculation accuracy; then, according to the collected material property data, define the material parameters in the simulation model to ensure that the model can accurately reflect the actual material characteristics of the product; next, according to the actual working conditions of the product, apply corresponding boundary conditions, such as fixed constraints, loads, temperatures, etc., to simulate the stress and working state of the product in the real environment;

[0159] Execute the simulation calculation: After setting up the simulation model and parameters, start the finite element analysis engine to perform multi-physics field simulation calculations; during the calculation process, the engine will solve the partial differential equations describing the mechanical properties of the product according to the set algorithm, and simulate the influence of defects on the internal stress and strain distribution of the product under the action of various physical fields; it is necessary to monitor the calculation status in real time during the calculation process to ensure the stability and convergence of the calculation. If an abnormality occurs, it is necessary to adjust the model parameters in time or re-set them;

[0160] Result Analysis and Visualization: After the simulation calculation is completed, obtain the calculation result data, including information such as the stress distribution, strain distribution, and displacement inside the product; use professional post-processing software to analyze the result data and extract key indicators, such as the maximum stress value, stress concentration area, deformation amount, etc.; by generating visualization charts such as the residual stress distribution cloud diagram, visually display the impact of defects on the mechanical properties of the product, enabling engineers to clearly understand the weak links and potential risks of the product, and providing a strong basis for product quality assessment and improvement; according to the analysis results, further compare with the design standards and quality requirements of the product to determine whether the product is qualified and whether it is necessary to optimize and adjust the product design or production process.

[0161] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A product quality inspection system based on deep learning and machine vision, characterized in that: Including: Multimodal image acquisition module: It consists of a high-resolution industrial camera array, an adaptive light source system, and an image acquisition card; the industrial camera array is distributed in a ring shape, covering the product surface comprehensively to achieve a 360° detection perspective; the adaptive light source system dynamically adjusts the multi-band light combination according to the surface reflectivity of the product, and the spectral range covers 400 - 1100nm; the image acquisition card has the ability to process at least 8 channels of camera signals in parallel; Deep learning defect detection module: Used to achieve end-to-end analysis from images to defect decisions, specifically including: Dual-path residual denoising unit: Adopts a dual-branch network structure to extract the low-frequency structure information and high-frequency detail features of the image respectively, and fuses the denoising results through residual connections; Multi-scale context awareness unit: With the help of the dilated convolutional pyramid network DCPN, synchronously obtains local defect details and global context correlation features; Graph topology defect classification unit: Constructs a spatial relationship graph of the defect area, and uses the graph convolutional network GCN to iteratively update the node features, and finally outputs the joint probability of the defect type and severity; Dynamic resource scheduling module: Generates a multi-objective optimization scheduling plan based on the defect detection results, covering: Skill graph-driven personnel scheduling unit: According to the defect type, matches the skill graph database of maintenance personnel, and generates a priority queue by calculating the skill matching degree; Spatio-temporal constrained material scheduling unit: Combines the defect position coordinates and the production line layout topology map, and uses the ant colony optimization algorithm to plan the material transportation path; Production line throughput balancing unit: Adopts the Markov decision process to dynamically adjust the resource allocation ratio between the detection station and the maintenance station; Augmented reality interaction module: Includes a holographic projection device and a multimodal interaction terminal, and overlays the 3D thermal map of the defect, the maintenance path navigation, and the process parameter correction suggestions in real time; The network structure of the dual-path residual denoising unit is: Low-frequency branch: Uses a 5×5 large convolutional kernel to extract the macroscopic structure features of the image, and the output channel number is 64; High-frequency branch: Uses a 3×3 small convolutional kernel and combines a dilated convolutional layer with a dilation rate of 2 to extract microscopic texture features, and the output channel number is 128; Feature fusion layer: Performs weighted fusion on the outputs of the two branches through a cross-channel attention mechanism, and the calculation formula is: , where , is a learnable weight matrix for weighted transformation of input features; is the Sigmoid function that maps the weighted result to between 0 and 1 to obtain the attention weights ; represents the features output by the low-frequency branch, represents the features output by the high-frequency branch; represents element-wise multiplication, and after multiplying the attention weights by the corresponding branch features and adding them together, the fused features are obtained.

2. The product quality inspection system based on deep learning and machine vision according to claim 1, characterized in that: The dilated convolutional pyramid network of the multi-scale context awareness unit includes: Basic feature layer: Extracts the underlying feature map with a resolution of 256×256 through a 3×3 standard convolution; Multi-scale context layer: Parallelly deploys dilated convolutional layers with dilation rates of 2, 4, and 8, and the output feature map resolutions are 128×128, 64×64, and 32×32 respectively; Feature refinement module: Adopts a bidirectional feature pyramid structure to fuse multi-scale features, and the calculation formula is: ; among them, represents a deconvolution operation that restores the feature maps obtained by dilated convolutions with different dilation rates to the same resolution as the base feature map, , corresponding to the feature maps at different dilation rates respectively; is an element-wise multiplication that multiplies the deconvolved feature map with the base feature map processed by the channel attention module element-wise; is the channel attention module, which is used to calculate the importance weights of each channel of the base feature map, highlight key information, and finally add the three to obtain the refined feature .

3. The product quality inspection system based on deep learning and machine vision according to claim 2, wherein: The working process of the graph topology defect classification unit includes: Node feature initialization: Uses the feature vector of the detected candidate defect area ROI as the graph node, and the feature dimension is 512; Dynamic edge weight calculation: Based on the spatial distance between defect regions and feature similarity Generate an adjacency matrix: , where is the distance attenuation coefficient; is the temperature parameter; represents the th and th spatial distance between defective regions, represents the th and th feature similarity of defective regions; is an element of the adjacency matrix, representing the edge weight between the th and th nodes; k is an index variable used to traverse all possible nodes; Multi-hop graph convolution: Iteratively updates the node representation through a 3-layer graph convolutional network, and the output dimensions of each layer are 256, 128, and 64 respectively.

4. The product quality inspection system based on deep learning and machine vision according to claim 3, characterized in that: The optimization objective function of the spatio-temporal constrained material scheduling unit is: , where is the transportation time of the th AGV; is the energy consumption cost of the th AGV; and are the weight coefficients of transportation time and energy consumption cost respectively; is the weight coefficient used to adjust the influence of the maximum transportation time on the optimization objective; is the number of AGVs. The Pareto optimal solution set is solved by improving the ant colony algorithm, respectively represent the transportation times of the 5. The product quality inspection system based on deep learning and machine vision according to claim 4, characterized in that: Also includes: Self-evolving detection model: Adopt an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, specifically including: Graph Convolutional Layer Weight Matrix: Update the learnable parameter matrix for each layer of graph convolutional operations , where and are the feature dimensions of the input and output of the th layer, respectively; Dynamic edge weight generation parameters: Update the distance decay coefficient in the adjacency matrix calculation and the temperature parameter , to optimize the spatial relationship modeling between defect regions; Attention mechanism parameters: If an attention mechanism is introduced in the classification unit, update the attention weight matrix and the linear transformation parameters , where and respectively represent the channel dimensions of the input and output in the attention mechanism; The weight update is achieved through the following formula: , where is the data of the current batch, is the sample in the historical memory bank, ; is the gradient of the loss function of the th sample in the historical memory bank with respect to the network weight ; is the learning rate parameter that controls the step size for updating the weights based on the data of the current batch; is the learning rate parameter that controls the step size for updating the weights based on the samples in the historical memory bank, so as to obtain the network weight at time ; Multi-physics field simulation module: Integrate a finite element analysis engine to simulate the impact of defects on the mechanical properties of products and generate a contour map of residual stress distribution.

Images