Product quality detection system based on deep learning and machine vision

Abstract

The invention relates to the technical field of product quality inspection, in particular to a product quality detection system based on deep learning and machine vision, which comprises a multi-modal image acquisition module, a deep learning defect detection module and a dynamic resource scheduling module. The multi-modal image acquisition module is used for acquiring product images in all directions by using a high-resolution camera array and a self-adaptive light source; the deep learning defect detection module accurately identifies defects through dual-path residual denoising, multi-scale context sensing and graph topology defect classification; the dynamic resource scheduling module realizes reasonable scheduling of personnel and materials and balance of a production line according to a detection result, a skill map, an ant colony algorithm and the like; the augmented reality interaction module visually displays defects and provides maintenance navigation through holographic projection and multi-mode interaction; the self-evolution detection model can update the weight based on a new sample, and the multi-physics field simulation module simulates the influence of defects on the mechanical property of a product. The detection precision is high, the efficiency is high, and the product quality detection and production management level is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of product quality inspection, and specifically to a product quality inspection system based on deep learning and machine vision. Background Art

[0002] In the field of industrial manufacturing, product quality is the lifeline of an enterprise. With the rapid development of the manufacturing industry, the market's requirements for product quality have become increasingly stringent. Traditional product quality testing methods have gradually exposed many problems and are difficult to meet the needs of modern production. This invention was born in this context.

[0003] Disadvantages of traditional personnel scheduling methods: In the past, after product defects were discovered during the production process, personnel scheduling often relied on manual experience and judgment. Managers arranged maintenance personnel based on their general understanding of employee skills and their subjective understanding of the defect situation. This method lacks systematicity and accuracy, and it is easy for the maintenance personnel's skills to not match the defect type. For example, in the manufacture of electronic equipment, when a welding defect is detected on a circuit board, personnel who are not good at electronic welding maintenance may be assigned to handle it, resulting in low maintenance efficiency and difficulty in ensuring maintenance quality. Improper maintenance may even cause product scrapping, increasing production costs.

[0004] The dual challenges of production efficiency and quality: As the pace of production accelerates, rapid response and efficient handling of product defects become critical. However, traditional personnel scheduling methods cannot quickly make optimal arrangements based on defect detection results. On large-scale production lines, if suitable personnel cannot be deployed to defect handling positions in a timely manner, production stagnation time will be extended, affecting overall production efficiency. At the same time, due to the mismatch of maintenance personnel's skills, defects may not be completely resolved, causing problems to occur again in subsequent links, reducing product quality and damaging corporate reputation.

[0005] Product diversification and increasing complexity of defects: Nowadays, there are many types of products and complex production processes, and the corresponding defect types have become more diverse and complex. Defects of different products may involve different areas of professional knowledge and skills. For example, in automobile manufacturing, body paint defects and engine parts defects require personnel with different professional backgrounds and skill levels to handle. Traditional personnel scheduling methods are difficult to cope with such complex situations. It is impossible to quickly and accurately select the most suitable candidates from a large number of maintenance personnel, resulting in a longer defect handling cycle and serious impact on production efficiency and quality.

[0006] Therefore, to address the above problems, a product quality inspection system based on deep learning and machine vision is proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a product quality inspection system based on deep learning and machine vision to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions: A product quality inspection system based on deep learning and machine vision, comprising: Multimodal image acquisition module: It consists of a high-resolution industrial camera array, an adaptive light source system and an image acquisition card. The industrial camera array is distributed in a ring shape, covering the product surface in all directions and achieving a 360° detection angle. The adaptive light source system dynamically adjusts the multi-band illumination combination according to the product surface reflectivity, and the spectral range covers 400-1100nm. The image acquisition card has the ability to process at least 8 camera signals in parallel. Deep learning defect detection module: used to achieve end-to-end analysis from image to defect decision, including: Dual-path residual denoising unit: It uses a dual-branch network structure to extract low-frequency structural information and high-frequency detail features of the image respectively, and fuses the denoising results through residual connections; Multi-scale context-aware unit: With the help of the dilated convolutional pyramid network (DCPN), local defect details and global context-related features are obtained simultaneously; Graph topology defect classification unit: constructs a spatial relationship graph of defect areas, uses graph convolutional network (GCN) to iteratively update node features, and finally outputs the joint probability of defect type and severity; Dynamic resource scheduling module: Generates multi-objective optimization scheduling solutions based on defect detection results, covering: Skill map-driven personnel scheduling unit: Match the maintenance personnel skill map database according to the defect type, and generate a priority queue by calculating the skill matching degree; Time-space constraint material scheduling unit: Combines the defect location coordinates with the production line layout topology map and uses the ant colony optimization algorithm to plan the material transportation path; Production line throughput balancing unit: uses Markov decision process to dynamically adjust the resource allocation ratio of inspection stations and maintenance stations; Augmented reality interaction module: includes holographic projection equipment and multimodal interactive terminals, which can superimpose three-dimensional thermal maps of defects, maintenance path navigation and process parameter correction suggestions in real time.

[0009] As a preferred solution, the network structure of the dual-path residual denoising unit is: Low-frequency branch: A large 5×5 convolution kernel is used to extract image macroscopic structural features, and the number of output channels is 64; High-frequency branch: A 3×3 small convolution kernel is used, combined with a dilated convolution layer with a dilation rate of 2 to extract micro-texture features, and the number of output channels is 128; Feature fusion layer: weighted fusion of dual-branch outputs through cross-channel attention mechanism, the calculation formula is: ,in, , is a learnable weight matrix used to perform weighted transformation on input features; Sigmoid function maps the weighted result to between 0 and 1 to obtain the attention weight ; represents the characteristics of the low-frequency branch output, Represents the characteristics of the high-frequency branch output; Represents element-by-element multiplication, and the attention weight Multiply and add the corresponding branch features to get the fused features .

[0010] As a preferred solution, the hole convolutional pyramid network of the multi-scale context-aware unit includes: Basic feature layer: extract the underlying feature map with a resolution of 256×256 through 3×3 standard convolution; Multi-scale context layer: The dilated convolution layers with expansion rates of 2, 4, and 8 are deployed in parallel, and the output feature map resolutions are 128×128, 64×64, and 32×32 respectively; Feature refinement module: A bidirectional feature pyramid structure is used to fuse multi-scale features. The calculation formula is: ;in, Represents the deconvolution operation, which is the feature map obtained by convolution with different dilation rates. Restore to the same resolution as the base feature map, , corresponding to the characteristic graphs under different expansion rates; To multiply element by element, the deconvolution feature map is combined with the channel attention module Processed basic feature map Element-wise multiplication; The channel attention module is used to calculate the importance weight of each channel of the basic feature map, highlight the key information, and finally add the three together to obtain the refined feature .

[0011] As a preferred solution, the workflow of the graph topology defect classification unit includes: Node feature initialization: The detected candidate defect region ROI feature vector is used as a graph node with a feature dimension of 512; Dynamic edge weight calculation: based on the spatial distance between defect areas and feature similarity Generate an adjacency matrix: ,in, is the distance attenuation coefficient; is the temperature parameter; Indicates and The spatial distance between the defect areas, Indicates and The feature similarity of the defect areas; is the adjacency matrix element, indicating the and The edge weights between nodes; k is the index variable used to traverse all possible nodes.

[0012] Multi-hop graph convolution: The node representation is iteratively updated through a 3-layer graph convolutional network, with the output dimensions of each layer being 256, 128, and 64 respectively.

[0013] As a preferred solution, the optimization objective function of the time-space constraint material scheduling unit is: ,in, For the AGV transportation time; For the Energy consumption cost of an AGV; , are the weight coefficients of transportation time and energy consumption cost respectively; is the weight coefficient, which is used to adjust the impact of the maximum transportation time on the optimization target; is the number of AGVs, and the Pareto optimal solution set is solved by improving the ant colony algorithm. Respectively The transportation time of one AGV.

[0014] As a preferred solution, it also includes: Self-evolving detection model: It uses an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, including: Graph convolution layer weight matrix: Update the learnable parameter matrix of each graph convolution operation ,in, and Respectively The feature dimensions of the layer inputs and outputs; Dynamic edge weight generation parameters: Update distance decay coefficient in adjacency matrix calculation and temperature parameters , to optimize the spatial relationship modeling between defect areas; Attention mechanism parameters: If the attention mechanism is introduced in the classification unit, the attention weight matrix is ​​updated And the linear transformation parameters , ; and Respectively represent the channel dimensions of input and output in the attention mechanism; The weight update is implemented by the following formula: ,in, is the current batch data, For historical memory bank samples, ; For the historical memory Samples about network weights The gradient of the loss function; is the learning rate parameter, which controls the step size of updating the weights based on the current batch of data; is the learning rate parameter, which controls the step size of updating weights based on historical memory samples, thus obtaining The network weight at the moment ; Multi-physics simulation module: Integrates finite element analysis engine to simulate the impact of defects on product mechanical properties and generate residual stress distribution cloud map.

[0015] It can be seen from the technical solutions provided by the present invention that the product quality inspection system based on deep learning and machine vision provided by the present invention has the following beneficial effects: 1. The dynamic resource scheduling module uses the skill map-driven personnel scheduling unit to reasonably arrange maintenance personnel according to the defect detection results, the time-space constraint material scheduling unit plans the optimal material transportation path, and the production line throughput balancing unit dynamically adjusts the detection and maintenance station resource allocation, which improves maintenance efficiency, reduces logistics costs, avoids idle or congested production line resources, and makes the entire production line run efficiently; 2. The multimodal image acquisition module uses the annular distribution of the high-resolution industrial camera array to achieve 360° detection of the product surface without blind spots. The adaptive light source system dynamically adjusts the lighting according to the reflectivity of the product surface to ensure that the acquired image is clear and comprehensive, providing a high-quality data foundation for subsequent defect detection, and effectively avoiding missed detection due to limited detection angle and poor image quality; 3. The deep learning defect detection module uses a dual-path residual denoising unit to remove image noise, a multi-scale context perception unit to capture local and global features, and a graph topology defect classification unit to accurately classify defect types and assess severity, significantly improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1This is a schematic diagram of the overall structure of a product quality inspection system based on deep learning and machine vision in the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figure 1 As shown, an embodiment of the present invention provides a product quality inspection system based on deep learning and machine vision, including a multimodal image acquisition module, a deep learning defect detection module, a dynamic resource scheduling module and an augmented reality interaction module.

[0020] In this embodiment, the multimodal image acquisition module is composed of a high-resolution industrial camera array, an adaptive light source system and an image acquisition card; the industrial camera array is distributed in a ring shape, covering the product surface in all directions, and realizing a 360° detection viewing angle; the adaptive light source system dynamically adjusts the multi-band illumination combination according to the reflectivity of the product surface, and the spectral range covers 400-1100nm; the image acquisition card has the ability to process at least 8 camera signals in parallel; Furthermore, the multimodal image acquisition module aims to obtain product surface image information in an all-round and high-precision manner, providing a rich and accurate data basis for subsequent product quality inspection; its core functions include: 360° all-round image acquisition: Through the circularly distributed high-resolution industrial camera array, 360° shooting of the product surface without blind spots is achieved, ensuring that the surface conditions of all parts of the product can be clearly recorded; Adapt to different product characteristics: With the help of the adaptive light source system, the multi-band lighting combination is dynamically adjusted according to the surface reflectivity of the product, effectively responding to the imaging needs of products with different materials and different surface roughness, and ensuring the clarity and contrast of the image; High-speed data acquisition and transmission: The image acquisition card supports parallel processing of at least 8-channel camera signals, quickly acquires image data taken by the camera, and efficiently transmits it to the subsequent processing module to meet real-time detection requirements; Its structural framework includes: High-resolution industrial camera array: It consists of multiple high-resolution industrial cameras placed around the product in a circular layout. These cameras have high pixel and high frame rate characteristics, and can capture subtle textures, defects and other details on the product surface, providing high-precision image materials for subsequent defect detection. Adaptive light source system: includes light source controller and multi-band light source components; the light source controller monitors the product surface reflectivity in real time, calculates the optimal lighting combination parameters according to the preset algorithm, and controls the multi-band light source components to output corresponding light; its spectral range covers 400-1100nm, which can cover visible light and part of the near-infrared light band, and adapt to the absorption and reflection characteristics of light of products made of different materials; Image acquisition card: As a bridge between the camera array and the subsequent processing system, the image acquisition card is responsible for receiving signals from at least 8 cameras. It has powerful parallel processing capabilities and can process data from multiple cameras at the same time, convert the analog image signals collected by the camera into digital signals, and perform preliminary data formatting and preprocessing so that subsequent modules can quickly read and process image data. The operation steps include: Camera array initialization: When the system starts, the high-resolution industrial camera array performs self-checking and parameter initialization, including adjusting the camera's focal length, aperture, exposure time and other parameters to ensure that the camera is in the best shooting state; Adaptive light source adjustment: When the adaptive light source system is turned on, the light source controller emits initial light to the product surface, and monitors the intensity and spectral distribution of the reflected light on the product surface in real time, and calculates the reflectivity of the product surface; based on the reflectivity data, the light source controller adjusts the light intensity and combination of the multi-band light source components to obtain the most suitable lighting conditions for the product surface; Image acquisition: After the light source is adjusted, the high-resolution industrial camera array performs 360° surround shooting of the product according to the preset shooting frequency and angle; each camera transmits the captured image signal to the image acquisition card; Data transmission and preprocessing: The image acquisition card receives multi-channel camera signals in parallel, converts analog signals into digital signals, and performs preliminary preprocessing operations such as image denoising and contrast enhancement. The processed image data is transmitted to the deep learning defect detection module for subsequent analysis according to a specific data format and transmission protocol. Its beneficial effects include: Improved comprehensiveness of inspection: The 360° circular camera array layout avoids blind spots in inspection, greatly improves the reliability of overall product quality inspection, and reduces the risk of missed inspections; for example, when inspecting mechanical parts with complex shapes, it can fully capture potential defects on all surfaces of the parts; Image quality optimization: The adaptive light source system dynamically adjusts the lighting according to the surface characteristics of the product, significantly improving the clarity and contrast of the captured image. For mirrored metal products, it can effectively avoid image blur caused by excessive reflection. For plastic products with rough surfaces, it can enhance image details and make it easier to find tiny defects. Improved detection efficiency: The parallel processing capability of the image acquisition card greatly shortens the image acquisition and transmission time, meeting the needs of high-speed production lines for real-time detection of product quality; on electronic product production lines, it can quickly complete product image acquisition to ensure efficient operation of the production line.

[0021] In this embodiment, the deep learning defect detection module is used to implement end-to-end analysis from image to defect decision, specifically including: Dual-path residual denoising unit: It uses a dual-branch network structure to extract low-frequency structural information and high-frequency detail features of the image respectively, and fuses the denoising results through residual connections; Multi-scale context-aware unit: With the help of the dilated convolutional pyramid network (DCPN), local defect details and global context-related features are obtained simultaneously; Graph topology defect classification unit: constructs a spatial relationship graph of defect areas, uses graph convolutional network (GCN) to iteratively update node features, and finally outputs the joint probability of defect type and severity; Furthermore, the deep learning defect detection module is the core of the product quality inspection system, and its main functions are as follows: Image denoising and feature extraction: Images collected in industrial production often contain noise. The dual-path residual denoising unit can remove noise interference and use different convolution kernels to extract low-frequency structural information of the image, such as the overall outline and general shape of the product, as well as high-frequency detail features, such as surface texture and tiny defects, to provide clear and valuable data for subsequent analysis. Multi-scale context perception: To judge product defects, local details and global information must be integrated. The multi-scale context perception unit uses the dilated convolutional pyramid network (DCPN) to simultaneously capture local defect details and global context association features at different scales. For example, when inspecting circuit boards, it can focus on the tiny solder joints of components while considering the overall circuit layout to avoid misjudgment due to local analysis. Defect classification and severity assessment: The graph topology defect classification unit constructs a spatial relationship graph of defect regions, treats defects as graph nodes, determines the edge weights between nodes based on spatial distance and feature similarity, and uses graph convolutional networks (GCN) to iteratively update node features. It ultimately outputs defect types, such as scratches, cracks, holes, etc., as well as the joint probability of severity, to achieve quantitative assessment of defects. Its specific structural framework includes: The network structure of the dual-path residual denoising unit is: Low-frequency branch: A large 5×5 convolution kernel is used to extract image macroscopic structural features, and the number of output channels is 64; High-frequency branch: A 3×3 small convolution kernel is used, combined with a dilated convolution layer with a dilation rate of 2 to extract micro-texture features, and the number of output channels is 128; Feature fusion layer: weighted fusion of dual-branch outputs through cross-channel attention mechanism, the calculation formula is: ,in, , is a learnable weight matrix used to perform weighted transformation on input features; Sigmoid function maps the weighted result to between 0 and 1 to obtain the attention weight ; represents the characteristics of the low-frequency branch output, Represents the characteristics of the high-frequency branch output; Represents element-by-element multiplication, and the attention weight Multiply and add the corresponding branch features to get the fused features ; The dilated convolutional pyramid network of multi-scale context-aware units contains: Basic feature layer: extract the underlying feature map with a resolution of 256×256 through 3×3 standard convolution; Multi-scale context layer: The dilated convolution layers with expansion rates of 2, 4, and 8 are deployed in parallel, and the output feature map resolutions are 128×128, 64×64, and 32×32 respectively; Feature refinement module: A bidirectional feature pyramid structure is used to fuse multi-scale features. The calculation formula is: ;in, Represents the deconvolution operation, which is the feature map obtained by convolution with different dilation rates. Restore to the same resolution as the base feature map, , corresponding to the characteristic graphs under different expansion rates; To multiply element by element, the deconvolution feature map is combined with the channel attention module Processed basic feature map Element-wise multiplication; The channel attention module is used to calculate the importance weight of each channel of the basic feature map, highlight the key information, and finally add the three together to obtain the refined feature ; The workflow of the graph topology defect classification unit includes: Node feature initialization: The detected candidate defect region ROI feature vector is used as a graph node with a feature dimension of 512; Dynamic edge weight calculation: based on the spatial distance between defect areas and feature similarity Generate an adjacency matrix: ,in, is the distance attenuation coefficient; is the temperature parameter; Indicates and The spatial distance between the defect areas, Indicates and The feature similarity of the defect areas; is the adjacency matrix element, indicating the and The edge weight between nodes; k is the index variable used to traverse all possible nodes; Multi-hop graph convolution: It iteratively updates node representation through a three-layer graph convolution network, with the output dimensions of each layer being 256, 128, and 64 respectively. The graph convolution network aggregates neighbor node information and gradually explores the intrinsic connections of defect features to achieve classification and evaluation of defect types and severity. The operation steps of the deep learning defect detection module are: Image input and denoising: The image input dual-path residual denoising unit obtained by the multimodal image acquisition module, the low-frequency and high-frequency branches work simultaneously, extract features respectively, perform weighted fusion in the feature fusion layer, remove noise, and obtain a feature-rich image representation; Multi-scale feature extraction and fusion: The denoised image representation enters the multi-scale context perception unit, first extracting the underlying features in the basic feature layer, then obtaining multi-scale context features through the hole convolution layer with different expansion rates, and finally fusion and refinement in the feature refinement module to obtain a feature representation containing rich context information; Defect classification and assessment: The above features are input into the graph topology defect classification unit, which first initializes the node features, then calculates the dynamic edge weights to generate the adjacency matrix, and finally iteratively updates the node features through multi-hop graph convolution, outputs the joint probability of defect type and severity, and completes defect classification and severity assessment; The deep learning defect detection module enables: High-accuracy detection: Multi-scale context perception and graph topology defect classification technology enable the module to accurately identify various defects. Compared with traditional detection methods, it significantly improves accuracy, reduces false detection and missed detection rates, and improves product quality. Strong robustness: The dual-path residual denoising unit effectively improves the model's ability to process noisy images, allowing the model to run stably in complex industrial environments and ensuring reliable detection results; Wide adaptability: This module can handle defect detection tasks for products of different shapes and materials. It has strong versatility and adaptability and can be widely used in various industrial production scenarios.

[0022] In this embodiment, the dynamic resource scheduling module generates a multi-objective optimization scheduling scheme based on the defect detection results. The dynamic resource scheduling module is a key part of the efficient operation of the product quality inspection system and has the following main functions: Optimize personnel scheduling: According to the defect types given by the deep learning defect detection module, accurately match the maintenance personnel skill map database; by calculating the skill matching degree, reasonably arrange maintenance personnel so that each defect can be handled by the most suitable person, improving maintenance efficiency and quality; Plan material scheduling: Combine the defect location coordinates and production line layout topology, comprehensively consider transportation time and energy consumption, and use the ant colony optimization algorithm to plan the optimal path for material transportation; ensure that repair materials are delivered in a timely and efficient manner, while reducing logistics costs; Balanced production line throughput: Based on the Markov decision process, dynamically adjust the resource allocation ratio of the inspection station and the maintenance station; according to the actual situation of each link on the production line, ensure the coordination of the production rhythm of the production line, avoid idle or congested station resources, and maximize the overall throughput of the production line; Furthermore, the structural framework of the dynamic resource scheduling module includes: Skill map-driven personnel scheduling unit: Contains the maintenance personnel skill map database, which stores information such as maintenance personnel's skills, work experience, and defect types they are good at handling. Through the matching algorithm between defect types and skill maps, the skill matching degree between maintenance personnel and current defect handling tasks is quickly calculated, and a priority queue is generated to provide a basis for personnel scheduling, including: Skill matching calculation: Assume that the maintenance personnel skill vector is ,in, Indicates The maintenance personnel The level of ability in a skill, ; The skill requirement vector corresponding to the defect type is , Indicates the first The demand for a skill; the cosine similarity algorithm is used to calculate the skill matching degree , the formula is: , where the molecule It is the dot product of two vectors, which measures the total matching degree between the maintenance personnel's skills and the skills required for the defect in each dimension; the denominator are the modulus lengths of the maintenance personnel skill vector and the defect skill requirement vector, respectively, and are used to normalize the numerator so that The value of The larger the value, the higher the skill matching degree; Time-space constraint material scheduling unit: Based on the production line layout topology map and defect location coordinates, it has a built-in ant colony optimization algorithm module. The algorithm module simulates the foraging behavior of ants to search for the best material transportation path based on optimization goals such as transportation time and energy consumption cost, while taking into account time and space constraints to ensure reasonable and efficient material transportation. Optimization objective function: The optimization goal is to minimize the weighted sum of transportation time and energy consumption cost, while considering the impact of the maximum transportation time on the overall situation. The objective function is: ,in, For the AGV transportation time; For the Energy consumption cost of an AGV; , are the weight coefficients of transportation time and energy consumption cost respectively; is the weight coefficient, which is used to adjust the impact of the maximum transportation time on the optimization target; is the number of AGVs; By improving the ant colony algorithm to solve the Pareto optimal solution set, in each iteration, the ants Slave Node Transfer to Node Probability The calculation formula is: ; in, It is the edge The higher the pheromone concentration, the more favored the path is by ants. It is the edge The heuristic information is usually inversely proportional to the distance between two points. The shorter the distance, the greater the heuristic information. and Respectively represent the relative importance of pheromone and heuristic information, The larger the pheromone concentration, the more likely the ants will choose the path with the highest pheromone concentration. The larger it is, the more ants tend to choose paths with shorter distances (larger heuristic information); Represents ants The next step allows the selection of a set of nodes, namely ants Nodes that can be reached directly from the current location; Production line throughput balancing unit: Using the Markov decision model, real-time monitoring of the working status, task volume and other information of the inspection and maintenance stations; this information will be integrated to define the state of the system at a certain moment; the system state set is used Indicates that each specific state They are all comprehensive descriptions of the workload, equipment status, and personnel configuration of the inspection and maintenance stations, namely ; The action set is recorded as ,action Represents the adjustment operation of the resource allocation of the inspection station and maintenance station, such as increasing the operating time of the inspection equipment, deploying maintenance personnel, etc. Take action Transfer to state The probability of , the immediate reward is , which means that in the state Take action Transfer to state The immediate benefits obtained when the state is changed, such as the increase in the throughput of the production line under this state transfer; Solving the optimal strategy through the Bellman equation ; ; in, Yes Status The optimal value function, which represents the optimal value function from the state Start by following the optimal strategy The maximum cumulative reward that can be obtained; is the discount factor, ranging from In between, it indicates the importance of future rewards. The closer , indicating that they attach more importance to future rewards, while they pay more attention to immediate rewards; Furthermore, the operation steps of the dynamic resource scheduling module include: Execute personnel scheduling: After the deep learning defect detection module identifies the product defect and determines the type, the skill map-driven personnel scheduling unit is started; first read the skill map information of all maintenance personnel from the database, match the defect type with the skill map one by one, and calculate the skill matching degree according to the above skill matching degree formula; generate a priority queue from high to low according to the matching degree, and the system arranges maintenance personnel to handle defects according to the queue order; Implement material scheduling: After the defect location is determined, the time-space constraint material scheduling unit starts working. First, obtain the production line layout topology map to clarify the position relationship between each workstation and material storage point. Combine the defect location coordinates, take transportation time and energy consumption cost as optimization goals, and use the above-mentioned ant colony optimization algorithm to plan the path. The algorithm continuously iterates to find the optimal material transportation path, and then controls the transportation equipment to transport materials according to the planned path. Adjust the balance of production line throughput: The production line throughput balance unit continuously collects real-time data from inspection stations and maintenance stations, including the number of completed tasks, the current task processing progress, equipment idle time, etc.; based on these data, the current state of the system is determined, and according to the Markov decision process, the Bellman equation is used to calculate the value function after taking different actions (such as adjusting equipment operating time, allocating manpower, etc.) under the current state, and the action with the greatest value is selected as the optimal resource allocation plan to achieve dynamic balance of production line throughput; The benefits of the dynamic resource scheduling module include: Improve maintenance efficiency: Accurate personnel scheduling ensures that each defect can be quickly handled by maintenance personnel with corresponding skills, reducing maintenance waiting time, improving maintenance accuracy and efficiency, and reducing production delays caused by untimely maintenance; Reduce logistics costs: The optimized material transportation route reduces transportation time and energy consumption, reduces logistics costs, and improves the timeliness of material distribution to ensure production continuity; Enhance the overall efficiency of the production line: Dynamically balance the resource allocation of inspection stations and maintenance stations to avoid resource waste and production bottlenecks, significantly improve the production efficiency of the entire production line, and improve the production benefits of the enterprise.

[0023] In this embodiment, the augmented reality interaction module includes a holographic projection device and a multimodal interaction terminal, which superimposes a three-dimensional thermal map of defects, maintenance path navigation, and process parameter correction suggestions in real time; Further, the augmented reality interaction module is described in detail below: In the product quality inspection system, the augmented reality interaction module builds an efficient communication bridge between people and inspection data and equipment, and has the following functions: Visual display of defects: Through holographic projection equipment, defects detected by products are visually presented in the form of a three-dimensional heat map, allowing operators to quickly and accurately locate the defect position and understand the distribution of defect severity; areas of different colors and brightness represent defects of different degrees, for example, red areas represent serious defects, yellow areas represent moderate defects, and green areas represent minor defects; Maintenance navigation assistance: Combined with multi-modal interactive terminals, it provides maintenance personnel with maintenance path navigation, guides maintenance personnel to quickly reach the defect location, and provides detailed maintenance operation step guidance based on the defect type and product structure, reducing maintenance time and erroneous operations; Real-time parameter adjustment: Based on the test results and maintenance requirements, the multi-modal interactive terminal displays the process parameter correction suggestions in real time. The operator can adjust the parameters directly on the terminal to achieve instant optimization of the production process and improve product quality. Holographic projection equipment: It consists of a projector, an optical lens group, an image processor, etc. The image processor receives the defect data from the deep learning defect detection module, converts it into a three-dimensional image format, and then projects the three-dimensional thermal map onto the product or related operating space through the projector and optical lens group to achieve a three-dimensional display of the defects. Multimodal interactive terminal: It integrates touch screen, voice interaction module, gesture recognition sensor, etc. The touch screen is used to display maintenance path, process parameter suggestions and other information, and the operator can confirm and adjust through touch operation. The voice interaction module supports voice command input and voice prompt output, which is convenient for operators to interact when their hands are busy. The gesture recognition sensor captures the operator's gestures to achieve contactless interaction and improve the convenience and efficiency of operation. Data reception and processing: The augmented reality interaction module obtains product defect information from the deep learning defect detection module, including defect location, type, severity, etc.; the image processor of the holographic projection device converts this data into image data of a three-dimensional thermal map, and the multimodal interactive terminal organizes the received maintenance path and process parameter correction suggestions and other data; Defect display and interactive guidance: The holographic projection device projects the generated three-dimensional thermal map to the corresponding position of the product to clearly display the defect distribution; at the same time, the multimodal interactive terminal displays the maintenance path and operation steps on the touch screen, and the voice interaction module gives voice prompts to guide the operator to the next step; Interactive operation and feedback: The operator interacts with the multimodal interactive terminal through touch screen, voice commands or gestures according to the displayed information; for example, click to confirm the maintenance steps, enter new process parameters, zoom in or out the defect display area through gestures, etc.; the interactive terminal feeds back the operation information to the system, and the system adjusts the display content or executes the corresponding control instructions based on the feedback.

[0024] Beneficial Effects Improve inspection and maintenance efficiency: intuitive defect display and maintenance navigation enable operators to quickly locate and handle defects, greatly shortening inspection and maintenance time and improving production efficiency; Reduced operating difficulty: Multimodal interaction adapts to different operating scenarios and operator habits, reduces the skill requirements for operators, and reduces the probability of erroneous operations; Optimize the production process: Real-time process parameter correction suggestions help operators adjust the production process in time, improve product quality and reduce the defective rate.

[0025] In this embodiment, it also includes: Self-evolving detection model: It uses an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, including: Graph convolution layer weight matrix: Update the learnable parameter matrix of each graph convolution operation ,in, and Respectively The feature dimensions of the layer inputs and outputs; Dynamic edge weight generation parameters: Update distance decay coefficient in adjacency matrix calculation and temperature parameters , to optimize the spatial relationship modeling between defect areas; Attention mechanism parameters: If the attention mechanism is introduced in the classification unit, the attention weight matrix is ​​updated And the linear transformation parameters , ;in, and Respectively represent the channel dimensions of input and output in the attention mechanism; The weight update is implemented by the following formula: ,in, is the current batch data, For historical memory bank samples, ; For the historical memory Samples about network weights The gradient of the loss function; is the learning rate parameter, which controls the step size of updating the weights based on the current batch of data; is the learning rate parameter, which controls the step size of updating weights based on historical memory samples, thus obtaining The network weight at the moment ; Multi-physics simulation module: integrated finite element analysis engine, simulates the impact of defects on product mechanical properties, and generates residual stress distribution cloud map; Furthermore, the specific operation steps of the self-evolution detection model are as follows: Data preparation: Collect new test sample data and organize it into a format that meets the model input requirements to form the current batch data ; At the same time, extract samples from the historical memory bank , ,These historical samples record various data in the previous detection process and are used to ,assist the model in updating so that it can be optimized by combining past ,experience and new data; Calculate the gradient of the current batch of data: Input into the network of the deep learning defect detection module, according to the loss function set by the model , calculate the current batch data about the network weight The gradient of the loss function ; This gradient reflects the direction and degree of adjustment of the network weight by the current batch of data, and can be efficiently calculated through the back propagation algorithm; Calculate the gradient of the historical memory sample: for each sample in the historical memory , also input into the network, calculate its weight on the network The gradient of the loss function ; These gradients reflect the impact of historical samples on weight adjustment. Comprehensively considering the gradients of historical samples can prevent the model from forgetting useful features learned previously; Update network weights: Based on the calculated gradient of the current batch data and the sample gradient of the historical memory library, update the network weights according to the update formula To update the network weights, is the current batch data, For historical memory bank samples, ; For the historical memory Samples about network weights The gradient of the loss function; is the learning rate parameter, which controls the step size of updating the weights based on the current batch of data; is the learning rate parameter, which controls the step size of updating weights based on historical memory samples, thus obtaining The network weight at the moment , so that the model can continuously adapt to new detection tasks and data changes, improving the accuracy and adaptability of defect detection; Furthermore, the specific operation steps of the multi-physics simulation module are as follows: Preliminary data collection and preparation: Obtain product defect information from the deep learning defect detection module, including key data such as defect location, size, and shape; at the same time, collect product material property data, such as elastic modulus, Poisson's ratio, yield strength, etc. These material parameters are crucial for accurately simulating product mechanical properties; in addition, the product's geometric model must be determined, which can be an original model designed by CAD software or a 3D model reconstructed based on actual product scans; Establish a multi-physics simulation model: Use the finite element analysis engine to mesh the product's geometric model and discretize the continuous solid model into a combination of finite units to facilitate subsequent numerical calculations. When meshing, the mesh density should be reasonably adjusted according to the product structure and defect location, and a finer mesh should be used near the defect and in the stress concentration area to improve the calculation accuracy. Then, based on the collected material property data, define the material parameters in the simulation model to ensure that the model can accurately reflect the actual material properties of the product. Then, according to the actual working conditions of the product, apply corresponding boundary conditions, such as fixed constraints, loads, temperature, etc., to simulate the force and working state of the product in a real environment. Execute simulation calculations: After setting the simulation model and parameters, start the finite element analysis engine to perform multi-physics field simulation calculations. During the calculation process, the engine will solve the partial differential equations describing the mechanical properties of the product according to the set algorithm, and simulate the impact of defects on the internal stress and strain distribution of the product under the action of various physical fields. During the calculation process, the calculation status needs to be monitored in real time to ensure the stability and convergence of the calculation. If any abnormality occurs, the model parameters need to be adjusted or reset in time. Result analysis and visualization: After the simulation calculation is completed, the calculation result data is obtained, including the stress distribution, strain distribution, displacement and other information inside the product; professional post-processing software is used to analyze the result data and extract key indicators, such as maximum stress value, stress concentration area, deformation, etc.; by generating visual charts such as residual stress distribution cloud map, the impact of defects on the mechanical properties of the product is intuitively displayed, so that engineers can clearly understand the weak links and potential risks of the product, and provide a strong basis for product quality assessment and improvement; according to the analysis results, it can also be further compared with the design standards and quality requirements of the product to determine whether the product is qualified and whether it is necessary to optimize the product design or production process.

[0026] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A product quality inspection system based on deep learning and machine vision, characterized by: include: Multimodal image acquisition module: It consists of a high-resolution industrial camera array, an adaptive light source system and an image acquisition card; the industrial camera array is distributed in a ring shape, covering the product surface in all directions, and realizing a 360° detection angle; the adaptive light source system dynamically adjusts the multi-band illumination combination according to the product surface reflectivity, and the spectral range covers 400-1100nm; the image acquisition card has the ability to process at least 8 camera signals in parallel; Deep learning defect detection module: used to achieve end-to-end analysis from image to defect decision, including: Dual-path residual denoising unit: It uses a dual-branch network structure to extract low-frequency structural information and high-frequency detail features of the image respectively, and fuses the denoising results through residual connections; Multi-scale context-aware unit: With the help of the dilated convolutional pyramid network (DCPN), local defect details and global context-related features are obtained simultaneously; Graph topology defect classification unit: constructs a spatial relationship graph of defect areas, uses graph convolutional network (GCN) to iteratively update node features, and finally outputs the joint probability of defect type and severity; Dynamic resource scheduling module: Generates multi-objective optimization scheduling solutions based on defect detection results, covering: Skill map-driven personnel scheduling unit: Match the maintenance personnel skill map database according to the defect type, and generate a priority queue by calculating the skill matching degree; Time-space constraint material scheduling unit: Combines the defect location coordinates with the production line layout topology map and uses the ant colony optimization algorithm to plan the material transportation path; Production line throughput balancing unit: uses Markov decision process to dynamically adjust the resource allocation ratio of inspection stations and maintenance stations; Augmented reality interaction module: includes holographic projection equipment and multimodal interactive terminals, which can superimpose three-dimensional thermal maps of defects, maintenance path navigation and process parameter correction suggestions in real time.

2. The product quality inspection system based on deep learning and machine vision according to claim 1, characterized in that: The network structure of the dual-path residual denoising unit is: Low-frequency branch: A large 5×5 convolution kernel is used to extract image macroscopic structural features, and the number of output channels is 64; High-frequency branch: A 3×3 small convolution kernel is used, combined with a dilated convolution layer with a dilation rate of 2 to extract micro-texture features, and the number of output channels is 128; Feature fusion layer: weighted fusion of dual-branch outputs through cross-channel attention mechanism, the calculation formula is: ,in, , is a learnable weight matrix used to perform weighted transformation on input features; Sigmoid function maps the weighted result to between 0 and 1 to obtain the attention weight ; represents the characteristics of the low-frequency branch output, Represents the characteristics of the high-frequency branch output; Represents element-by-element multiplication, and the attention weight Multiply and add the corresponding branch features to get the fused features .

3. The product quality inspection system based on deep learning and machine vision according to claim 2, characterized in that: The hole convolutional pyramid network of the multi-scale context-aware unit includes: Basic feature layer: extract the underlying feature map with a resolution of 256×256 through 3×3 standard convolution; Multi-scale context layer: The dilated convolution layers with expansion rates of 2, 4, and 8 are deployed in parallel, and the output feature map resolutions are 128×128, 64×64, and 32×32 respectively; Feature refinement module: A bidirectional feature pyramid structure is used to fuse multi-scale features. The calculation formula is: ;in, Represents the deconvolution operation, which is the feature map obtained by convolution with different dilation rates. Restore to the same resolution as the base feature map, , corresponding to the characteristic graphs under different expansion rates; To multiply element by element, the deconvolution feature map is combined with the channel attention module Processed basic feature map Element-wise multiplication; The channel attention module is used to calculate the importance weight of each channel of the basic feature map, highlight the key information, and finally add the three together to obtain the refined feature .

4. The product quality inspection system based on deep learning and machine vision according to claim 3, characterized in that: The workflow of the graph topology defect classification unit includes: Node feature initialization: The detected candidate defect region ROI feature vector is used as a graph node with a feature dimension of 512; Dynamic edge weight calculation: based on the spatial distance between defect areas and feature similarity Generate an adjacency matrix: ,in, is the distance attenuation coefficient; is the temperature parameter; Indicates and The spatial distance between the defect areas, Indicates and The feature similarity of the defect areas; is the adjacency matrix element, indicating the and The edge weight between nodes; k is the index variable used to traverse all possible nodes; Multi-hop graph convolution: The node representation is iteratively updated through a 3-layer graph convolutional network, with the output dimensions of each layer being 256, 128, and 64 respectively.

5. The product quality inspection system based on deep learning and machine vision according to claim 4, characterized in that: The optimization objective function of the time-space constraint material scheduling unit is: ,in, For the AGV transportation time; For the Energy consumption cost of an AGV; , are the weight coefficients of transportation time and energy consumption cost respectively; is the weight coefficient, which is used to adjust the impact of the maximum transportation time on the optimization target; is the number of AGVs, and the Pareto optimal solution set is solved by improving the ant colony algorithm. Respectively The transportation time of one AGV.

6. The product quality inspection system based on deep learning and machine vision according to claim 5, characterized in that: Also includes: Self-evolving detection model: It uses an online incremental learning mechanism to dynamically update the network weights of the multi-task graph convolutional network MT-GCN in the graph topology defect classification unit based on new detection samples, including: Graph convolution layer weight matrix: Update the learnable parameter matrix of each graph convolution operation ,in, and Respectively The feature dimensions of the layer input and output; Dynamic edge weight generation parameters: Update distance decay coefficient in adjacency matrix calculation and temperature parameters , to optimize the spatial relationship modeling between defect areas; Attention mechanism parameters: If the attention mechanism is introduced in the classification unit, the attention weight matrix is ​​updated And the linear transformation parameters , in, and Respectively represent the channel dimensions of input and output in the attention mechanism; The weight update is implemented by the following formula: ,in, is the current batch data, For historical memory bank samples, ; For the historical memory Samples about network weights The gradient of the loss function; is the learning rate parameter, which controls the step size of updating the weights based on the current batch of data; is the learning rate parameter, which controls the step size of updating weights based on historical memory samples, thus obtaining The network weight at the moment ; Multi-physics simulation module: Integrates finite element analysis engine to simulate the impact of defects on product mechanical properties and generate residual stress distribution cloud map.

Images