Photoelectric intelligent error-proofing workbench control method and system based on deep learning
By employing a deep learning method that combines multi-color space feature fusion and adaptive threshold segmentation, the problems of recognition accuracy and response speed of the photoelectric intelligent error-proof workbench in complex environments were solved, achieving high-precision and low-latency assembly control.
Patent Information
- Application Number
- CN202511110552.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-08
- Publication Date
- 2025-11-21
AI Technical Summary
Existing photoelectric intelligent error-proofing workbenches suffer from low recognition accuracy in complex environments, frequent false detections and missed detections, and difficulty in achieving high-speed response and real-time closed-loop control. As a result, assembly errors cannot be intercepted in time, leading to batch defective products and production line downtime losses.
By employing a deep learning-based approach, multi-scale feature fusion across multiple color spaces and adaptive threshold segmentation, combined with illuminance sensor data and historical detection data, accurate identification and real-time control of assemblies are achieved.
Even in complex environments with fluctuating brightness and color, the recognition accuracy remains above 98.5%, and the system latency is controlled within 70 ms, meeting the real-time response requirements of high-speed assembly lines and effectively preventing the transmission of assembly errors and batch scrap.
Smart Images
Figure CN120997169A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent error-proofing workbench technology, and in particular to a control method and system for an optoelectronic intelligent error-proofing workbench based on deep learning. Background Technology
[0002] With the widespread application of intelligent manufacturing and automated assembly technologies, optoelectronic intelligent error-proofing workbenches have been rapidly promoted in high-end assembly lines for automobiles, electronics, and home appliances. Currently, error-proofing workbenches mostly use machine vision systems to perform online detection and judgment of the status of assembled parts, realizing defect alarms and closed-loop control of the assembly process.
[0003] Traditional machine vision inspection methods are mostly based on threshold segmentation in a single color space. In actual production environments, due to the constant changes in workpiece batches, materials, surface treatment processes, and workstation lighting conditions, single color spaces and static threshold settings are easily limited by interference factors such as lighting changes, reflections, and surface contamination, resulting in a significant decrease in recognition accuracy, frequent false detections and missed detections, which seriously affect the stability and versatility of error prevention systems. Although some systems have introduced adaptive thresholds or automatic gain compensation, they are mostly limited to local grayscale histogram processing and are difficult to cover global feature changes in environments with high mixing, multiple varieties, and multiple materials.
[0004] Existing error-proofing workbench vision systems typically use image processing units linked with programmable logic controllers. However, due to limitations in data transmission, processing latency, and control architecture, it is difficult to achieve millisecond-level high-speed response and real-time closed-loop. In situations where assembly lines operate at high speeds and require high resolution, the system is prone to response lag, resulting in assembly errors not being intercepted in time, causing batch defects and line downtime losses. Summary of the Invention
[0005] One objective of this invention is to propose a photoelectric intelligent error-proofing workbench control method and system based on deep learning, which effectively prevents the transmission of assembly errors and the risk of batch scrapping.
[0006] A method and system for controlling a photoelectric intelligent error-proofing workbench based on deep learning, according to an embodiment of the present invention, includes: Acquire real-time image data of the target workpiece and synchronous illuminance sensor data, and complete exposure trigger synchronization. Use the illuminance sensor data to perform illumination compensation and color drift correction on the real-time image data of the target workpiece to generate corrected image data. The corrected image data were converted into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively. Multi-scale image pyramids were constructed for the RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively to obtain multi-scale image feature data. Pixel-level color feature data and pixel-level texture feature data are extracted from multi-scale image feature data to form a multi-color space multi-scale feature set. Weighted feature fusion is performed on the multi-color space multi-scale feature set to obtain fused saliency image data. A threshold prediction model is trained based on fused saliency image data, illuminance sensor data, production batch identification data and historical threshold parameters. The model outputs prior threshold prediction data. Using the prior threshold prediction data as the initial threshold, adaptive threshold segmentation is performed in combination with historical false detection and false detection statistics to generate binary mask image data. Morphological denoising and boundary refinement are performed on the binary mask image data to obtain fine mask image data. The fine mask image data is then compared with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space to verify mask consistency. The fine mask image data verified by mask consistency is matched and compared with the pre-stored workpiece standard model data to generate assembly status judgment result data. The assembly status judgment result data is transmitted to the programmable logic controller control unit in real time, and the output control action instruction data drives the execution component to implement alarm, stop or reset operations, thus completing the control of the photoelectric intelligent error-proof workbench.
[0007] Optionally, generating the corrected image data includes: The photoelectric acquisition module acquires real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located. The real-time image data of the target workpiece is subjected to illumination compensation processing to obtain an illumination-compensated color image matrix. Color shift correction is performed on the color image matrix after illumination compensation to obtain corrected image data after illumination compensation and color shift correction.
[0008] Optionally, the construction of the multi-scale image feature data includes: The corrected image data is converted to RGB color space image dataset, HSV color space image dataset, Lab color space image dataset and YCbCr color space image dataset respectively. Multi-scale image pyramids were constructed for RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively. The input color space image datasets were sampled in descending order of resolution according to the scale ratio sequence to form a multi-scale image set. Multi-scale image feature data is extracted from the multi-scale image set corresponding to each color space, and the multi-scale image feature data of all color spaces are uniformly organized into a multi-color space multi-scale image feature set.
[0009] Optionally, the generation of the fused saliency image data includes: Extracting pixel-level color feature data and pixel-level texture feature data from multi-color space, multi-scale image feature sets; Pixel-level color feature data and pixel-level texture feature data of each color space and scale are combined according to color space and scale to form a fused input feature vector; Based on the fusion input feature vector set, weighted fusion is performed according to the preset color space weight coefficient and scale weight coefficient to obtain the fusion saliency value. The fusion saliency values of all pixel positions are combined to form fusion saliency image data.
[0010] Optionally, the generation of the binary mask image data includes: Based on the fusion of saliency image data, illuminance sensor data, production batch identification data and historical threshold parameter set, a threshold prediction model input feature vector is constructed. The input feature vector to the threshold prediction model is fed into the threshold prediction model for inference calculation to obtain the prior threshold prediction data. Adaptive threshold adjustment is performed based on prior threshold prediction data combined with historical false positive and false negative rates: Based on the prior threshold prediction data, the threshold is adjusted proportionally according to the difference between the historical false positive rate and the historical false negative rate to obtain an adaptive updated threshold. Using the adaptive update threshold as the segmentation benchmark, binarization segmentation is performed on the fused saliency image data. Pixels with pixel intensity greater than or equal to the adaptive update threshold are marked as foreground pixels, and the remaining pixels are marked as background pixels, generating binary mask image data.
[0011] Optionally, the generation of the fine mask image data includes: Morphological denoising is performed on the binary mask image data. Opening and closing operations are performed on the binary mask image data using structuring elements, which are pixel neighborhood templates with fixed shapes and sizes. After denoising, denoised binary mask image data is obtained. Boundary refinement processing is performed on the denoised binary mask image data. Based on the pixel neighborhood iterative deletion rule, redundant pixels on the boundary are gradually removed without destroying the target connectivity, while preserving the target topology and obtaining fine mask image data. Binarization segmentation was performed on the RGB color space image dataset, HSV color space image dataset, Lab color space image dataset, and YCbCr color space image dataset respectively according to the adaptively updated threshold, to obtain RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data, and YCbCr color space binary mask image data. Mask consistency verification is performed on fine mask image data and binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space. When the consistency threshold is exceeded, the adaptive threshold segmentation process is executed via callback. The adaptive update threshold is adjusted using the consistency verification feedback result, new binary mask image data is generated, and morphological denoising, boundary thinning, binarization segmentation of each color space and mask consistency verification are repeatedly executed until the consistency verification passes.
[0012] Optionally, the mask consistency verification includes: Using the fine mask image data as a benchmark, the intersection-union ratio (IUR) is calculated with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space, respectively. The ratio of the number of pixels in the intersection to the number of pixels in the union of the fine mask image data and each color space binary mask image data is then calculated. The mask consistency verification result is the IUR value. When any IUR value is lower than the consistency threshold, it is determined that the consistency has exceeded the limit.
[0013] Optionally, the generation of assembly status determination result data includes: The fine mask image data is matched and compared with the pre-stored workpiece standard model data. When calculating the comprehensive similarity index, the mask region is assigned a significant weight by combining the fused significant image data. The outline shape and key structure position are cross-validated by RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data and YCbCr color space binary mask image data. Assembly status is determined based on a comprehensive similarity index and a preset assembly qualification threshold, wherein: When the comprehensive similarity index is greater than or equal to the assembly qualification threshold and the consistency verification result does not exceed the limit, the assembly is judged to be qualified. When the overall similarity index is less than the assembly qualification threshold, but the area ratio of the mask missing region is less than or equal to the allowable missing area ratio, and the consistency verification result does not exceed the limit, it is judged as a minor anomaly. When the overall similarity index is less than the assembly qualification threshold, or the proportion of the mask missing area is greater than the allowable missing area proportion, or the consistency verification result exceeds the limit, the assembly is judged as unqualified. The assembly status determination result data is transmitted to the programmable logic controller control unit in real time, and control action instruction data is generated based on the determination result.
[0014] Optionally, the control action command data includes: For a qualified assembly state, output control action command data to maintain the normal operation of the worktable; For minor abnormal conditions, output control action command data for prompt alarm, including triggering audible and visual prompts, illuminating workstation guidance indicator lights, and displaying review information on the HMI interface; For assembly defects, the system outputs control action commands to force the workbench to stop, trigger audible and visual alarms, and perform automatic reset to prepare for the next round of testing.
[0015] Optional, including: The image acquisition module is used to acquire real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located, and generate corrected image data. The color space conversion and multi-scale feature construction module is used to convert the corrected image data into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively, and construct multi-scale image pyramids for each color space, extract multi-scale image feature data and form a unified multi-color space multi-scale image feature set. The feature fusion module is used to extract pixel-level color feature data and pixel-level texture feature data from a multi-color space, multi-scale image feature set to generate fused saliency image data. The threshold segmentation module is used to generate binary mask image data; The mask optimization and consistency verification module is used to process binary mask image data and perform consistency verification with binary mask image data in various color spaces. The assembly status determination and execution control module is used to match and compare the fine mask image data that has been verified for consistency with the pre-stored workpiece standard model data, calculate the comprehensive similarity index and determine the assembly status. The determination result is transmitted to the programmable logic controller control unit in real time to drive the execution component to perform alarm, stop or reset operations.
[0016] The beneficial effects of this invention are: (1) This invention integrates multi-scale features of multiple color spaces such as RGB, HSV, Lab, and YCbCr, and introduces saliency weights and pixel-level texture gradients to achieve highly adaptive recognition of assemblies under factors such as brightness, hue, surface material, and batch variation. It combines a priori threshold prediction model with an adaptive historical false detection and missed detection callback mechanism, so that threshold segmentation not only dynamically tracks environmental changes, but also self-evolves and optimizes based on historical data of the detection process, greatly reducing human parameter adjustments. Practical application shows that the assembly recognition accuracy of this method remains above 98.5% under conditions of brightness of 1000–30000 lx, surface color fluctuation of ±20%, and batch and material switching, which is far superior to the traditional scheme of single color space and fixed threshold segmentation.
[0017] (2) This invention introduces cross-validation and dynamic consistency threshold mechanism of multi-color space binary mask in the decision closed loop, which can not only effectively avoid single channel failure, but also call back the pre-threshold parameter in real time, ensuring that the false interception rate and false detection rate are minimized in complex environments. The end-to-end delay of the system is controlled within 70 ms, which fully meets the strict real-time response requirements of high-speed assembly lines and effectively prevents assembly error transmission and batch scrap risk. Attached Figure Description
[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a photoelectric intelligent error-proofing workbench control method and system based on deep learning proposed in this invention. Detailed Implementation
[0019] Example 1: refer to Figure 1 A deep learning-based optoelectronic intelligent error-proofing workbench control method and system, comprising: Acquire real-time image data of the target workpiece and synchronous illuminance sensor data, and complete exposure trigger synchronization. Use the illuminance sensor data to perform illumination compensation and color drift correction on the real-time image data of the target workpiece to generate corrected image data. In this embodiment, generating corrected image data includes: The photoelectric acquisition module acquires real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located. The real-time image data of the target workpiece reflects the original color image matrix acquired by the industrial camera under exposure-triggered synchronization, while the synchronous illuminance sensor data represents the ambient light intensity at the moment of acquisition at the assembly station.
[0020] The industrial camera and illuminance sensor are controlled by a synchronous trigger signal to achieve synchronized acquisition, ensuring that the real-time image data of the target workpiece and the synchronous illuminance sensor data correspond to the environmental state of the workstation at the same moment. The synchronous trigger signal provides a hard trigger pulse signal from the programmable logic controller.
[0021] The real-time image data of the target workpiece is subjected to illumination compensation processing to obtain an illumination-compensated color image matrix. The illumination compensation process is as follows: multiply the real-time image data of the target workpiece by the ratio of the illumination reference value to the synchronous illuminance sensor data. The illumination reference value represents the illumination normalization reference intensity, and the ratio is used to normalize the intensity of the real-time image data of the target workpiece to obtain the color image matrix after illumination compensation.
[0022] Color shift correction is performed on the color image matrix after illumination compensation to obtain corrected image data after illumination compensation and color shift correction.
[0023] Color shift correction is performed by using a color shift correction transformation matrix to perform a linear color space transformation on the color image matrix after illumination compensation. The element values of the color shift correction transformation matrix are obtained by comparing the reference values of the standard color card on the workpiece surface with the current acquired values on the workpiece surface. The color shift correction transformation matrix corrects the color shift of the image in different color channels after illumination compensation, resulting in corrected image data after illumination compensation and color shift correction.
[0024] The corrected image data were converted into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively. Multi-scale image pyramids were constructed for the RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively to obtain multi-scale image feature data. In this embodiment, the construction of multi-scale image feature data includes: The corrected image data is converted to RGB color space image dataset, HSV color space image dataset, Lab color space image dataset and YCbCr color space image dataset respectively. The RGB color space image dataset consists of pixel matrices for the red, green, and blue channels; the HSV color space image dataset consists of pixel matrices for the hue, saturation, and lightness channels; the Lab color space image dataset consists of pixel matrices for the lightness channel, a channel, and b channel; and the YCbCr color space image dataset consists of pixel matrices for the lightness channel, blue chromaticity channel, and red chromaticity channel.
[0025] Multi-scale image pyramids were constructed for RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively. The input color space image datasets were sampled in descending order of resolution according to the scale ratio sequence to form a multi-scale image set. Multi-scale image feature data is extracted from the multi-scale image set corresponding to each color space, and the multi-scale image feature data of all color spaces are uniformly organized into a multi-color space multi-scale image feature set.
[0026] Multiscale image feature data consists of pixel intensity distribution, color channel histogram, and texture orientation gradient information at each scale.
[0027] Pixel-level color feature data and pixel-level texture feature data are extracted from multi-scale image feature data to form a multi-color space multi-scale feature set. Weighted feature fusion is performed on the multi-color space multi-scale feature set to obtain fused saliency image data. In this embodiment, the generation of saliency image data includes: Extracting pixel-level color feature data and pixel-level texture feature data from multi-color space, multi-scale image feature sets; Pixel-level color feature data represents the color channel intensity value of a pixel location in different color spaces and at different scales, with the unit consistent with the pixel intensity of the corrected image data. Pixel-level texture feature data represents the local texture direction gradient and amplitude information of a pixel location in different color spaces and at different scales.
[0028] Pixel-level color feature data and pixel-level texture feature data of each color space and scale are combined according to color space and scale to form a fused input feature vector; The fused input feature vector describes the color-texture composite characteristics of a specific pixel in a specific color space and scale.
[0029] Based on the fusion input feature vector set, weighted fusion is performed according to the preset color space weight coefficient and scale weight coefficient to obtain the fusion saliency value. The fusion saliency values of all pixel positions are combined to form fusion saliency image data.
[0030] The fusion saliency value represents the degree of fusion saliency at a pixel location. It is obtained by weighting the fusion input feature vectors at each scale in each color space, then normalizing and calculating the saliency. The sum of all color space weight coefficients is 1, and the sum of all scale weight coefficients is 1. A threshold prediction model is trained based on fused saliency image data, illuminance sensor data, production batch identification data and historical threshold parameters. The model outputs prior threshold prediction data. Using the prior threshold prediction data as the initial threshold, adaptive threshold segmentation is performed in combination with historical false detection and false detection statistics to generate binary mask image data. In this embodiment, the generation of binary mask image data includes: Based on the fusion of saliency image data, illuminance sensor data, production batch identification data and historical threshold parameter set, a threshold prediction model input feature vector is constructed. The saliency image data reflects the pixel distribution of the salient area of the current workpiece, the illuminance sensor data represents the ambient light intensity at the moment of image acquisition, the production batch identification data identifies the production batch to which the current workpiece belongs, and the historical threshold parameter set records the optimal threshold parameters obtained by workpieces of the same or similar production batches under different lighting conditions.
[0031] The input feature vector to the threshold prediction model is fed into the threshold prediction model for inference calculation to obtain the prior threshold prediction data. Adaptive threshold adjustment is performed based on prior threshold prediction data combined with historical false positive and false negative rates: Based on the prior threshold prediction data, the threshold is adjusted proportionally according to the difference between the historical false positive rate and the historical false negative rate to obtain an adaptive updated threshold. Using an adaptively updated threshold as the segmentation criterion, binarization segmentation is performed on the fused saliency image data. Pixels with an intensity greater than or equal to the adaptively updated threshold are marked as foreground pixels, and the remaining pixels are marked as background pixels, generating binary mask image data. Morphological denoising and boundary refinement are performed on the binary mask image data to obtain fine mask image data. The fine mask image data is then compared with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space to verify mask consistency. In this embodiment, the generation of fine mask image data includes: Morphological denoising is performed on the binary mask image data. Opening and closing operations are performed on the binary mask image data using structuring elements, which are pixel neighborhood templates with fixed shapes and sizes. After denoising, denoised binary mask image data is obtained. Boundary refinement processing is performed on the denoised binary mask image data. Based on the pixel neighborhood iterative deletion rule, redundant pixels on the boundary are gradually removed without destroying the target connectivity, while preserving the target topology and obtaining fine mask image data. Binarization segmentation was performed on the RGB color space image dataset, HSV color space image dataset, Lab color space image dataset, and YCbCr color space image dataset respectively according to the adaptively updated threshold, to obtain RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data, and YCbCr color space binary mask image data. Mask consistency verification is performed on fine mask image data and binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space. When the consistency threshold is exceeded, the adaptive threshold segmentation process is executed via callback. The adaptive update threshold is adjusted based on the consistency verification feedback, new binary mask image data is generated, and morphological denoising, boundary thinning, binarization segmentation in each color space, and mask consistency verification are repeated until the consistency verification passes. The fine mask image data verified by mask consistency is matched and compared with the pre-stored workpiece standard model data to generate assembly status judgment result data. The assembly status judgment result data is transmitted to the programmable logic controller control unit in real time, and the output control action instruction data drives the execution component to implement alarm, stop or reset operations, thus completing the control of the photoelectric intelligent error-proof workbench.
[0032] In this embodiment, mask consistency verification includes: Using the fine mask image data as a benchmark, the intersection-union ratio (IUR) is calculated with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space, respectively. The ratio of the number of pixels in the intersection to the number of pixels in the union of the fine mask image data and each color space binary mask image data is then calculated. The mask consistency verification result is the IUR value. When any IUR value is lower than the consistency threshold, it is determined that the consistency has exceeded the limit.
[0033] The intersection pixel count is the number of pixels that are simultaneously marked as foreground pixels in both the fine-grained mask image data and the color space binary mask image data. The union pixel count is the number of pixels that are marked as foreground pixels in at least one of the fine-grained mask image data and the color space binary mask image data. The intersection-union ratio quantifies the degree of consistency between the fine-grained mask image data and each color space binary mask image data.
[0034] In this embodiment, the assembly status determination result data is generated, including: The fine mask image data is matched and compared with the pre-stored workpiece standard model data. When calculating the comprehensive similarity index, the mask region is assigned a significant weight by combining the fused significant image data. The outline shape and key structure position are cross-validated by RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data and YCbCr color space binary mask image data. In Example 1, the fine mask image data that has passed consistency verification is matched and compared with the pre-stored workpiece standard model data. The matching and comparison includes: calculating the pixel overlap rate, contour shape overlap degree, and key structure position deviation between the mask area and the corresponding area of the standard model, and assigning saliency weights to the pixels of the mask area in combination with the fused saliency image data; based on the cross-validation results of RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data, and YCbCr color space binary mask image data, the overlap rate, shape overlap degree, and position deviation are weighted, summed, and normalized to obtain a comprehensive similarity index.
[0035] Assembly status is determined based on a comprehensive similarity index and a preset assembly qualification threshold, wherein: When the comprehensive similarity index is greater than or equal to the assembly qualification threshold and the consistency verification result does not exceed the limit, the assembly is judged to be qualified. When the overall similarity index is less than the assembly qualification threshold, but the area ratio of the mask missing region is less than or equal to the allowable missing area ratio, and the consistency verification result does not exceed the limit, it is judged as a minor anomaly. When the overall similarity index is less than the assembly qualification threshold, or the proportion of the mask missing area is greater than the allowable missing area proportion, or the consistency verification result exceeds the limit, the assembly is judged as unqualified. The assembly status determination result data is transmitted to the programmable logic controller control unit in real time, and control action instruction data is generated based on the determination result.
[0036] In this embodiment, the control action command data includes: For a qualified assembly state, output control action command data to maintain the normal operation of the worktable; For minor abnormal conditions, output control action command data for prompt alarm, including triggering audible and visual prompts, illuminating workstation guidance indicator lights, and displaying review information on the HMI interface; For assembly defects, the system outputs control action commands to force the workbench to stop, trigger audible and visual alarms, and perform automatic reset to prepare for the next round of testing.
[0037] This embodiment includes: The image acquisition module is used to acquire real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located, and generate corrected image data. The color space conversion and multi-scale feature construction module is used to convert the corrected image data into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively, and construct multi-scale image pyramids for each color space, extract multi-scale image feature data and form a unified multi-color space multi-scale image feature set. The feature fusion module is used to extract pixel-level color feature data and pixel-level texture feature data from a multi-color space, multi-scale image feature set to generate fused saliency image data. The threshold segmentation module is used to generate binary mask image data; The mask optimization and consistency verification module is used to process binary mask image data and perform consistency verification with binary mask image data in various color spaces. The assembly status determination and execution control module is used to match and compare the fine mask image data that has been verified for consistency with the pre-stored workpiece standard model data, calculate the comprehensive similarity index and determine the assembly status. The determination result is transmitted to the programmable logic controller control unit in real time to drive the execution component to perform alarm, stop or reset operations.
[0038] Example 2: On a high-mixed-flow intelligent assembly line, a fault-proofing workbench is used for the appearance inspection and automatic fault-proofing control of power battery pack assemblies. The inspection tasks cover battery packs with various paint colors (silver-white, dark gray, orange) and different surface materials. There are significant differences in surface appearance between batches due to oil stains, reflections, and minor bumps. The ambient lighting is affected by weather and supplemental lighting from neighboring stations, with the collected illuminance fluctuating between 1100 lx and 29600 lx. The following actual scenarios occur during the inspection process: In one batch of inspections, a randomly selected power battery pack had a dark gray surface with multiple 1mm² oil stains and a large area of reflection in the upper right region. After the system started automatic detection, the original image captured by the photoelectric acquisition module had extremely uneven brightness, with the illuminance sensor reading 21400 lx. After illumination compensation processing, the system automatically adjusted the brightness normalization coefficient. After color drift correction, the average color difference of the corrected image decreased from 8.7 to 1.5. Subsequently, the image was synchronously converted to four color spaces: RGB, HSV, Lab, and YCbCr. The system constructed a three-layer pyramid multi-scale feature set in each space, reducing the standard deviation of the pixel-level texture direction gradient from 15.2 to 7.6, significantly improving edge consistency.
[0039] For this batch of workpieces, the fusion saliency weight model is used to automatically adjust the weight coefficients, and the weight distribution is RGB 0.29, HSV 0.23, Lab 0.31, and YCbCr 0.17. The system trains the model according to the batch identification and historical thresholds. The preliminary prediction threshold is 88. By introducing the false detection rate (0.61%) and missed detection rate (0.19%) in the last 100 detections, the threshold is adjusted to 91 after adaptive callback. After threshold segmentation of the fused saliency map, the system generates a preliminary mask. After morphological opening and closing operations, the number of noise points is reduced from 12 to 1. After boundary refinement, the average Hausdorff distance between the mask and the manually marked boundary is 1.03 pixels.
[0040] To improve the robustness of the determination, the system simultaneously generates multi-channel binary masks for the RGB, HSV, Lab, and YCbCr space images using the same threshold. The average cross-IoU consistency is 0.91. If the IoU of a certain space is lower than 0.88, the system automatically calls back the threshold segmentation process until the average IoU of all masks meets the standard. Finally, the consistency verification passes, and the mask data is used for comparison with the pre-stored standard model.
[0041] During the matching process, the system calculates the pixel coincidence rate (0.95), contour coincidence degree (0.92), and key structure point position deviation (0.42 mm) between the mask area and the standard model respectively. After weighting with the fused saliency weights, the comprehensive similarity index is 0.963. Combining the cross-channel verification results, it is determined that the packaging of this power battery is qualified. The system sends a qualified signal to the PLC in real time, the workbench keeps running, and the HMI interface displays the qualified thumbnail and stores the detection log.
[0042] In another detection, there is 2 mm² of paint peeling off on the left edge of the randomly selected orange battery pack. The supplementary light lamp causes a shadow in the upper left due to the station adjustment, and the illuminance sensor data is 11700 lx. After the original image is collected, the system automatically performs light compensation, and the color difference is reduced from 7.2 to 2.3. The recognition accuracy of the boundary texture in the Lab space is significantly improved, and the fused saliency weights are adjusted to RGB 0.25, HSV 0.22, Lab 0.38, and YCbCr 0.15. The threshold model's preliminary prediction is 93, and it is adjusted back to 95 in combination with the historical missed detection rate of 0.27%. After morphological processing of the segmentation mask, the noise is eliminated, and the IoU between the mask and the standard model is 0.85.
[0043] Because the IoU of the Lab channel mask was below the consistency threshold, the system triggered a callback, adaptively updating the threshold to 97, resulting in a final multi-channel average consistency value of 0.90. The overall similarity index was 0.901, and the area of the mask missing region accounted for 2.1%. The system determined this to be a minor anomaly, outputting a warning alarm, illuminating an indicator light, and highlighting the abnormal area in red on the HMI interface, prompting the operator to conduct a manual review. The final review revealed that the abnormal area was a small area of paint peeling, which did not affect assembly quality.
[0044] In another batch of inspections, a silver-white battery pack's right-side handle was misaligned during assembly, causing a 1.8mm deviation in the position of key structural points on the standard model. The system identified a pixel overlap rate of only 0.87 and an outline overlap rate of 0.85 between the fine mask and the standard model, resulting in a comprehensive similarity index of only 0.814 and a mask missing area ratio of 4.7%. The system immediately determined this as an assembly defect, automatically sending a stop command to the PLC, locking the workbench, activating audible and visual alarms, and displaying an abnormal image in a pop-up window on the HMI. Subsequent investigation confirmed that the battery pack handle was not fully tightened; failure to intercept it in time would have affected the safety of subsequent assembly.
[0045] The method of this invention achieves a detection accuracy of 98.6%, a false negative rate of 0.21%, and a false alarm rate of 0.55% in batches of samples under various conditions, including different colors, materials, varying lighting levels, impurities, and reflectivity. The total time delay for single-piece inspection is 65.7 ms. In contrast, the traditional HSV single-space fixed threshold method for the same batch of workpieces achieves an accuracy of 92.9%, a false negative rate of 1.31%, a false alarm rate of 2.07%, and a processing time delay of 220 ms. Furthermore, it frequently misjudges samples when encountering oil stains, reflectivity, or color differences in the coating.
[0046] Example of data details: The 48th piece in a certain batch had a gray oil stain area of 0.7 mm². Under 24500 lx illumination, the IoU of the present invention method was 0.95, and the comprehensive similarity index was 0.971, which was deemed qualified. The IoU of the traditional method was 0.82, which was misjudged as abnormal.
[0047] For the 275th piece in a certain batch, the orange spray paint color difference ΔE=9.5. The traditional method has false detection due to threshold drift. The method of this invention uses a priori threshold + adaptive segmentation correction, and the final boundary matches the actual value.
[0048] In a certain batch, item number 1432, the width of the reflective band on the matte green surface is 2.3mm. After fusing multi-scale texture features, the segmentation boundary accurately encloses the target, and the misjudgment rate is significantly lower than that of the control group.
[0049] In summary, Example 2 demonstrates that the present invention is significantly superior to traditional methods in terms of detection speed, accuracy, and adaptability to different lighting and surface conditions. It also achieves low maintenance, high reliability error prevention, and closed-loop control in high-mixed-flow flexible production line scenarios.
[0050] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A control method for a photoelectric intelligent error-proofing workbench based on deep learning, characterized in that, include: Acquire real-time image data of the target workpiece and synchronous illuminance sensor data, and generate corrected image data; The corrected image data were converted into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively, and a multi-scale image pyramid was constructed to obtain multi-scale image feature data. Pixel-level color feature data and pixel-level texture feature data are extracted from multi-scale image feature data to form a multi-color space multi-scale feature set. Weighted feature fusion is performed on the multi-color space multi-scale feature set to obtain fused saliency image data. Train the threshold prediction model and output the prior threshold prediction data. Use the prior threshold prediction data as the initial threshold basis and combine it with historical false detection and missed detection statistics to perform adaptive threshold segmentation and generate binary mask image data. Morphological denoising and boundary refinement are performed on the binary mask image data to obtain fine mask image data. The fine mask image data is then compared with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space to verify mask consistency. The fine mask image data, which has been verified for mask consistency, is matched and compared with the pre-stored workpiece standard model data to generate assembly status judgment result data. The assembly status judgment result data is transmitted to the programmable logic controller control unit in real time, and the output control action instruction data drives the execution component to perform alarm, stop or reset operations.
2. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The generation of corrected image data includes: The photoelectric acquisition module acquires real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located. The real-time image data of the target workpiece is subjected to illumination compensation processing to obtain an illumination-compensated color image matrix. Color shift correction is performed on the color image matrix after illumination compensation to obtain corrected image data after illumination compensation and color shift correction.
3. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The construction of the multi-scale image feature data includes: The corrected image data is converted to RGB color space image dataset, HSV color space image dataset, Lab color space image dataset and YCbCr color space image dataset respectively. Multi-scale image pyramids were constructed for RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively. The input color space image datasets were sampled in descending order of resolution according to the scale ratio sequence to form a multi-scale image set. Multi-scale image feature data is extracted from the multi-scale image set corresponding to each color space, and the multi-scale image feature data of all color spaces are uniformly organized into a multi-color space multi-scale image feature set.
4. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The generation of the fused saliency image data includes: Extracting pixel-level color feature data and pixel-level texture feature data from multi-color space, multi-scale image feature sets; Pixel-level color feature data and pixel-level texture feature data of each color space and scale are combined according to color space and scale to form a fused input feature vector; Based on the fusion input feature vector set, weighted fusion is performed according to the preset color space weight coefficient and scale weight coefficient to obtain the fusion saliency value. The fusion saliency values of all pixel positions are combined to form fusion saliency image data.
5. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The generation of the binary mask image data includes: Based on the fusion of saliency image data, illuminance sensor data, production batch identification data and historical threshold parameter set, a threshold prediction model input feature vector is constructed. The input feature vector to the threshold prediction model is fed into the threshold prediction model for inference calculation to obtain the prior threshold prediction data. Adaptive threshold adjustment is performed based on prior threshold prediction data combined with historical false positive and false negative rates: Based on the prior threshold prediction data, the threshold is adjusted proportionally according to the difference between the historical false positive rate and the historical false negative rate to obtain an adaptive updated threshold. Using the adaptive update threshold as the segmentation benchmark, binarization segmentation is performed on the fused saliency image data. Pixels with pixel intensity greater than or equal to the adaptive update threshold are marked as foreground pixels, and the remaining pixels are marked as background pixels, generating binary mask image data.
6. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The generation of the fine mask image data includes: Morphological denoising is performed on the binary mask image data. Opening and closing operations are performed on the binary mask image data using structuring elements, which are pixel neighborhood templates with fixed shapes and sizes. After denoising, denoised binary mask image data is obtained. Boundary refinement processing is performed on the denoised binary mask image data. Based on the pixel neighborhood iterative deletion rule, redundant pixels on the boundary are gradually removed without destroying the target connectivity, while preserving the target topology and obtaining fine mask image data. Binarization segmentation was performed on the RGB color space image dataset, HSV color space image dataset, Lab color space image dataset, and YCbCr color space image dataset respectively according to the adaptively updated threshold, to obtain RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data, and YCbCr color space binary mask image data. Mask consistency verification is performed on fine mask image data and binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space. When the consistency threshold is exceeded, the adaptive threshold segmentation process is executed via callback. The adaptive update threshold is adjusted using the consistency verification feedback result, new binary mask image data is generated, and morphological denoising, boundary thinning, binarization segmentation of each color space and mask consistency verification are repeatedly executed until the consistency verification passes.
7. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The mask consistency verification includes: Using the fine mask image data as a benchmark, the intersection-union ratio (IUR) is calculated with binary mask image data in RGB color space, HSV color space, Lab color space, and YCbCr color space, respectively. The ratio of the number of pixels in the intersection to the number of pixels in the union of the fine mask image data and each color space binary mask image data is then calculated. The mask consistency verification result is the IUR value. When any IUR value is lower than the consistency threshold, it is determined that the consistency has exceeded the limit.
8. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The generated assembly status determination result data includes: The fine mask image data is matched and compared with the pre-stored workpiece standard model data. When calculating the comprehensive similarity index, the mask region is assigned a significant weight by combining the fused significant image data. The outline shape and key structure position are cross-validated by RGB color space binary mask image data, HSV color space binary mask image data, Lab color space binary mask image data and YCbCr color space binary mask image data. Assembly status is determined based on a comprehensive similarity index and a preset assembly qualification threshold, wherein: When the comprehensive similarity index is greater than or equal to the assembly qualification threshold and the consistency verification result does not exceed the limit, the assembly is judged to be qualified. When the overall similarity index is less than the assembly qualification threshold, but the area ratio of the mask missing region is less than or equal to the allowable missing area ratio, and the consistency verification result does not exceed the limit, it is judged as a minor anomaly. When the overall similarity index is less than the assembly qualification threshold, or the proportion of the mask missing area is greater than the allowable missing area proportion, or the consistency verification result exceeds the limit, the assembly is judged as unqualified. The assembly status determination result data is transmitted to the programmable logic controller control unit in real time, and control action instruction data is generated based on the determination result.
9. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, The control action command data includes: For a qualified assembly state, output control action command data to maintain the normal operation of the worktable; For minor abnormal conditions, output control action command data for prompt alarm, including triggering audible and visual prompts, illuminating workstation guidance indicator lights, and displaying review information on the HMI interface; For assembly defects, the system outputs control action commands to force the workbench to stop, trigger audible and visual alarms, and perform automatic reset to prepare for the next round of testing.
10. The photoelectric intelligent error-proofing workbench control method based on deep learning according to claim 1, characterized in that, include: The image acquisition module is used to acquire real-time image data of the target workpiece and synchronous illuminance sensor data at the assembly station where the target workpiece is located, and generate corrected image data. The color space conversion and multi-scale feature construction module is used to convert the corrected image data into RGB color space image datasets, HSV color space image datasets, Lab color space image datasets and YCbCr color space image datasets respectively, and construct multi-scale image pyramids for each color space, extract multi-scale image feature data and form a unified multi-color space multi-scale image feature set. The feature fusion module is used to extract pixel-level color feature data and pixel-level texture feature data from a multi-color space, multi-scale image feature set to generate fused saliency image data. The threshold segmentation module is used to generate binary mask image data; The mask optimization and consistency verification module is used to process binary mask image data and perform consistency verification with binary mask image data in various color spaces. The assembly status determination and execution control module is used to match and compare the fine mask image data that has been verified for consistency with the pre-stored workpiece standard model data, calculate the comprehensive similarity index and determine the assembly status. The determination result is transmitted to the programmable logic controller control unit in real time to drive the execution component to perform alarm, stop or reset operations.