A product assembly missing parts detection system and method based on multimodal vision inspection

Abstract

This invention relates to the fields of computer vision and industrial automation inspection technology, and discloses a product assembly part missing detection system and method based on multimodal vision inspection. The system includes an image acquisition module, a multimodal vision inspection module, an ROI management module, an industrial communication module, a task scheduling module, a hardware control module, and a multi-threaded processing architecture. The multimodal vision inspection module adopts a cascaded strategy of ORB feature matching and template matching. The industrial communication module is based on the Modbus TCP protocol to achieve deep integration with a robotic arm. The task scheduling module supports four working modes to adapt to various production scenarios. The ROI management module supports rapid changeover for multiple regions and specifications. The method includes system initialization, mode selection, image acquisition and preprocessing, ROI partitioning detection, cascaded judgment, result output and communication feedback, and loop execution. This invention achieves automated and high-precision detection of missing product assembly parts, with high detection efficiency and low false negative rate, and can be widely used in the manufacturing industry.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial automation inspection technology, and in particular to a product assembly part missing detection system and method based on multimodal vision inspection. Background Technology

[0002] In the vibrating screen manufacturing industry, the product is a core component, and its assembly quality directly affects the performance and service life of the equipment. The product is usually assembled from multiple small parts (such as clips, gaskets, fasteners, etc.). The omission or missing of any part may lead to the scrapping of the entire unit, causing serious economic losses.

[0003] Traditional manual inspection methods have the following problems: low inspection efficiency, difficulty in meeting the cycle time requirements of production lines, visual fatigue due to prolonged operation, high missed detection rate, difficulty in standardizing inspection standards, and poor consistency.

[0004] Effective traceability of test data cannot be achieved;

[0005] Existing automated inspection systems mostly use a single vision algorithm (such as simple template matching or edge detection), and the inspection accuracy drops significantly under the following complex working conditions: changes in lighting conditions (such as fluctuations in workshop lighting and shadow interference), changes in part posture (such as slight rotation and tilt), surface reflection or complex texture, and the need for rapid changeover in multi-variety, small-batch production.

[0006] In addition, existing vision inspection systems have low communication integration with industrial robotic arms, and mostly use simple switching signals or custom protocols, which have the following drawbacks: they cannot transmit detailed workstation information and inspection parameters; they lack a reliable connection status monitoring mechanism; they are difficult to support complex multi-step inspection tasks; and they are complex to program, requiring high technical skills from operators.

[0007] Therefore, developing a product assembly part missing detection system and method that can adapt to multiple scenarios, has high precision and high stability, and supports multi-station switching and industrial linkage has become the key to solving the pain points of existing technologies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a product assembly part missing detection system and method based on multimodal vision inspection. By integrating multiple vision algorithms, optimizing industrial communication protocols, and innovating task management mechanisms, the invention solves technical problems in existing technologies such as insufficient detection accuracy, low communication integration, and difficulty in model changeover.

[0009] A product assembly part missing detection system based on multimodal vision inspection includes an image acquisition module, an image preprocessing module, a multimodal vision inspection module, a region of interest management module, an industrial communication module, a task scheduling module, a hardware control module, and a main control module.

[0010] The main control module is electrically connected to the image acquisition module, image preprocessing module, multimodal vision detection module, region of interest management module, industrial communication module, task scheduling module, and hardware control module, and performs data interaction and collaborative control.

[0011] The image acquisition module is equipped with an industrial camera to acquire raw image data of the product to be inspected in real time and transmit it to the image preprocessing module. The image preprocessing module preprocesses the raw image data and outputs a standardized inspection image to the multimodal vision inspection module. At the same time, it performs a sharpness detection on the raw image. If the image does not meet the inspection requirements, it sends a supplementary light adjustment command to the hardware control module.

[0012] The Region of Interest (ROI) management module receives the workstation configuration command issued by the main control module and outputs ROI parameters, independent detection thresholds, and multi-scale template library data to the multimodal vision detection module.

[0013] The multi-scale template library is pre-built and stored by the region of interest management module. It contains template images of the same ROI region under multiple scales, multiple poses, and multiple illumination intensities. This module also supports ROI calibration, configuration storage, and dynamic switching of multiple workstations, and can calibrate ROI offset and correct detection coordinates.

[0014] The multimodal visual detection module receives standardized detection images and ROI configuration data, uses ORB feature matching and template matching cascade strategy to complete feature extraction and matching filtering, and outputs feature matching results and two types of matching confidence to the main control module;

[0015] The cascading strategy is to activate template matching as a backup detection mechanism when ORB feature matching fails to meet the judgment criteria.

[0016] The task scheduling module receives the mode configuration command issued by the main control module and outputs scheduling signals for four modes: independent detection, robotic arm linkage, task package, and sequential linkage, and coordinates with each module to perform full-process detection.

[0017] The industrial communication module receives the detection results and control signals from the main control module. Based on the ModbusTCP protocol, it completes bidirectional signal transmission between the main control module and the robotic arm controller, and sends back external trigger commands and workstation data. It also has communication anomaly detection and data transmission protection functions.

[0018] The hardware control module receives status control commands from the main control module and supplementary lighting adjustment commands from the image preprocessing module, executes status indication, alarm triggering and supplementary lighting adjustment actions, and feeds back the hardware operating status to the main control module.

[0019] The main control module aggregates data from each module, integrates two types of matching confidence to complete the final determination of missing parts and locate the missing parts, realizes data scheduling, process control and collaborative operation of the whole system, adopts a multi-threaded asynchronous architecture to achieve thread decoupling, and triggers the missing retry control algorithm and the self-check of the whole system modules.

[0020] Furthermore: The ORB feature matching uses BFMatcher to complete feature point matching, selects high-quality matching pairs through the KNN algorithm, filters false matches by combining ratio test, and uses the RANSAC algorithm to iteratively calculate the homography matrix and remove outliers. Finally, the number of inliers meets the valid matching criteria.

[0021] Furthermore: the template matching traverses all templates in the multi-scale template library, takes the maximum matching degree value as the template matching confidence, compares it with the preset detection threshold, and outputs the template matching result and the corresponding confidence to the multimodal vision detection module for cascaded judgment;

[0022] The number of in-points in ORB feature matching is calculated after removing outliers using the RANSAC algorithm, and template matching uses the Normalized Correlation Coefficient (NCC) algorithm to complete the similarity comparison.

[0023] Furthermore: the image preprocessing module supports manual exposure and white balance parameter configuration, controls camera acquisition parameters through the V4L2 interface, and uses a Gaussian filtering algorithm for image denoising. It can adapt different sizes of filter kernels to complete denoising processing according to the detection scene. The image sharpness detection and fill light adjustment trigger logic forms a closed loop with the fill light adjustment action of the hardware control module to ensure that the acquired image always maintains a stable detection standard.

[0024] The preprocessing is performed sequentially using a grayscale conversion algorithm, a Gaussian denoising algorithm, and a CLAHE local contrast enhancement algorithm.

[0025] Furthermore, the task package mode of the task scheduling module is defined based on a JSON configuration file. The configuration parameters include the task package name, execution step sequence, workstation name, step interval time, and number of loops. The system automatically switches workstations according to the steps and executes the detection task cyclically according to the set number of loops.

[0026] Furthermore, the industrial communication module adopts a 32-bit big-endian byte order register communication mechanism. The registers include a trigger register, a workstation register, a result register, a task package identifier register, a task package control register, a current step register, and a heartbeat register. The heartbeat register periodically sends a fixed heartbeat value. If a heartbeat signal is not received within a preset time, the communication is automatically determined to be abnormal.

[0027] Furthermore: the Region of Interest (ROI) management module supports graphical manual labeling of rectangular and triangular ROIs. ROI configurations are persistently stored in JSON format. Each ROI is independently associated with a multi-scale template library and detection thresholds, and supports dynamic loading and switching across multiple workstations.

[0028] The missing part retry control algorithm is as follows: after the part is initially determined to be missing, multiple re-detections are automatically performed, with no less than two re-detections. If any one of the re-detections is qualified, the part is finally determined to exist. If multiple consecutive detections fail, the part is finally determined to be missing.

[0029] Furthermore: A method for detecting missing parts in product assembly based on multimodal visual inspection includes the following steps:

[0030] S1: System Initialization: Load the multi-station region of interest configuration and multi-scale template library, establish a communication connection with the robotic arm controller based on the ModbusTCP protocol, start multi-threaded image acquisition, vision inspection, and industrial communication, complete the initialization of each module, synchronously calibrate the internal and external parameters of the industrial camera by the main control module, establish the mapping relationship between image pixels and actual physical size, load the part missing judgment weight parameters of each station, complete the initialization of the multi-threaded queue buffer, and ensure smooth and stable data transmission;

[0031] S2: Operation mode configuration: The main control module sends a mode configuration instruction to the task scheduling module, selecting any working mode such as independent detection, robotic arm linkage, task package, or sequential linkage. The system automatically loads the scheduling and operation logic of the corresponding mode.

[0032] S3: Image Acquisition and Preprocessing: The image acquisition module acquires the original product image in real time and transmits it to the image preprocessing module. The image preprocessing module performs a sharpness detection on the original image to determine whether the image quality meets the detection requirements. If the sharpness does not meet the requirements, it sends a supplementary light adjustment command to the hardware control module. After the hardware control module adjusts the brightness of the supplementary light device, the image acquisition module re-acquires the image.

[0033] The qualified original image is sequentially processed by grayscale conversion, image denoising, and local contrast enhancement. At the same time, the enhancement parameters are dynamically adjusted according to the local brightness of the image, and finally a standardized detection image adapted to subsequent detection is output.

[0034] S4: Region of Interest (ROI) Partition Detection: The ROI management module traverses all preset ROIs at the current workstation, identifies product edge feature points, calibrates the offset between the current ROI and the standard area, and automatically corrects the detection coordinates to ensure accurate detection.

[0035] The multimodal visual detection module completes the entire process of feature extraction and multi-level matching for a single region of interest, and outputs the feature matching results and two types of matching confidence to the main control module. For regions that are initially identified as missing, the main control module triggers the missing retry control algorithm, automatically performs multiple re-examinations, and finally outputs the final detection result of the single region of interest.

[0036] S5: Result Fusion Decision: The main control module integrates ORB feature matching confidence and template matching confidence, calculates the comprehensive confidence by weighted summation, and dynamically adjusts the weights of the two types of matching confidence based on changes in ambient lighting. It summarizes all detection results of regions of interest and completes the location of missing parts by combining pixel physical size mapping. If the entire region is qualified, an overall qualified signal is output. If any region is missing, a missing part signal, the missing location and the corresponding confidence are output, and instructions are sent to the hardware control module to trigger an alarm.

[0037] S6: Industrial Communication Feedback: The industrial communication module synchronously writes the detection results, missing location information, detection confidence and image quality indicators into the ModbusTCP result register, verifies the transmitted data to ensure transmission accuracy, clears the detection trigger signal, and synchronously updates the step index and detection log in task package mode. If a communication transmission failure occurs, it automatically performs data retransmission. If the retransmission fails, it triggers a communication alarm and retains the fault information.

[0038] S7: Loop-loop control: Based on the currently selected operating mode, determine whether to continue to the next round of detection or wait for an external trigger signal. After each batch detection task is completed, the main control module automatically performs a self-check of the status of all modules of the system, checks the operating status of camera acquisition, communication links, and template library, and generates a self-check report. If a module abnormality is detected, a status prompt is triggered synchronously and detection is paused. After the fault is resolved, operation is resumed, realizing closed-loop management of the entire process.

[0039] Furthermore: the robotic arm linkage mode process is as follows: after the robotic arm moves to the designated detection station, it writes a trigger signal to the trigger register. After the system detects the trigger signal, it reads the station register to obtain the current station identifier and switches to the corresponding region of interest configuration. It then sequentially executes the entire process of image acquisition, preprocessing, partition detection, and result decision. After the detection is completed, the result is written to the result register. The robotic arm performs the corresponding sorting action according to the detection result. After the action is completed, a synchronization signal is fed back. After the system receives the signal, it starts the next round of station detection, realizing full-process linkage synchronization.

[0040] The task package mode process is as follows: After receiving the task package start signal, the system parses the corresponding configuration file information and completes the workstation switching, image acquisition, area detection, and result feedback in sequence according to the step sequence. The steps are executed with a preset time delay. After all steps are completed, the task package control signal is cleared and the system waits for the next round of task instructions. If a single step times out during the detection process, the abnormal step is automatically skipped and the fault information is recorded. The corresponding image data is retained throughout the process for subsequent traceability.

[0041] Furthermore: The image quality assessment and supplementary lighting adjustment adopt a closed-loop control algorithm, which dynamically adapts and adjusts the brightness of the supplementary lighting device according to the real-time image sharpness detection results, so that the acquired image always maintains a stable detection standard and ensures that the detection accuracy is not affected by the light throughout the process.

[0042] During the dynamic calibration of the region of interest, an adaptive edge detection algorithm is used to extract the edge features of the product. The detection parameters are automatically adjusted according to the real-time grayscale distribution of the image to accurately locate the product outline, ensuring accurate calibration of the region of interest and no deviation in the detection position.

[0043] The present invention has the following beneficial effects:

[0044] 1. Significantly improved anti-interference capability and detection accuracy, adaptable to complex industrial conditions: This invention adopts a multimodal visual detection strategy that cascades ORB feature matching and template matching, combined with KNN algorithm, ratio test, RANSAC outlier removal and NCC normalized correlation coefficient comparison, which can effectively overcome interference factors such as workshop lighting fluctuations, slight rotation and tilt of parts, surface reflection and complex texture, and break through the detection limitations of single visual algorithms; at the same time, it is equipped with a standardized preprocessing process of image grayscale conversion, Gaussian denoising, and CLAHE local contrast enhancement, as well as Laplacian sharpness detection and incremental PID supplementary lighting closed-loop control, which ensures stable image quality from the source, and then achieves ROI dynamic calibration and coordinate correction through adaptive edge detection, which greatly reduces the probability of missed detection and false detection, and can still maintain high-precision detection under complex conditions;

[0045] 2. Flexible multi-station adaptation and rapid changeover to meet the needs of multi-variety, small-batch production: The system has a built-in multi-scale template library with multiple scales, orientations, and lighting intensities. It supports graphical manual calibration of rectangular and triangular ROI areas. ROI configurations are persistently stored in JSON format, and each ROI can be independently associated with a detection threshold. It can realize dynamic loading and one-click switching of multi-station ROI parameters and template data. Without the need for repeated debugging of hardware and algorithm parameters, it can quickly complete the detection and changeover of different product models, perfectly adapting to the multi-variety, small-batch production mode of the vibrating screen industry, greatly shortening the production line changeover time and improving production flexibility.

[0046] 3. More reliable detection results and enhanced system automation and linkage capabilities: For areas initially identified as missing, the system triggers a missing retry control algorithm, automatically performing at least two re-detections. Only after multiple consecutive failures is the part ultimately determined to be missing, completely avoiding misjudgments caused by momentary interference. Simultaneously, the main control module achieves precise positioning of missing parts by weighted summation and fusion of two types of matching confidence scores, dynamically adjusting the weights based on illumination, and combining pixel-to-physical size mapping. A standardized industrial communication link is built based on the Modbus TCP protocol, coupled with a 32-bit big-endian register and heartbeat detection mechanism, to achieve high-precision bidirectional linkage with the robotic arm. The task scheduling module supports four modes: independent detection, robotic arm linkage, task package, and sequential linkage, which can fully match the production line cycle and achieve fully unmanned automatic detection.

[0047] 4. The system operates stably and controllably, replacing manual labor to reduce costs, increase efficiency, and achieve full-chain traceability: The main control module adopts a multi-threaded asynchronous architecture to decouple the threads of each module, making data scheduling and process control more efficient. Coupled with a full-module self-inspection mechanism, it can check for faults in camera acquisition, communication links, and template library operation in real time and generate self-inspection reports. When an anomaly occurs, the detection is automatically suspended, forming a closed-loop management of the entire process and ensuring the long-term stable operation of the system. This system can completely replace manual visual inspection, unify inspection standards, eliminate inspection errors caused by human visual fatigue and inconsistent standards, and match the inspection efficiency with the production line rhythm. At the same time, it retains inspection data and images throughout the process, realizing full-chain traceability of the inspection process, effectively reducing labor costs and product scrap losses, and improving the level of product assembly quality control. Attached Figure Description

[0048] Figure 1 This is a flowchart of a product assembly part missing detection method based on multimodal visual inspection. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] Please see the appendix Figure 1 The present invention provides an embodiment 1: a product assembly part missing detection system based on multimodal visual inspection, including an image acquisition module, an image preprocessing module, a multimodal visual inspection module, a region of interest management module, an industrial communication module, a task scheduling module, a hardware control module, and a main control module;

[0051] The main control module is electrically connected to the image acquisition module, image preprocessing module, multimodal vision detection module, region of interest management module, industrial communication module, task scheduling module, and hardware control module, and performs data interaction and collaborative control.

[0052] The image acquisition module is equipped with an industrial camera to acquire raw image data of the product to be inspected in real time and transmit it to the image preprocessing module. The image preprocessing module preprocesses the raw image data and outputs a standardized inspection image to the multimodal vision inspection module. At the same time, it performs a sharpness detection on the raw image. If the image does not meet the inspection requirements, it sends a supplementary light adjustment command to the hardware control module.

[0053] The Region of Interest (ROI) management module receives workstation configuration instructions from the main control module and outputs ROI parameters, independent detection thresholds, and multi-scale template library data to the multimodal vision detection module.

[0054] The multi-scale template library is pre-built and stored by the Region of Interest (ROI) management module. It contains template images of the same ROI region at multiple scales, in multiple poses, and under multiple illumination intensities. This module also supports ROI calibration, configuration storage, and dynamic switching of multiple workstations, and can calibrate ROI offset and correct detection coordinates.

[0055] The multimodal visual detection module receives standardized detection images and ROI configuration data, uses ORB feature matching and template matching cascade strategy to complete feature extraction and matching filtering, and outputs feature matching results and two types of matching confidence to the main control module;

[0056] The cascaded strategy is to activate template matching as a backup detection mechanism when ORB feature matching fails to meet the judgment criteria.

[0057] The task scheduling module receives the mode configuration instructions issued by the main control module and outputs scheduling signals for four modes: independent detection, robotic arm linkage, task package, and sequential linkage, and coordinates with each module to execute the entire process of detection.

[0058] The industrial communication module receives the detection results and control signals from the main control module. Based on the Modbus TCP protocol, it completes bidirectional signal transmission between the main control module and the robotic arm controller, and sends back external trigger commands and workstation data. It also has the functions of communication anomaly detection and data transmission protection.

[0059] The hardware control module receives status control commands from the main control module and supplementary lighting adjustment commands from the image preprocessing module, executes status indication, alarm triggering and supplementary lighting adjustment actions, and feeds back the hardware operating status to the main control module.

[0060] The main control module aggregates data from various modules, integrates two types of matching confidence to complete the final determination of missing parts and locate missing parts, realizes data scheduling, process control and collaborative operation of the whole system, adopts a multi-threaded asynchronous architecture to achieve thread decoupling, and triggers the missing retry control algorithm and system-wide module self-check;

[0061] ORB feature matching uses BFMatcher to complete feature point matching, selects high-quality matching pairs through the KNN algorithm, filters false matches by combining ratio test, and uses the RANSAC algorithm to iteratively calculate the homography matrix and remove outlier points. Finally, the number of inliers meets the valid matching criteria.

[0062] Template matching iterates through all templates in the multi-scale template library, takes the maximum matching degree value as the template matching confidence, compares it with the preset detection threshold, and outputs the template matching result and the corresponding confidence to the multimodal vision detection module for cascaded judgment.

[0063] The number of in-points in ORB feature matching is calculated after removing outliers using the RANSAC algorithm. Template matching uses the Normalized Correlation Coefficient (NCC) algorithm to perform similarity comparison.

[0064] The image preprocessing module supports manual exposure and white balance parameter configuration. It controls the camera acquisition parameters through the V4L2 interface. The image denoising uses a Gaussian filtering algorithm and can adapt to different sizes of filter kernels according to the detection scene to complete the denoising process. The image sharpness detection and supplementary light adjustment trigger logic forms a closed loop with the supplementary light adjustment action of the hardware control module to ensure that the acquired image always maintains a stable detection standard.

[0065] Preprocessing was performed sequentially using grayscale conversion algorithm, Gaussian denoising algorithm, and CLAHE local contrast enhancement algorithm.

[0066] The task package mode of the task scheduling module is defined based on a JSON configuration file. The configuration parameters include the task package name, execution step sequence, workstation name, step interval time, and number of loops. The system automatically switches workstations according to the steps and executes the detection task in a loop according to the set number of loops.

[0067] The industrial communication module adopts a 32-bit big-endian byte order register communication mechanism. The registers include a trigger register, workstation register, result register, task package identifier register, task package control register, current step register, and heartbeat register. The heartbeat register periodically sends a fixed heartbeat value. If no heartbeat signal is received after a preset time, the communication is automatically judged to be abnormal.

[0068] The Region of Interest (ROI) management module supports graphical manual labeling of rectangular and triangular ROIs. ROI configurations are persistently stored in JSON format. Each ROI is independently associated with a multi-scale template library and detection thresholds, and supports dynamic loading and switching between multiple workstations.

[0069] The missing part retry control algorithm is as follows: after the part is initially determined to be missing, multiple re-detections are automatically performed, with no less than two re-detections. If any one of the re-detections is successful, the part is finally determined to exist. If multiple consecutive re-detections fail, the part is finally determined to be missing.

[0070] An embodiment 2 of the present invention provides a method for detecting missing parts in product assembly based on multimodal visual inspection, comprising the following steps:

[0071] S1: System Initialization: Load the multi-station region of interest configuration and multi-scale template library, establish a communication connection with the robotic arm controller based on the ModbusTCP protocol, start multi-threaded image acquisition, vision inspection, and industrial communication, complete the initialization of each module, synchronously calibrate the internal and external parameters of the industrial camera by the main control module, establish the mapping relationship between image pixels and actual physical size, load the part missing judgment weight parameters of each station, complete the initialization of the multi-threaded queue buffer, and ensure smooth and stable data transmission;

[0072] S2: Operation mode configuration: The main control module sends a mode configuration instruction to the task scheduling module, selecting any working mode such as independent detection, robotic arm linkage, task package, or sequential linkage. The system automatically loads the scheduling and operation logic of the corresponding mode.

[0073] S3: Image Acquisition and Preprocessing: The image acquisition module acquires the original product image in real time and transmits it to the image preprocessing module. The image preprocessing module performs a sharpness detection on the original image to determine whether the image quality meets the detection requirements. If the sharpness does not meet the requirements, it sends a supplementary light adjustment command to the hardware control module. After the hardware control module adjusts the brightness of the supplementary light device, the image acquisition module re-acquires the image.

[0074] The qualified original image is sequentially processed by grayscale conversion, image denoising, and local contrast enhancement. At the same time, the enhancement parameters are dynamically adjusted according to the local brightness of the image, and finally a standardized detection image adapted to subsequent detection is output.

[0075] S4: Region of Interest (ROI) Partition Detection: The ROI management module traverses all preset ROIs at the current workstation, identifies product edge feature points, calibrates the offset between the current ROI and the standard area, and automatically corrects the detection coordinates to ensure accurate detection.

[0076] The multimodal visual detection module completes the entire process of feature extraction and multi-level matching for a single region of interest, and outputs the feature matching results and two types of matching confidence to the main control module. For regions that are initially identified as missing, the main control module triggers the missing retry control algorithm, automatically performs multiple re-examinations, and finally outputs the final detection result of the single region of interest.

[0077] S5: Result Fusion Decision: The main control module integrates ORB feature matching confidence and template matching confidence, calculates the comprehensive confidence by weighted summation, and dynamically adjusts the weights of the two types of matching confidence based on changes in ambient lighting. It summarizes all detection results of regions of interest and completes the location of missing parts by combining pixel physical size mapping. If the entire region is qualified, an overall qualified signal is output. If any region is missing, a missing part signal, the missing location and the corresponding confidence are output, and instructions are sent to the hardware control module to trigger an alarm.

[0078] S6: Industrial Communication Feedback: The industrial communication module synchronously writes the detection results, missing location information, detection confidence and image quality indicators into the ModbusTCP result register, verifies the transmitted data to ensure transmission accuracy, clears the detection trigger signal, and synchronously updates the step index and detection log in task package mode. If a communication transmission failure occurs, it automatically performs data retransmission. If the retransmission fails, it triggers a communication alarm and retains the fault information.

[0079] S7: Loop-loop control: Based on the currently selected operating mode, determine whether to continue to the next round of detection or wait for an external trigger signal. After each batch detection task is completed, the main control module automatically performs a self-check of the status of all modules of the system, checks the operating status of camera acquisition, communication links, and template library, and generates a self-check report. If a module abnormality is detected, a status prompt is triggered synchronously and detection is paused. After the fault is resolved, operation is resumed, realizing closed-loop management of the entire process.

[0080] The robotic arm linkage mode process is as follows: After the robotic arm moves to the designated detection station, it writes a trigger signal to the trigger register. After the system detects the trigger signal, it reads the station register to obtain the current station identifier and switches to the corresponding region of interest configuration. It then sequentially executes the entire process of image acquisition, preprocessing, partition detection, and result decision. After the detection is completed, the result is written to the result register. The robotic arm performs the corresponding sorting action according to the detection result. After the action is completed, a synchronization signal is fed back. When the system receives the signal, it starts the next round of station detection, thus achieving full-process linkage synchronization.

[0081] The task package mode process is as follows: After receiving the task package start signal, the system parses the corresponding configuration file information and completes the workstation switching, image acquisition, area detection, and result feedback in sequence according to the step sequence. The steps are executed with a preset time delay. After all steps are completed, the task package control signal is cleared and the system waits for the next round of task instructions. If a single step times out during the detection process, the abnormal step is automatically skipped and the fault information is recorded. The corresponding image data is retained throughout the process for easy subsequent traceability.

[0082] Image quality assessment and supplementary lighting adjustment adopt a closed-loop control algorithm. Based on the real-time image sharpness detection results, the brightness of the supplementary lighting device is dynamically adjusted to ensure that the acquired image always maintains a stable detection standard and that the detection accuracy is not affected by the lighting throughout the process.

[0083] During the dynamic calibration of the region of interest, an adaptive edge detection algorithm is used to extract the edge features of the product. The detection parameters are automatically adjusted according to the real-time grayscale distribution of the image to accurately locate the product outline, ensuring accurate calibration of the region of interest and no deviation in the detection position.

[0084] This invention provides an embodiment 3: This section supplements and completes the control algorithm, function formula, parameter definition and physical meaning of a product assembly part missing detection system and method based on multimodal visual inspection.

[0085] I. The image preprocessing module algorithm includes:

[0086] RGB grayscale conversion algorithms convert color images into single-channel grayscale images, reducing computational complexity.

[0087]

[0088] Explanation of the parameters:

[0089] : Grayscale image pixel values, range of values ;

[0090] The original image's red, green, and blue channel pixel values, and their range. .

[0091] Two-dimensional Gaussian kernel function:

[0092]

[0093] Filtered output:

[0094]

[0095] Explanation of the parameters:

[0096] Target pixel coordinates; Gaussian kernel standard deviation, values ;

[0097] : Filter kernel half-length, kernel size is ,support ;

[0098] : Original image pixels; : Pixels in the filtered image.

[0099] CLAHE Local Contrast Enhancement

[0100] Contrast limit threshold:

[0101]

[0102] Enhanced output:

[0103]

[0104] Explanation of the parameters:

[0105] Contrast limit threshold; (default);

[0106] Number of image blocks, default ; : Adjustment coefficient, ;

[0107] Total number of pixels per block; Histogram equalization mapping function.

[0108] Image sharpness detection (Laplacian variance method)

[0109] Laplace operator:

[0110]

[0111] Sharpness value:

[0112] nn

[0113] Judgment rules:

[0114]

[0115] Parameter explanation:

[0116] : Variance function; : Sharpness value; (Default threshold);

[0117] II. The multimodal visual detection module algorithm includes:

[0118] ORB feature matching

[0119] Hamming distance (BFMatcher)

[0120]

[0121] Parameter explanation: A 256-bit ORB descriptor; The smaller the value, the better the match.

[0122] KNN ratio test

[0123]

[0124] Parameter explanation: The optimal matching distance; The second-best matching distance; (default).

[0125] RANSAC homography matrix and interior point determination

[0126] Projection formula:

[0127]

[0128] Formula for calculating reprojection error:

[0129]

[0130] Interior point determination:

[0131]

[0132] Parameter explanation: for Homography matrix; Pixel (reprojection error threshold); These are standard coordinates.

[0133] Template matching (NCC normalized correlation coefficient)

[0134] NCC calculation formula:

[0135]

[0136] Template matching confidence:

[0137]

[0138] determination: ( () indicates a successful match.

[0139] Parameter explanation: To detect ROI; For templates; The mean; It is a multi-scale template.

[0140] III. Dynamic Calibration Algorithm for Region of Interest (ROI)

[0141] Adaptive Canny edge detection

[0142]

[0143] Parameter explanation: ; The average grayscale value of the ROI; These are the high and low thresholds.

[0144] ROI offset calibration

[0145] Offset:

[0146]

[0147] Coordinate correction:

[0148]

[0149] Parameter explanation: Real-time edge coordinates; Standard coordinates; ROI coordinates after calibration.

[0150] IV. Result Fusion Decision Algorithm

[0151] Overall confidence level:

[0152]

[0153] Dynamic weights:

[0154]

[0155] Final judgment:

[0156]

[0157] Parameter explanation: ORB confidence level; ; Real-time brightness; Standard brightness; .

[0158] V. Missing Retry Control Algorithm

[0159] Set the number of retries (Default 3 times), Detection result set Judgment rules:

[0160]

[0161] VI. Closed-loop control of supplemental lighting (incremental PID)

[0162] error:

[0163] Controlling the increment:

[0164]

[0165] Output brightness:

[0166]

[0167] Parameter explanation: ; Current fill light brightness .

[0168] VII. Industrial Communication Heartbeat Detection Algorithm

[0169]

[0170] Parameter explanation: (Timeout threshold).

[0171] VIII. Task Package Mode Scheduling Algorithm

[0172] Total task duration:

[0173]

[0174] Parameter explanation: Number of steps; Single-step detection time; Step interval; Number of loops.

[0175] An embodiment 4 of the present invention is implemented according to the product assembly part missing detection system and method based on multimodal visual inspection. It is used for the automated detection of missing parts in vibrating screen product assembly. It can be adapted to multi-station switching and linkage operation with industrial robotic arms to realize the full automation of detection, judgment, sorting and data traceability.

[0176] 1. System Hardware Configuration

[0177] The image acquisition module uses a 1920×1080 resolution industrial camera with a frame rate of ≥30fps. Exposure and white balance parameters are adjusted via the V4L2 interface, and it is equipped with an adjustable brightness ring light device. The industrial communication module uses an Ethernet interface, based on the Modbus TCP protocol, with a communication rate of 100Mbps, and supports 32-bit big-endian byte order register interaction. The hardware control module uses a GPIO expansion board to link the power indicator, operation indicator, alarm indicator, buzzer, and light adjustment device. The system main controller uses an industrial control computer, equipped with a multi-core processor, 16GB of memory, and a 512GB solid-state drive. It adopts a multi-threaded asynchronous architecture to support the decoupled operation of image acquisition, visual inspection, industrial communication, and task scheduling threads.

[0178] 2. Core Functional Module Parameter Configuration

[0179] 2.1 Image Preprocessing Module

[0180] The preprocessing process sequentially performs grayscale conversion, Gaussian filtering for noise reduction, and CLAHE local contrast enhancement. The Gaussian filtering is adapted to 3×3 / 5×5 / 7×7 kernel sizes. The Laplacian variance method is used for image sharpness detection, and the sharpness threshold is set to 30. If the threshold is not met, the supplementary lighting closed-loop adjustment is triggered, and the supplementary lighting brightness is dynamically adjusted through an incremental PID algorithm.

[0181] 2.2 Multimodal Vision Inspection Module

[0182] A cascaded strategy of ORB feature matching and template matching is adopted: ORB feature matching uses BFMatcher matching, KNN to select high-quality matching pairs, ratio test to filter mismatches, and RANSAC algorithm to remove outliers. The part is considered qualified if the number of effective inliers reaches the standard. Template matching uses the NCC normalized correlation coefficient algorithm with a matching threshold of 0.85. The maximum matching degree is taken as the confidence score by traversing the multi-scale template library. The multi-scale template library pre-stores template images of the same ROI region under multiple scales, multiple poses (0°, ±15°, ±30°), and multiple illumination intensities. The results are fused by weighted summation, with ORB confidence weight of 0.7 and template confidence weight of 0.3. The part is considered to exist if the comprehensive confidence score is ≥0.82.

[0183] 2.3 Region of Interest (ROI) Management Module

[0184] It supports graphical calibration of rectangular and triangular ROIs, and ROI parameters are persistently stored in JSON format. Each ROI is independently bound to a detection threshold and template library. It adopts an adaptive edge detection algorithm to calibrate ROI offsets and automatically corrects detection coordinates. It supports dynamic switching and loading of ROI configurations for 6 workstations.

[0185] 2.4 Industrial Communication Module

[0186] The registers use 32-bit big-endian byte order and include the trigger register (address 21900), workstation register (address 21902), result register (address 21904), task package identifier register (address 21908), task package control register (address 21910), current step register (address 21912), and heartbeat register (address 21914). The heartbeat register sends a fixed heartbeat value every 100ms. If no heartbeat is received within 500ms, a communication abnormality is determined. In case of an abnormality, automatic retransmission and alarm trigger will be initiated.

[0187] 2.5 Missing Retry and Task Scheduling Module

[0188] The missing part retry control algorithm sets the number of retries N=3. After the initial determination of missing part, it automatically re-inspects twice. If any one of the retryes is qualified, the part is finally determined to exist. If three consecutive retryes are unqualified, the part is confirmed to be missing. The task scheduling module supports four modes: independent detection, robotic arm linkage, task package, and sequential linkage. The task package mode is based on a JSON configuration file, which defines the workstation sequence, step interval, and number of loops.

[0189] 3. Detection Method Execution Procedure

[0190] S1: System initialization, loading multi-station ROI configuration and multi-scale template library, establishing robotic arm communication connection based on ModbusTCP, starting multi-threaded operation, calibrating camera internal and external parameters, establishing pixel-to-physical size mapping relationship, and initializing thread queue buffer;

[0191] S2: The main control module issues a configuration command, selects the robotic arm linkage + task package operation mode, and the system loads the corresponding scheduling logic;

[0192] S3: The image acquisition module acquires the original image of the product. The preprocessing module first performs a sharpness detection. If the image does not meet the standard, the hardware module is instructed to adjust the supplementary light and re-acquire the image. The qualified image undergoes grayscale conversion, Gaussian denoising, and CLAHE enhancement in sequence, and outputs a standardized test image.

[0193] S4: The ROI management module traverses all ROIs in the current workstation, calibrates the offset and corrects the detection coordinates through product edge features; the multimodal detection module performs cascaded matching and outputs two types of confidence to the main controller; the main controller triggers a retry algorithm for the initial missing area, and outputs the final result of a single ROI after multiple re-inspections.

[0194] S5: The main control module integrates two types of confidence to calculate the comprehensive confidence, dynamically adjusts the weights to adapt to changes in lighting, summarizes all ROI results, and completes the location of missing parts; if the entire area is qualified, a qualified signal is output; if any area is missing, a missing signal and location are output, and the hardware module is instructed to trigger an alarm.

[0195] S6: The industrial communication module writes the detection results, positioning information, and confidence level into the ModbusTCP result register to complete data verification and trigger signal clearing; in task package mode, it synchronously updates the step index and detection log, automatically retransmits in case of communication failure, and alarms and retains fault information if it fails.

[0196] S7: It performs cyclic detection according to the current operating mode. After each batch task is completed, the main controller performs a full module self-check, checks the status of the camera, communication and template library and generates a self-check report. If there is an abnormality, the detection is paused. After the fault is resolved, it will automatically resume, realizing closed-loop control of the whole process.

[0197] 4. Details of robotic arm linkage and task package execution

[0198] After the robotic arm moves to the designated workstation, it writes a trigger signal to the trigger register. The system reads the workstation identifier and switches the ROI configuration. After completing the entire inspection process, the result is written to the register. The robotic arm performs sorting according to the results. Qualified items are transferred to the next process, while defective items are sent to the rework station. After a synchronization signal is received, the next round of inspection is started. In task package mode, the system parses the JSON configuration, automatically switches workstations, performs delayed execution, and performs cyclic inspections according to the step sequence. If a single step times out, it is skipped and the fault is recorded. The entire inspection process saves inspection images and data, supporting subsequent quality traceability.

[0199] This embodiment fully implements the system architecture, algorithm strategy, communication mechanism and control process described in the patent, and can effectively adapt to complex working conditions such as workshop lighting fluctuations, part posture deviations, and multiple model changes, to achieve high-precision and high-stability detection of missing parts in product assembly.

[0200] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A product assembly part missing detection system based on multimodal vision inspection, characterized in that, include: Image acquisition module, image preprocessing module, multimodal vision detection module, region of interest management module, industrial communication module, task scheduling module, hardware control module, and main control module; The main control module is electrically connected to the image acquisition module, image preprocessing module, multimodal vision detection module, region of interest management module, industrial communication module, task scheduling module, and hardware control module, and performs data interaction and collaborative control. The image acquisition module is equipped with an industrial camera to acquire raw image data of the product to be inspected in real time and transmit it to the image preprocessing module. The image preprocessing module preprocesses the raw image data and outputs a standardized inspection image to the multimodal vision inspection module. At the same time, it performs a sharpness detection on the raw image. If the image does not meet the inspection requirements, it sends a supplementary light adjustment command to the hardware control module. The Region of Interest (ROI) management module receives the workstation configuration command issued by the main control module and outputs ROI parameters, independent detection thresholds, and multi-scale template library data to the multimodal vision detection module. The multi-scale template library is pre-built and stored by the region of interest management module. It contains template images of the same ROI region under multiple scales, multiple poses, and multiple illumination intensities. This module also supports ROI calibration, configuration storage, and dynamic switching of multiple workstations, and can calibrate ROI offset and correct detection coordinates. The multimodal visual detection module receives standardized detection images and ROI configuration data, uses ORB feature matching and template matching cascade strategy to complete feature extraction and matching filtering, and outputs feature matching results and two types of matching confidence to the main control module; The cascading strategy is to activate template matching as a backup detection mechanism when ORB feature matching fails to meet the judgment criteria. The task scheduling module receives the mode configuration command issued by the main control module and outputs scheduling signals for four modes: independent detection, robotic arm linkage, task package, and sequential linkage, and coordinates with each module to perform full-process detection. The industrial communication module receives the detection results and control signals from the main control module. Based on the ModbusTCP protocol, it completes bidirectional signal transmission between the main control module and the robotic arm controller, and sends back external trigger commands and workstation data. It also has communication anomaly detection and data transmission protection functions. The hardware control module receives status control commands from the main control module and supplementary lighting adjustment commands from the image preprocessing module, executes status indication, alarm triggering and supplementary lighting adjustment actions, and feeds back the hardware operating status to the main control module. The main control module aggregates data from each module, integrates two types of matching confidence to complete the final determination of missing parts and locate the missing parts, realizes data scheduling, process control and collaborative operation of the whole system, adopts a multi-threaded asynchronous architecture to achieve thread decoupling, and triggers the missing retry control algorithm and the self-check of the whole system modules.

2. A product assembly part missing detection system based on multi-modal vision detection according to claim 1, characterized in that, The ORB feature matching uses BFMatcher to complete feature point matching, selects high-quality matching pairs through the KNN algorithm, filters false matches by combining ratio test, and uses the RANSAC algorithm to iteratively calculate the homography matrix and remove outliers. Finally, the number of inliers meets the valid matching criteria.

3. The product assembly part missing detection system based on multi-modal vision detection according to claim 1, characterized in that, The template matching process traverses all templates in the multi-scale template library, takes the maximum matching degree value as the template matching confidence, compares it with the preset detection threshold, and outputs the template matching result and the corresponding confidence to the multimodal vision detection module for cascaded judgment. The number of in-points in ORB feature matching is calculated after removing outliers using the RANSAC algorithm, and template matching uses the Normalized Correlation Coefficient (NCC) algorithm to complete the similarity comparison.

4. The product assembly part missing detection system based on multi-modal vision detection of claim 1, wherein, The image preprocessing module supports manual exposure and white balance parameter configuration. It controls the camera acquisition parameters through the V4L2 interface. The image denoising uses a Gaussian filtering algorithm and can adapt to different sizes of filter kernels according to the detection scene to complete the denoising process. The image sharpness detection and supplementary light adjustment trigger logic forms a closed loop with the supplementary light adjustment action of the hardware control module to ensure that the acquired image always maintains a stable detection standard. The preprocessing is performed sequentially using a grayscale conversion algorithm, a Gaussian denoising algorithm, and a CLAHE local contrast enhancement algorithm.

5. The product assembly part missing detection system based on multi-modal vision detection of claim 1, wherein, The task package mode of the task scheduling module is defined based on a JSON configuration file. The configuration parameters include the task package name, execution step sequence, workstation name, step interval time, and number of loops. The system automatically switches workstations according to the steps and executes the detection task in a loop according to the set number of loops.

6. The product assembly part missing detection system based on multi-modal vision detection of claim 1, wherein, The industrial communication module adopts a 32-bit big-endian byte order register communication mechanism. The registers include a trigger register, a workstation register, a result register, a task package identifier register, a task package control register, a current step register, and a heartbeat register. The heartbeat register periodically sends a fixed heartbeat value. If no heartbeat signal is received after a preset time, the communication is automatically determined to be abnormal.

7. The product assembly part missing detection system based on multimodal visual inspection according to claim 1, wherein the region of interest management module supports graphical manual calibration of rectangles and triangles ROI, the ROI configuration is persistently stored in JSON format, each ROI is independently associated with a multi-scale template library and detection threshold, and supports dynamic loading and switching of multiple workstations; The missing part retry control algorithm is as follows: after the part is initially determined to be missing, multiple re-detections are automatically performed, with no less than two re-detections. If any one of the re-detections is qualified, the part is finally determined to exist. If multiple consecutive detections fail, the part is finally determined to be missing.

8. A method for detecting missing parts in product assembly based on multimodal visual inspection, applied to the product assembly part missing detection system based on multimodal visual inspection as described in any one of claims 1-7, characterized in that, Includes the following steps: S1: System Initialization: Load the multi-station region of interest configuration and multi-scale template library, establish a communication connection with the robotic arm controller based on the ModbusTCP protocol, start multi-threaded image acquisition, vision inspection, and industrial communication, complete the initialization of each module, synchronously calibrate the internal and external parameters of the industrial camera by the main control module, establish the mapping relationship between image pixels and actual physical size, load the part missing judgment weight parameters of each station, complete the initialization of the multi-threaded queue buffer, and ensure smooth and stable data transmission; S2: Operation mode configuration: The main control module sends a mode configuration instruction to the task scheduling module, selecting any working mode such as independent detection, robotic arm linkage, task package, or sequential linkage. The system automatically loads the scheduling and operation logic of the corresponding mode. S3: Image Acquisition and Preprocessing: The image acquisition module acquires the original product image in real time and transmits it to the image preprocessing module. The image preprocessing module performs a sharpness detection on the original image to determine whether the image quality meets the detection requirements. If the sharpness does not meet the requirements, it sends a supplementary light adjustment command to the hardware control module. After the hardware control module adjusts the brightness of the supplementary light device, the image acquisition module re-acquires the image. The qualified original image is sequentially processed by grayscale conversion, image denoising, and local contrast enhancement. At the same time, the enhancement parameters are dynamically adjusted according to the local brightness of the image, and finally a standardized detection image adapted to subsequent detection is output. S4: Region of Interest (ROI) Partition Detection: The ROI management module traverses all preset ROIs at the current workstation, identifies product edge feature points, calibrates the offset between the current ROI and the standard area, and automatically corrects the detection coordinates to ensure accurate detection. The multimodal visual detection module completes the entire process of feature extraction and multi-level matching for a single region of interest, and outputs the feature matching results and two types of matching confidence to the main control module. For regions that are initially identified as missing, the main control module triggers the missing retry control algorithm, automatically performs multiple re-examinations, and finally outputs the final detection result of the single region of interest. S5: Result Fusion Decision: The main control module integrates ORB feature matching confidence and template matching confidence, calculates the comprehensive confidence by weighted summation, and dynamically adjusts the weights of the two types of matching confidence based on changes in ambient lighting. It summarizes all detection results of regions of interest and completes the location of missing parts by combining pixel physical size mapping. If the entire region is qualified, an overall qualified signal is output. If any region is missing, a missing part signal, the missing location and the corresponding confidence are output, and instructions are sent to the hardware control module to trigger an alarm. S6: Industrial Communication Feedback: The industrial communication module synchronously writes the detection results, missing location information, detection confidence and image quality indicators into the ModbusTCP result register, verifies the transmitted data to ensure transmission accuracy, clears the detection trigger signal, and synchronously updates the step index and detection log in task package mode. If a communication transmission failure occurs, it automatically performs data retransmission. If the retransmission fails, it triggers a communication alarm and retains the fault information. S7: Loop-loop control: Based on the currently selected operating mode, determine whether to continue to the next round of detection or wait for an external trigger signal. After each batch detection task is completed, the main control module automatically performs a self-check of the status of all modules of the system, checks the operating status of camera acquisition, communication links, and template library, and generates a self-check report. If a module abnormality is detected, a status prompt is triggered synchronously and detection is paused. After the fault is resolved, operation is resumed, realizing closed-loop management of the entire process.

9. The method of claim 8, wherein, The robotic arm linkage mode process is as follows: After the robotic arm moves to the designated detection station, it writes a trigger signal to the trigger register. After the system detects the trigger signal, it reads the station register to obtain the current station identifier and switches to the corresponding region of interest configuration. It then sequentially executes the entire process of image acquisition, preprocessing, partition detection, and result decision. After the detection is completed, the result is written to the result register. The robotic arm performs the corresponding sorting action according to the detection result. After the action is completed, a synchronization signal is fed back. When the system receives the signal, it starts the next round of station detection, thus realizing full-process linkage synchronization. The task package mode process is as follows: After receiving the task package start signal, the system parses the corresponding configuration file information and completes the workstation switching, image acquisition, area detection, and result feedback in sequence according to the step sequence. The steps are executed with a preset time delay. After all steps are completed, the task package control signal is cleared and the system waits for the next round of task instructions. If a single step times out during the detection process, the abnormal step is automatically skipped and the fault information is recorded. The corresponding image data is retained throughout the process for subsequent traceability.

10. The method of claim 8, wherein, The image quality assessment and supplementary lighting adjustment adopt a closed-loop control algorithm. Based on the real-time image clarity detection results, the brightness of the supplementary lighting device is dynamically adjusted to ensure that the acquired image always maintains a stable detection standard and that the detection accuracy is not affected by the lighting throughout the process. During the dynamic calibration of the region of interest, an adaptive edge detection algorithm is used to extract the edge features of the product. The detection parameters are automatically adjusted according to the real-time grayscale distribution of the image to accurately locate the product outline, ensuring accurate calibration of the region of interest and no deviation in the detection position.

Images