Method, system and apparatus for fast detection of chewing gum defects in an industrial setting

Abstract

The present application belongs to the technical field of industrial automation detection, and discloses a method, system and device for rapid detection of chewing gum defects in an industrial scene. The method comprises the following steps: based on an industrial camera and a photoelectric sensor, collecting multi-modal data containing image data and position signals; pre-processing and multi-channel concurrent optimization of the collected multi-channel images; spatio-temporal alignment of the pre-processed multi-modal data to realize multi-modal data fusion; based on the fused multi-modal data, using an optimized YOLOv8 target detection model to realize rapid defect detection; and based on machine vision, realizing real-time detection of line congestion. The chewing gum defect rapid detection method, system and device in the industrial scene are suitable for real-time detection of multi-channel defects and monitoring of line congestion in a chewing gum production line, realize high-precision and high-speed automated detection, replace traditional manual detection, and improve production efficiency and product quality.

Description

Technical Field

[0001] This invention relates to the field of industrial automation inspection technology, and in particular to a method, system and equipment for rapid detection of chewing gum defects in industrial settings. Background Technology

[0002] In the chewing gum manufacturing industry, product appearance defect detection and production line smoothness monitoring are crucial for ensuring production quality and efficiency. Traditional detection methods rely on manual visual judgment, which has limited speed and cannot match the conveying speed of modern production lines (180-200 pieces / minute / column). Especially in multi-channel production scenarios with 20 columns, manual inspection cannot achieve real-time coverage across all channels. Furthermore, manual inspection is insufficient for identifying minute defects (such as 0.1mm-level missing pieces or blistering), and is susceptible to fatigue and subjective judgment differences, leading to high rates of missed and false detections, failing to meet the high-precision inspection requirements of over 95%. Manual inspection also struggles to accurately identify multiple defect types simultaneously, such as missing pieces, blistering, and breakage, and lacks standardized qualitative and quantitative assessments of defects. When manually monitoring congestion in 20 channels, significant blind spots and reaction delays exist, easily leading to increased congestion and impacting the overall production line efficiency.

[0003] While existing machine vision inspection solutions have partially replaced manual inspection, they still have shortcomings. Single-channel image processing capabilities are limited, failing to meet the requirement of inspecting at least 30 targets per second across 20 channels; the lack of multimodal data fusion mechanisms results in insufficient detection accuracy and stability; and the absence of dedicated real-time detection algorithms designed for production line congestion makes it difficult to link alarm and shutdown systems for rapid response.

[0004] To address this, the present invention proposes a rapid chewing gum defect detection solution that integrates multimodal data fusion, real-time production line congestion detection, and multi-channel image concurrency optimization, thereby solving the pain points of efficiency, accuracy, and functional coverage in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and equipment for rapid detection of chewing gum defects in industrial settings. This allows for rapid detection of chewing gum across 20 channels, achieving a detection speed of at least 30 targets per second while maintaining a detection accuracy of 0.1mm, covering three types of defects: missing gum, blistering, and breakage. Based on machine vision, it enables real-time detection of production line congestion, accurately identifying single-channel or multi-channel congestion and triggering alarm and shutdown systems. Furthermore, it develops efficient multi-channel image preprocessing algorithms and concurrent optimization techniques to improve image data processing efficiency and ensure the real-time performance and stability of multi-channel detection.

[0006] To achieve the above objectives, this invention provides a rapid detection method for chewing gum defects in industrial settings, comprising the following steps: Step S1: Based on industrial cameras and photoelectric sensors, acquire multimodal data including image data and position signals; Step S2: Preprocess and optimize the acquired multi-channel images for concurrent processing; Step S3: Perform spatiotemporal alignment on the preprocessed multimodal data to achieve multimodal data fusion; Step S4: Based on the fused multimodal data, an optimized YOLOv8 target detection model is used to achieve rapid defect detection; Step S5: Real-time detection of production line congestion based on machine vision.

[0007] Preferably, in step S1, acquiring multimodal data including image data and position signals specifically includes: First, industrial cameras are deployed above the chewing gum conveyor line, with each camera covering 10 chewing gum conveyor channels. A strip LED light source is installed directly below the camera to capture images at a frame rate of 30 frames per second. Then, a photoelectric sensor is installed in each channel to synchronously acquire the chewing gum position signal, forming multimodal data containing image data and position signal.

[0008] Preferably, in step S2, the acquired multi-channel images are preprocessed and multi-channel concurrent optimization is performed, and the specific process is as follows: Step S21: For the multi-channel images input from industrial cameras, design a step-by-step preprocessing pipeline, as follows: The first step is to perform geometric correction using OpenCV to eliminate camera distortion, and to use the checkerboard calibration method to obtain camera intrinsic parameters and correct image pixel offset. The second step involves using a combination of Gaussian filtering and median filtering algorithms to remove noise caused by light source fluctuations. The third step is to use adaptive histogram equalization to enhance contrast and highlight defect features; The fourth step is to extract the region of interest, retaining only the image region corresponding to the gum delivery channel; Step S22: Based on the CPU and GPU of the edge computing terminal, design a CPU-GPU collaborative concurrent architecture, as follows: The CPU is responsible for image segmentation and preprocessing tasks in the chewing gum delivery channel. It uses multi-threading technology to allocate an independent thread to each channel to achieve parallel preprocessing. The GPU is responsible for the inference of the defect detection model. Batch inference optimization is adopted, which combines the preprocessed images of the chewing gum delivery channel into batch data and utilizes the parallel computing power of Tensor Cores to improve inference efficiency. At the same time, image data is managed through memory pool technology to avoid frequent memory allocation and release, reduce data transmission latency, and ensure the real-time performance of multi-channel image processing.

[0009] Preferably, in step S3, the preprocessed multimodal data is spatiotemporally aligned to achieve multimodal data fusion. The specific process is as follows: Step S31, Spatial Alignment: Mapping image coordinates to physical channel coordinates; Let the intrinsic parameter matrix of the industrial camera be... pixel coordinate system With camera coordinate system The mapping relationship satisfies the perspective projection formula, as shown below: ; Among them, the intrinsic parameter matrix for: ; in, This is the camera's intrinsic parameter matrix, a 3×3 matrix; Focal length along the X-axis ; The focal length is in the Y-axis direction. f is the lens focal length; The pixel size along the X-axis; The pixel size along the Y-axis; The coordinates of the camera's principal point are obtained using the checkerboard calibration method; These are coordinates in the pixel coordinate system; These are the three-dimensional coordinates in the camera coordinate system. Determine the camera mounting height; Known physical channel coordinates The transformation relationship with the camera coordinate system is as follows: ; in, These are the coordinates along the conveying direction in the physical channel coordinate system; The coordinates are for the vertical conveying direction; The camera installation height is fixed at 2000mm; By combining the above formulas, the pixel coordinates can be derived. To physical channel coordinates The mapping formula is as follows: ; ; Step S32, Time Alignment: Synchronize sensor signals with image frames; Let the timestamp of the photoelectric sensor trigger signal be . The timestamp of the camera image frame acquisition is The time alignment criteria are as follows: in, This represents the maximum permissible time deviation.

[0010] Preferably, in step S4, the optimized YOLOv8 object detection model includes a feature extraction layer and a multi-scale detection head; the feature extraction layer includes a convolutional layer and a pooling layer; the multi-scale detection head is a dedicated detection branch designed for different features such as missing flesh, blistering, and breakage; The target detection model captures minute defect features at the 0.1mm level on the surface of chewing gum through a feature extraction layer, and combines this with a multi-scale detection head to extract multi-scale defect features, thereby achieving the classification and localization of three types of defects. The specific process is as follows: Step S411: Employ a feature pyramid network structure to fuse feature maps at different scales; let the first... The size of the layer feature map is ,in For height, For width, For the number of channels, the upsampling operation is as follows: in, This is the result after upsampling the feature map of layer l; This is a bilinear interpolation operation; For the first Layer feature map; , For the first The height and width of the layer feature map; Step S412, the feature fusion formula is: in, This is the feature map after fusion at layer l; For the first Original feature map of the layer; For the first Feature map after layer upsampling; This is an element-wise addition operation; Step S413: Design a loss function for a dedicated detection head to address issues such as missing flesh, blistering, and breakage; (1) Missing meat defect: Cross-union loss is used to focus on the overlap between the defect area and the annotation box, as shown below: ; in, The IoU loss function value for the meat shortage defect; For prediction boxes; For annotation boxes; This represents the area of ​​the intersection between the predicted bounding box and the labeled bounding box. This is the area of ​​the union of the predicted bounding box and the labeled bounding box; (2) Bubbling defects: CIoU Loss is used to increase the constraint on the distance between the defect centers, as shown below: ; in, The CIoU loss function value for bubbling defects; This is the intersection-union ratio (IUU) of the predicted bounding boxes and the labeled bounding boxes. It is the Euclidean distance function; The coordinates of the center point of the prediction box; The coordinates of the center point of the annotation box; The length of the diagonal of the smallest rectangle enclosing the prediction box and the label box; These are the weighting coefficients; Aspect ratio consistency parameter; (3) Fracture defects: DIoU Loss is adopted to increase the constraint on the distance of the defect boundary, as shown below: ; in, The DIoU loss function value for fracture defects; (4) The total loss function is the weighted sum of the losses of the three types of defects, as shown below: in, This represents the total loss function value. These are the weighting coefficients; These are the loss function values ​​for defects such as missing flesh, blistering, and breakage, respectively.

[0011] Preferably, in step S4, the object detection model inference speed is accelerated by the GPU based on the Jetson Orin NX Tensor Core, as detailed below: Let the batch size of images processed in a single inference be N, corresponding to the chewing gum delivery channel, and the number of pixels in each image be P. The inference time Tinfer of TensorRT satisfies: ; in, For TensorRT inference time; The image batch size for a single inference operation; The number of pixels in each image; The computational cost of the model; = This represents the INT8 computing power of the Jetson Orin NX; This refers to the utilization rate of computing power.

[0012] Preferably, in step S5, real-time detection of production line congestion is achieved based on machine vision, and the specific process is as follows: Step S51, Congestion Determination Rule Construction: Based on historical congestion data, two types of congestion determination conditions are set: First, static congestion: if the positional offset of the same chewing gum target within a certain channel is less than 0.5mm in three consecutive frames, it is determined to be clogged. The formula for determining static congestion is as follows: Let the physical coordinates of a chewing gum target in the i-th channel be given by the following formula in the t-th frame: The coordinates of the (t+1)th frame are Position offset for: ; in, Let be the position offset of the target in the i-th channel from frame t to frame t+1. Let X be the physical coordinates of the target in the X direction at frame t+1; Let be the physical coordinates of the target in the X direction at frame t; Static congestion is determined when the offset of three consecutive frames is less than a threshold. As shown below: ; in, The first Frame position offset; This is the position offset threshold; Second, dynamic congestion occurs when the number of targets in five consecutive frames within a certain channel exceeds a normal threshold. The formula for determining dynamic congestion is as follows: Let the number of targets in the i-th channel in the t-th frame be... Calculate the threshold for the number of normal targets in a single frame. As shown below: ; in, The threshold for the number of normal targets per frame; For production line speed; 1.2 is the image frame rate; 1.2 is the redundancy coefficient, used to avoid misjudgment due to normal fluctuations. This is for rounding up; The dynamic congestion determination condition is that the number of targets in 5 consecutive frames is greater than 1. As shown below: ; in, Let i be the number of targets in the i-th channel during frames t to t+4. The threshold for the number of normal targets per frame; Step S52, Real-time congestion detection: The image acquired by the industrial camera is divided into independent detection areas corresponding to the number of chewing gum delivery channels according to the channel. A dedicated congestion detection algorithm is deployed in each area. The algorithm counts the number and position changes of targets in each area in real time, and outputs the congestion judgment result within 100ms based on the above judgment rules. Step S53, Alarm and Shutdown Linkage: If congestion is detected, the system immediately triggers an audible and visual alarm and records the congested channel number and time; if it is not resolved within 1 minute, the system sends a signal to the production line shutdown device through the GPIO pin to trigger the corresponding channel to stop, thus preventing the congestion from spreading. Let the time of the first congestion determination be... Alarm trigger time ,in =0s, immediately trigger an alarm; Stop trigger time =60s, if no action is taken within the time limit, the machine will be shut down.

[0013] A rapid detection system for chewing gum defects in industrial settings includes an image acquisition module, a multimodal data fusion module, a defect detection module, a congestion detection module, an alarm and shutdown control module, and a terminal display module. The modules communicate with each other through gigabit Ethernet or GPIO interfaces to collaboratively realize the function of a rapid detection method for chewing gum defects in industrial settings.

[0014] Preferably, the image acquisition module includes an industrial camera, a strip LED light source, and a photoelectric sensor; the industrial camera outputs image data through a GigE interface, and the photoelectric sensor outputs position signals through a signal line; The multimodal data fusion module, deployed on the edge computing terminal, has built-in spatial and temporal alignment algorithms and outputs the image to be detected with physical coordinate markers; The defect detection module loads an optimized YOLOv8 model, supports INT8 precision inference, and outputs defect category, physical coordinates, and confidence level. Congestion detection module: Built-in static / dynamic congestion determination logic, outputs congestion signals by channel, and the congestion determination delay does not exceed 100ms; The alarm and shutdown control module includes an audible and visual alarm light and a PLC. After receiving a defect / congestion signal, it triggers an alarm or outputs a shutdown signal through a relay. The terminal display module, developed based on PyQt5, supports 1920×1200 resolution display and displays the chewing gum delivery channel inspection screen, defect statistics, congestion status, and inspection logs in real time.

[0015] A computer device includes a hardware main body and supporting software. The hardware main body is fixed in an industrial control cabinet next to the production line by screws, and the software is deployed on an edge computing terminal to jointly realize a method for rapid detection of chewing gum defects in industrial scenarios. The main hardware components include: edge computing terminal, image acquisition component, alarm and shutdown component, and terminal display component; The supporting software includes: image preprocessing software, multimodal data fusion software, YOLOv8 defect detection software, congestion detection software, and UI display software. It supports cross-platform operation on Windows and Linux, and the software startup time is no more than 30 seconds.

[0016] Therefore, the present invention employs the above-mentioned rapid detection method, system, and equipment for chewing gum defects in industrial scenarios, and the beneficial effects are as follows: (1) Significantly improved detection efficiency: It achieves a detection speed of 35-40 targets per second in 20 channels, replacing 6-8 manual inspectors, and improving detection efficiency by 5-8 times, meeting the needs of high-capacity production lines.

[0017] (2) The detection accuracy has been greatly improved: the detection accuracy of 0.1mm level defects reaches 96.5%, and the rate of missed detection and false detection is much lower than that of manual detection, ensuring the stability of product quality and reducing the customer complaint rate.

[0018] (3) Comprehensive functionality: It can simultaneously detect three types of defects, such as missing meat, blistering, and breakage, as well as monitor production line congestion without the need to deploy additional detection equipment, thus reducing equipment investment costs.

[0019] (4) Real-time response and linkage: Congestion detection response time ≤100ms, alarm and shutdown linkage is fast, reducing production line downtime and improving the overall operating efficiency of the production line (expected to increase by 15-20%).

[0020] (5) High scalability: It supports multiple cameras and multiple hosts for expansion, and can be adapted to production lines with more channels such as 30 columns and 40 columns. At the same time, it reserves a model iteration interface, which can add more defect category detection functions in the future.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is an overall structural block diagram of the rapid detection system for chewing gum defects of the present invention; Figure 2 This is a top view of the installation layout of the hardware device of the present invention; Figure 3 This is a side view of the installation layout of the hardware device of the present invention; Figure 4This is a schematic diagram of the multi-channel image efficient preprocessing process of the present invention; Figure 5 This is the algorithm logic diagram for real-time detection of production line congestion in this invention; Figure 6 This is a schematic diagram of the terminal display interface of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] Example 1 This invention provides a rapid detection method for chewing gum defects in industrial settings, comprising the following steps: Step S1: Based on industrial cameras and photoelectric sensors, collect multimodal data including image data and position signals.

[0025] First, two industrial cameras (each covering 10 lanes) are deployed above the 20-column gum conveyor line, along with strip LED light sources (installed directly below the cameras), to capture images at a frame rate of 30 frames per second. Then, one photoelectric sensor is installed in each lane to simultaneously acquire gum position signals, forming multimodal data containing both image data and position signals. The hardware installation layout is as follows: Figure 2 and Figure 3 As shown.

[0026] Step S2: Perform efficient preprocessing and multi-channel concurrent optimization on the acquired multi-channel images, such as... Figure 4 As shown.

[0027] Step S21: For the multi-channel images input from two cameras, design a step-by-step preprocessing pipeline as follows: The first step is to perform geometric correction (to eliminate camera distortion) using OpenCV, and to obtain camera intrinsic parameters using the checkerboard calibration method to correct image pixel offset.

[0028] The second step involves using a combination of Gaussian filtering and median filtering algorithms to remove noise caused by light source fluctuations.

[0029] The third step is to use adaptive histogram equalization to enhance contrast and highlight defect features.

[0030] The fourth step is to extract the region of interest (ROI), retaining only the image regions corresponding to the 20 channels to reduce the amount of invalid data (the amount of data is reduced by about 40%).

[0031] Step S22: Based on the multi-core CPU and GPU of the edge computing terminal (Jetson Orin NX), design a CPU-GPU collaborative concurrent architecture, as follows: The CPU is responsible for image segmentation and preprocessing tasks for 20 channels. It uses multi-threading technology to allocate an independent thread for each channel to achieve parallel preprocessing.

[0032] The GPU is responsible for the inference of the defect detection model. Batch inference optimization is adopted, which combines the preprocessed images of 20 channels into batch data and utilizes the parallel computing power of Tensor Cores to improve inference efficiency. At the same time, image data is managed through memory pool technology to avoid frequent memory allocation and release, reduce data transmission latency, and ensure the real-time performance of multi-channel image processing.

[0033] Step S3: Perform spatiotemporal alignment on the preprocessed multimodal data to achieve multimodal data fusion.

[0034] The 2448×2048 resolution image acquired by the industrial camera is spatiotemporally aligned with the position signal of the photoelectric sensor. The sensor trigger signal is then precisely matched with the chewing gum area in the image using a coordinate mapping algorithm to locate the target to be detected, eliminate invalid calculations for areas without targets, and improve detection efficiency.

[0035] Step S31, Spatial Alignment: Mapping image coordinates to physical channel coordinates.

[0036] Let the intrinsic parameter matrix of the industrial camera be... pixel coordinate system With camera coordinate system The mapping relationship satisfies the perspective projection formula, as shown below: ; Among them, the intrinsic parameter matrix for: ; in, This is the camera's intrinsic parameter matrix, a 3×3 matrix; The focal length (in pixels) is the focal length along the X-axis. ; The focal length (in pixels) is the focal length along the Y-axis. f is the lens focal length (unit: mm); The pixel size along the X-axis (unit: mm / pixel); The pixel size along the Y-axis (unit: mm / pixel); The coordinates of the camera principal point (in pixels) are obtained using the checkerboard calibration method; These are coordinates in a pixel coordinate system (unit: pixels); These are the three-dimensional coordinates in the camera coordinate system (unit: mm). Install the camera at a fixed height. =2000mm).

[0037] Known physical channel coordinates (Taking the starting point of the production line as the origin,) Along the conveying direction, The transformation relationship between the vertical transport direction and the camera coordinate system is as follows: ; in, The coordinates (in mm) are along the conveying direction in the physical channel coordinate system. Coordinates for the vertical conveying direction (unit: mm); =2000mm is the camera installation height.

[0038] By combining the above formulas, the pixel coordinates can be derived. To physical channel coordinates The mapping formula is as follows: ; ; in, , Physical channel coordinates (unit: mm); , These are pixel coordinates (unit: pixels); , The coordinates of the camera's principal point (in pixels); =2000mm is the camera mounting height; , Focal length in the X / Y axis direction (unit: pixels).

[0039] This formula determines the physical installation location of the photoelectric sensor. ,i), that is, the i-th channel The coordinates, i=1,2,...,20, are converted into image pixel coordinates to determine the ROI region of each channel in the image, thus achieving spatial alignment.

[0040] Step S32, Time Alignment: Synchronize sensor signals with image frames.

[0041] Let the timestamp of the photoelectric sensor trigger signal be . The timestamp of the camera image frame acquisition is (Each frame interval is approximately 1 / 30 ≈ 33.3 ms). The timing alignment criteria are as follows: ; in, The timestamp for the acquisition of camera image frames (unit: ms); Timestamp of the photoelectric sensor trigger signal (unit: ms); =10ms is the maximum allowable time deviation (calculated based on a production line speed of 180-200 pieces / minute, the movement distance of chewing gum within 10ms is ≤0.3mm, and the positional offset can be ignored). When the above condition is met, the image frame is bound to the sensor signal and determined to be multimodal data of the same chewing gum target, thus achieving time alignment.

[0042] Step S4: Based on the fused multimodal data, an optimized YOLOv8 target detection model is used to achieve rapid defect detection.

[0043] The target detection model captures minute defect features on the surface of chewing gum at the 0.1mm level through a feature extraction layer, and combines it with a multi-scale detection head to classify and locate three types of defects. The feature extraction layer includes a convolutional layer and a pooling layer. The multi-scale detection head is a dedicated detection branch designed for different features such as missing gum, blistering, and breakage.

[0044] Based on Jetson Orin NX's Tensor Core, the object detection model's inference speed is accelerated by the GPU, ensuring the detection requirement of no less than 30 targets per second across 20 channels, with a detection accuracy of over 95%.

[0045] Step S41: For 0.1mm level defects, perform multi-scale defect feature extraction.

[0046] Step S411: Employ the FPN (Feature Pyramid Network) structure to fuse feature maps of different scales. Let the first... The size of the layer feature map is ( For height, For width, (where the number of channels is the number of channels), the upsampling operation is as follows: ; in, This is the result after upsampling the feature map of layer l; This is a bilinear interpolation operation; For the first Layer feature map; , For the first Height and width of the layer feature map (in pixels).

[0047] Step S412, the feature fusion formula is: ; in, This is the feature map after fusion at layer l; For the first Original feature map of the layer; For the first Feature map after layer upsampling; For element-wise addition (requires first performing a 1×1 convolution), The number of channels was adjusted to ).

[0048] Step S413: Design a loss function for a dedicated detection head for defects such as missing flesh (area defects), blistering (protrusion defects), and breakage (contour defects).

[0049] (1) Missing meat defect: The intersection-over-union loss (IoU Loss) is used to focus on the overlap between the defect area and the annotation box, as shown below: ; in, The IoU loss function value for the meat shortage defect; The predicted bounding box. This is the ground truth bounding box. The area of ​​intersection between the predicted bounding box and the labeled bounding box (unit: pixels) 2 ); The area of ​​the union of the predicted bounding box and the labeled bounding box (unit: pixels) 2 ).

[0050] (2) Bubbling defects: CIoU Loss is used to increase the constraint on the distance between the defect centers, as shown below: ; in, The CIoU loss function value for bubbling defects; This is the intersection-union ratio (IUU) of the predicted bounding boxes and the labeled bounding boxes. It is the Euclidean distance function; The coordinates of the center point of the prediction box; The coordinates of the center point of the annotation box; The length of the diagonal of the smallest rectangle that encloses the prediction box and the label box (in pixels). These are the weighting coefficients; This is a parameter for aspect ratio consistency.

[0051] (3) Fracture defects: DIoU Loss is adopted to increase the constraint on the distance of the defect boundary, as shown below: ; in, This represents the DIoU loss function value for the fracture defect.

[0052] (4) The total loss function is the weighted sum of the losses of the three types of defects, as shown below: ; in, This represents the total loss function value. These are the weighting coefficients (determined through sample balancing training). , , These are the loss function values ​​for defects such as missing flesh, blistering, and breakage, respectively.

[0053] Step S42: Accelerate GPU batch inference based on TensorRT.

[0054] Assume the batch size of images processed in a single inference is N=20 (corresponding to 20 channels), the number of pixels in each image is P=2448×2048, and the computational cost of the model is Cmodel=10. 9 FLOPs (after YOLOv8-nano optimization). The inference time of TensorRT, Tinfer, satisfies: ; in, TensorRT inference time (unit: ms); =20 represents the image batch size for a single inference operation; =2448×2048 is the number of pixels in each image; =10 9 FLOPs represent the computational cost of the model; =100TOPS is the INT8 computing power of Jetson Orin NX; =0.8 represents the computing power utilization rate.

[0055] Substituting the values, we get: It meets the speed requirement of "no less than 30 targets per second" (20 targets per frame, 30fps corresponds to 600 targets / second, and the inference time does not exceed 33ms).

[0056] Step S5: Based on machine vision, achieve real-time detection of production line congestion, such as... Figure 5 As shown.

[0057] Step S51, Congestion Judgment Rule Construction: Based on historical congestion data, two types of congestion judgment conditions are set: one is "static congestion", that is, the positional offset of the same chewing gum target in a certain channel is less than 0.5mm in 3 consecutive frames of images (judged as accumulation); the other is "dynamic congestion", that is, the number of targets in 5 consecutive frames of images in a certain channel exceeds the normal threshold (calculated based on the production line speed of 180-200 pieces / minute·column, the normal threshold is 3-4 pieces / frame).

[0058] (1) Static congestion judgment formula (target position offset).

[0059] Let the physical coordinates of a chewing gum target in the i-th channel be given by the following formula in the t-th frame: ( Fixed, for channel i (Coordinates), the coordinates of the (t+1)th frame are Position offset for: ; in, The positional offset of the target in the i-th channel from frame t to frame t+1 (unit: mm). Let X be the physical coordinates of the target in the X direction at frame t+1 (unit: mm). Let X be the physical coordinates of the target in the X direction in frame t (unit: mm).

[0060] Static congestion determination criteria: The offset of three consecutive frames (t, t+1, t+2) is less than the threshold. =0.5mm: ; in, , , These are the position offsets (in mm) for frames t, t+1, and t+2, respectively. =0.5mm is the position offset threshold.

[0061] (2) Dynamic congestion judgment formula (target quantity exceeds limit).

[0062] Let the number of targets in the i-th channel in the t-th frame be... According to the production line speed =180-200 grains / minute = 3-3.33 grains / second, image frame rate =30fps, calculate the threshold for the number of normal targets per frame. As shown below: ; in, The threshold for the number of normal targets per frame; =3-3.33 particles / second is the production line speed; =30fps is the image frame rate; 1.2 is a redundancy coefficient used to avoid misjudgment due to normal fluctuations; This is for rounding up.

[0063] Substituting into the calculation, we get: =2.

[0064] Dynamic congestion determination criteria: The number of targets in 5 consecutive frames is greater than [a certain value]. As shown below: ; in, ~ Let i be the number of targets in the i-th channel during frames t to t+4. =2 is the threshold for the number of normal targets in a single frame.

[0065] Step S52, Real-time congestion detection: The images acquired by the two industrial cameras are divided into 20 independent detection areas according to the channels. A dedicated congestion detection algorithm is deployed in each area. The algorithm counts the number and position changes of targets in each area in real time, and outputs the congestion judgment result within 100ms based on the above judgment rules.

[0066] Step S53, Alarm and Shutdown Linkage: If congestion is detected, the system immediately triggers an audible and visual alarm (alarm light + buzzer) and records the congested channel number and time; if it is not handled within 1 minute, the system sends a signal to the production line shutdown device through the GPIO pin to trigger the corresponding channel to stop, thus preventing the congestion from spreading.

[0067] Let the time of the first congestion determination be... Alarm trigger time (Δt1=0s, alarm triggered immediately); shutdown trigger time (Δ) =60s, if no action is taken within the timeout period, the machine will be shut down, as shown below: ; in, Stop trigger time (unit: seconds); The time of the first congestion determination (in seconds); Δ Δ is the downtime delay. =60s.

[0068] Example 2 like Figure 1 As shown, this invention also proposes a rapid detection system for chewing gum defects in industrial scenarios, including an image acquisition module, a multimodal data fusion module, a defect detection module, a congestion detection module, an alarm and shutdown control module, and a terminal display module. Each module communicates through a gigabit Ethernet or GPIO interface to collaboratively realize the function of a rapid detection method for chewing gum defects in industrial scenarios.

[0069] Image acquisition module: includes 2 Hikvision MV-CS050-10GC industrial cameras (equipped with 8mm focal length lenses), 2 12W strip LED light sources (5000K natural light, IP65 protection), and 20 Omron E3Z-D61 photoelectric sensors; the industrial cameras output image data through the GigE interface, and the photoelectric sensors output position signals through signal lines.

[0070] Multimodal data fusion module: Deployed on the edge computing terminal (Jetson Orin NX), it has built-in spatial alignment (perspective projection formula) and temporal alignment (timestamp matching) algorithms, and outputs the image to be detected with physical coordinate markers.

[0071] Defect detection module: Loads an optimized YOLOv8 model (including a multi-scale detection head and a hybrid loss function), supports INT8 precision inference, and outputs defect categories (missing flesh / bubbling / fracture) and physical coordinates. Confidence level (a score greater than or equal to 0.7 is considered a valid judgment).

[0072] Congestion detection module: Built-in static / dynamic congestion determination logic, outputs congestion signals (active high) by channel, and the congestion determination delay does not exceed 100ms.

[0073] Alarm and shutdown control module: includes one Tianyi AD16-22SM audible and visual alarm light and one Schneider TM221CE24RPLC. After receiving defect / congestion signals, it triggers an alarm or outputs a shutdown signal through a relay.

[0074] Terminal display module: Developed based on PyQt5, supporting 1920×1200 resolution display, real-time display of 20-channel inspection screen (divided into 2 camera areas), defect statistics (updated hourly to show missing material / bubbles / breakage counts), congestion status (congested channels marked in red), and inspection logs (data from the past 30 days can be queried, and filtering is supported by channel / defect type). Figure 6 As shown.

[0075] Example 3 A rapid detection device for chewing gum defects in industrial settings includes a hardware unit and supporting software. The hardware unit is fixed in an industrial control cabinet next to the production line with screws, and the software is deployed on an edge computing terminal to collaboratively achieve defect and congestion detection.

[0076] The main hardware components include: edge computing terminal, image acquisition component, alarm and shutdown component, and terminal display component.

[0077] ① Edge computing terminal: 2 NVIDIA Jetson Orin NX 16G (100TOPS computing power, 16GB LPDDR5 memory, supports 4-channel camera input), running Ubuntu 18.04 LTS + Jetson SDK Manager 1.10.0.

[0078] The edge computing terminal runs on the Ubuntu 18 operating system and Jetson SDK Manager, supports four-channel camera input, connects to cameras via Gigabit Ethernet, and connects to sensors, alarm and shutdown components via GPIO pins. The edge computing terminal supports iterative model updates; newly trained ONNX format models can be imported via USB, and the software automatically completes the model conversion (ONNX → TensorRT engine) and deployment. The update process does not require a system restart, and the update time is ≤5 minutes.

[0079] ② Image acquisition components: 2 Hikvision MV-CS050-10GC industrial cameras (operating temperature -30℃~60℃), 2 800mm long strip LED light sources, 20 photoelectric sensors, camera installation height 2m, light source installation height 1.5m.

[0080] ③ Alarm and shutdown components: 1 audible and visual alarm light (24V DC power supply), 1 PLC (16 inputs and 8 outputs, supporting Modbus-RTU protocol), which is linked with the edge computing terminal through the GPIO interface.

[0081] ④ Terminal display components: One 10.1-inch Advantech TPC-1051T touchscreen monitor (capacitive touch, 350cd / ㎡ brightness), connected to the edge computing terminal via HDMI interface.

[0082] The supporting software includes: image preprocessing software (developed with OpenCV 4.10.0), multimodal data fusion software (developed with Python 3.9), YOLOv8 defect detection software (developed with PyTorch 2.0.1 + TensorRT 8.5.2), congestion detection software (developed with C++ and optimized for real-time performance), and UI display software (developed with PyQt5). It supports cross-platform operation on Windows / Linux, and the software startup time is ≤30s.

[0083] Example 4 (a) Hardware deployment.

[0084] 1. Camera and light source installation: Fix two industrial cameras 300mm directly above the chewing gum production line, each camera covering 10 columns of channels (the field of view of a single camera covers the width of 10 columns of channels, approximately 160mm); install a strip LED light source 250mm below each camera, with the light source parallel to the channel to ensure that the light shines evenly on the chewing gum surface without shadow interference.

[0085] 2. Sensor installation: Install one photoelectric sensor at the entrance of each channel (5cm upstream of the camera detection area), with the sensor probe facing the channel surface, to ensure that a signal is triggered when chewing gum passes by (detection distance 3-5cm, response time ≤1ms).

[0086] 3. Edge computing terminal deployment: Install the Jetson Orin NX edge computing terminal in the industrial control cabinet next to the production line. Connect two industrial cameras through a gigabit Ethernet switch and connect 20 photoelectric sensors, audible and visual alarm lights, and a shutdown control module through GPIO pins. Install a touch screen display in the industrial control cabinet and connect it to the terminal through an HDMI interface.

[0087] (ii) Software configuration and operation.

[0088] 1. Software Installation: Install the Ubuntu 18 operating system and Jetson SDK Manager on the edge computing terminal, configure the Python 3.9 environment, and install dependent libraries such as OpenCV (version 4.10.0), PyTorch (adapted to the Jetson platform version), and PyQt5; deploy image preprocessing software, multimodal data fusion software, YOLOv8 defect detection software, congestion detection software, and UI display software.

[0089] 2. Model Training and Deployment: Collect 10,000 chewing gum sample images (including three types of defects: missing pieces, bubbles, and breaks, with no less than 3,000 samples for each type of defect, and defect sizes covering 0.1-1mm), and perform data annotation (annotating bounding boxes and defect categories); train the YOLOv8 model using a 4×Tesla A100 GPU server, and use data augmentation (rotation, flipping, brightness adjustment) during training to improve the model's generalization ability; after training, export the model in ONNX format, deploy it to an edge computing terminal, and accelerate inference using TensorRT.

[0090] 3. System Operation: After the system starts, the image acquisition module acquires images at a frame rate of 30fps, and the photoelectric sensor outputs position signals synchronously; the multimodal data fusion module performs spatiotemporal alignment on the data and outputs the image of the target to be detected; the multi-channel image preprocessing module processes the images of 20 channels in parallel to remove noise and invalid areas; the defect detection module performs batch inference on the preprocessed images and outputs the defect results; the congestion detection module monitors the changes in the number and position of targets in each channel in real time to determine whether there is congestion; the terminal display module displays the detection screen, the number of defects, and the congestion status in real time. If a defect or congestion is detected, the alarm and shutdown control module triggers the corresponding action (alarm / shutdown).

[0091] (III) Performance verification.

[0092] 1. Detection speed: With 20 channels, the system can detect 35-40 targets per second, meeting the requirement of "no less than 30 targets per second"; the single-channel detection delay is ≤100ms, which meets the speed requirement of 180-200 particles / minute / column in the production line.

[0093] 2. Detection accuracy: The detection accuracy for defects such as missing flesh, blistering, and breakage at the 0.1mm level reaches 96.5%, with a false negative rate of ≤2% and a false positive rate of ≤1.5%, exceeding the accuracy target of 95%.

[0094] 3. Congestion detection: The congestion judgment response time is ≤100ms. If no action is taken within 1 minute, the shutdown signal trigger delay is ≤50ms, which can effectively prevent the spread of congestion.

[0095] Therefore, this invention employs the aforementioned rapid detection method, system, and equipment for chewing gum defects in industrial scenarios, achieving rapid detection of chewing gum across 20 channels, reaching a detection speed of no less than 30 targets per second, while ensuring a detection accuracy of 0.1mm, covering three types of defects: missing gum, blistering, and breakage; it also achieves real-time detection of production line congestion based on machine vision, accurately identifying single-channel or multi-channel congestion, and linking alarm and shutdown systems; and it develops efficient multi-channel image preprocessing algorithms and concurrent optimization techniques to improve image data processing efficiency and ensure the real-time performance and stability of multi-channel detection.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A rapid detection method for chewing gum defects in industrial settings, characterized in that, Includes the following steps: Step S1: Based on industrial cameras and photoelectric sensors, acquire multimodal data including image data and position signals; Step S2: Preprocess and optimize the acquired multi-channel images for concurrent processing; Step S3: Perform spatiotemporal alignment on the preprocessed multimodal data to achieve multimodal data fusion; Step S4: Based on the fused multimodal data, an optimized YOLOv8 target detection model is used to achieve rapid defect detection; Step S5: Real-time detection of production line congestion based on machine vision.

2. The rapid detection method for chewing gum defects in industrial settings according to claim 1, characterized in that, In step S1, multimodal data including image data and position signals is acquired, specifically including: First, industrial cameras are deployed above the chewing gum conveyor line, with each camera covering 10 chewing gum conveyor channels. A strip LED light source is installed directly below the camera to capture images at a frame rate of 30 frames per second. Then, a photoelectric sensor is installed in each channel to synchronously acquire the chewing gum position signal, forming multimodal data containing image data and position signal.

3. The rapid detection method for chewing gum defects in industrial settings according to claim 2, characterized in that, In step S2, the acquired multi-channel images are preprocessed and multi-channel concurrent optimization is performed. The specific process is as follows: Step S21: For the multi-channel images input from industrial cameras, design a step-by-step preprocessing pipeline, as follows: The first step is to perform geometric correction using OpenCV to eliminate camera distortion, and to use the checkerboard calibration method to obtain camera intrinsic parameters and correct image pixel offset. The second step involves using a combination of Gaussian filtering and median filtering algorithms to remove noise caused by light source fluctuations. The third step is to use adaptive histogram equalization to enhance contrast and highlight defect features; The fourth step is to extract the region of interest, retaining only the image region corresponding to the gum delivery channel; Step S22: Based on the CPU and GPU of the edge computing terminal, design a CPU-GPU collaborative concurrent architecture, as follows: The CPU is responsible for image segmentation and preprocessing tasks in the chewing gum delivery channel. It uses multi-threading technology to allocate an independent thread to each channel to achieve parallel preprocessing. The GPU is responsible for the inference of the defect detection model. Batch inference optimization is adopted, which combines the preprocessed images of the chewing gum delivery channel into batch data and utilizes the parallel computing power of Tensor Cores to improve inference efficiency. At the same time, image data is managed through memory pool technology to avoid frequent memory allocation and release, reduce data transmission latency, and ensure the real-time performance of multi-channel image processing.

4. The rapid detection method for chewing gum defects in industrial settings according to claim 3, characterized in that, In step S3, the preprocessed multimodal data is spatiotemporally aligned to achieve multimodal data fusion. The specific process is as follows: Step S31, Spatial Alignment: Mapping image coordinates to physical channel coordinates; Let the intrinsic parameter matrix of the industrial camera be... pixel coordinate system With camera coordinate system The mapping relationship satisfies the perspective projection formula, as shown below: ； Among them, the intrinsic parameter matrix for: ； in, This is the camera's intrinsic parameter matrix, a 3×3 matrix; The focal length is in the X-axis direction. ; The focal length is in the Y-axis direction. f is the lens focal length; The pixel size along the X-axis; The pixel size along the Y-axis; The coordinates of the camera's principal point are obtained using the checkerboard calibration method; These are coordinates in the pixel coordinate system; These are the three-dimensional coordinates in the camera coordinate system. Determine the camera mounting height; Known physical channel coordinates The transformation relationship with the camera coordinate system is as follows: ； in, These are the coordinates along the conveying direction in the physical channel coordinate system; The coordinates are for the vertical conveying direction; The camera installation height is fixed at 2000mm; By combining the above formulas, the pixel coordinates can be derived. To physical channel coordinates The mapping formula is as follows: ； ； Step S32, Time Alignment: Synchronize sensor signals with image frames; Let the timestamp of the photoelectric sensor trigger signal be . The timestamp of the camera image frame acquisition is The time alignment criteria are as follows: ； in, This represents the maximum permissible time deviation.

5. The rapid detection method for chewing gum defects in industrial settings according to claim 4, characterized in that, In step S4, the optimized YOLOv8 object detection model includes a feature extraction layer and a multi-scale detection head; the feature extraction layer includes a convolutional layer and a pooling layer; the multi-scale detection head is a dedicated detection branch designed for different features such as missing flesh, blistering, and breakage; The target detection model captures minute defect features at the 0.1mm level on the surface of chewing gum through a feature extraction layer, and combines this with a multi-scale detection head to extract multi-scale defect features, thereby achieving the classification and localization of three types of defects. The specific process is as follows: Step S411: Employ a feature pyramid network structure to fuse feature maps at different scales; let the first... The size of the layer feature map is ,in For height, For width, For the number of channels, the upsampling operation is as follows: ； in, This is the result after upsampling the feature map of layer l; This is a bilinear interpolation operation; For the first Layer feature map; , For the first The height and width of the layer feature map; Step S412, the feature fusion formula is: ； in, This is the feature map after fusion at layer l; For the first Original feature map of the layer; For the first Feature map after layer upsampling; This is an element-wise addition operation; Step S413: Design a loss function for a dedicated detection head to address issues such as missing flesh, blistering, and breakage; (1) Missing meat defect: Cross-union loss is used to focus on the overlap between the defect area and the annotation box, as shown below: ； in, The IoU loss function value for the meat shortage defect; For prediction boxes; For annotation boxes; This represents the area of ​​the intersection between the predicted bounding box and the labeled bounding box. This is the area of ​​the union of the predicted bounding box and the labeled bounding box; (2) Bubbling defects: CIoU Loss is used to increase the constraint on the distance between the defect centers, as shown below: ； in, The CIoU loss function value for bubbling defects; This is the intersection-union ratio (IUU) of the predicted bounding boxes and the labeled bounding boxes. It is the Euclidean distance function; The coordinates of the center point of the prediction box; The coordinates of the center point of the annotation box; The length of the diagonal of the smallest rectangle enclosing the prediction box and the label box; These are the weighting coefficients; Aspect ratio consistency parameter; (3) Fracture defects: DIoU Loss is adopted to increase the constraint on the distance of the defect boundary, as shown below: ； in, The DIoU loss function value for fracture defects; (4) The total loss function is the weighted sum of the losses of the three types of defects, as shown below: ； in, This represents the total loss function value. These are the weighting coefficients; , , These are the loss function values ​​for defects such as missing flesh, blistering, and breakage, respectively.

6. The rapid detection method for chewing gum defects in industrial settings according to claim 5, characterized in that, In step S4, based on the Jetson Orin NX Tensor Core, the inference speed of the object detection model is accelerated by the GPU. The specific process is as follows: Let the batch size of images processed in a single inference be N, corresponding to the chewing gum delivery channel, and the number of pixels in each image be P. The inference time Tinfer of TensorRT satisfies: ； in, For TensorRT inference time; The image batch size for a single inference operation; The number of pixels in each image; The computational cost of the model; = This represents the INT8 computing power of the Jetson Orin NX; This refers to the utilization rate of computing power.

7. The rapid detection method for chewing gum defects in industrial settings according to claim 6, characterized in that, In step S5, real-time detection of production line congestion is achieved based on machine vision. The specific process is as follows: Step S51, Congestion Determination Rule Construction: Based on historical congestion data, two types of congestion determination conditions are set: First, static congestion: if the positional offset of the same chewing gum target within a certain channel is less than 0.5mm in three consecutive frames, it is determined to be clogged. The formula for determining static congestion is as follows: Let the physical coordinates of a chewing gum target in the i-th channel be given by the following formula in the t-th frame: The coordinates of the (t+1)th frame are ; Position offset for: ； in, Let be the position offset of the target in the i-th channel from frame t to frame t+1. Let X be the physical coordinates of the target in the X direction at frame t+1; Let be the physical coordinates of the target in the X direction at frame t; Static congestion is determined when the offset of three consecutive frames is less than a threshold. As shown below: ； in, , , These are the position offsets for frames t, t+1, and t+2, respectively. This is the position offset threshold; Second, dynamic congestion occurs when the number of targets in five consecutive frames within a certain channel exceeds a normal threshold. The formula for determining dynamic congestion is as follows: Let the number of targets in the i-th channel in the t-th frame be... Calculate the threshold for the number of normal targets in a single frame. As shown below: ； in, The threshold for the number of normal targets per frame; For production line speed; 1.2 is the image frame rate; 1.2 is the redundancy coefficient, used to avoid misjudgment due to normal fluctuations. This is for rounding up; The dynamic congestion determination condition is that the number of targets in 5 consecutive frames is greater than 1. As shown below: ； in, ~ Let i be the number of targets in the i-th channel during frames t to t+4. The threshold for the number of normal targets per frame; Step S52, Real-time congestion detection: The image acquired by the industrial camera is divided into independent detection areas corresponding to the number of chewing gum delivery channels according to the channel. A dedicated congestion detection algorithm is deployed in each area. The algorithm counts the number and position changes of targets in each area in real time, and outputs the congestion judgment result within 100ms based on the above judgment rules. Step S53, Alarm and Shutdown Linkage: If congestion is detected, the system immediately triggers an audible and visual alarm and records the congested channel number and time; if it is not resolved within 1 minute, the system sends a signal to the production line shutdown device through the GPIO pin to trigger the corresponding channel to stop, thus preventing the congestion from spreading. Let the time of the first congestion determination be... Alarm trigger time ,in =0s, immediately trigger an alarm; Stop trigger time ,in =60s, if no action is taken within the time limit, the machine will be shut down.

8. A rapid detection system for chewing gum defects in industrial settings, characterized in that, It includes an image acquisition module, a multimodal data fusion module, a defect detection module, a congestion detection module, an alarm and shutdown control module, and a terminal display module. Each module communicates through a gigabit Ethernet or GPIO interface to collaboratively realize the function of the rapid detection method for chewing gum defects in industrial scenarios as described in any one of claims 1-7.

9. The rapid detection system for chewing gum defects in industrial settings according to claim 8, characterized in that, The image acquisition module includes an industrial camera, a strip LED light source, and a photoelectric sensor; the industrial camera outputs image data through a GigE interface, and the photoelectric sensor outputs position signals through a signal line; The multimodal data fusion module, deployed on the edge computing terminal, has built-in spatial and temporal alignment algorithms and outputs the image to be detected with physical coordinate markers; The defect detection module loads an optimized YOLOv8 model, supports INT8 precision inference, and outputs defect category, physical coordinates, and confidence level. Congestion detection module: Built-in static / dynamic congestion determination logic, outputs congestion signals by channel, and the congestion determination delay does not exceed 100ms; The alarm and shutdown control module includes an audible and visual alarm light and a PLC. After receiving a defect / congestion signal, it triggers an alarm or outputs a shutdown signal through a relay. The terminal display module, developed based on PyQt5, supports 1920×1200 resolution display and displays the chewing gum delivery channel inspection screen, defect statistics, congestion status, and inspection logs in real time.

10. A computer device, characterized in that, The invention includes a hardware unit and supporting software. The hardware unit is fixed to an industrial control cabinet next to the production line with screws, and the software is deployed on an edge computing terminal to jointly realize the rapid detection method for chewing gum defects in an industrial setting as described in any one of claims 1-7. The main hardware components include: edge computing terminal, image acquisition component, alarm and shutdown component, and terminal display component; The supporting software includes: image preprocessing software, multimodal data fusion software, YOLOv8 defect detection software, congestion detection software, and UI display software. It supports cross-platform operation on Windows and Linux, and the software startup time is no more than 30 seconds.

Images