Material blockage detection method and system based on multi-modal information fusion

Abstract

The invention discloses a material blockage detection method and system based on multi-modal information fusion. The method comprises the following steps: synchronously acquiring a depth image and a color image through a binocular depth camera; positioning a calibrated key monitoring area on the depth image, calculating a relative difference value between the depth value of each pixel point in the key monitoring area and the thickness reference surface, and calculating an average thickness value based on each relative difference value; when a plurality of average thickness values calculated for the plurality of continuous depth images meet a continuous thickness anomaly condition, generating a thickness anomaly signal; intercepting a target area image corresponding to the key monitoring area in the depth image from the color image, inputting the target area image into a pre-trained color image classification model, and when an output result meets a visual overflow condition, generating a visual overflow signal; and a material blockage detection result is generated according to the detection states of the thickness abnormal signal and the visual overflow signal, so that the timeliness, the accuracy and the environmental adaptability of material blockage detection are improved.

Description

Technical Field

[0001] This invention relates to the fields of automated cigarette production and material transportation technology, and in particular to a material blockage detection method and system based on multimodal information fusion. Background Technology

[0002] In continuous process industrial production such as tobacco, grain, chemical, and mining, materials need to be transported through equipment such as belt conveyors, chutes, or screw feeders. Material blockage is one of the most common and extremely dangerous malfunctions in such production. Therefore, detecting material blockage is crucial to ensuring safe production.

[0003] For a long time, material blockage detection has mainly relied on mechanical contact detection, represented by rotary paddle level switches and baffle switches, which rely on the physical contact between the material and the mechanical device when it accumulates to a certain height to trigger a signal; or non-contact electrical detection, such as radio frequency admittance switches and capacitive level gauges, which determine the material level by detecting changes in the dielectric constant or capacitance value of the material; or detection based on monocular vision, which uses industrial cameras to collect images and uses image processing algorithms to identify the material edges to infer whether accumulation or overflow has occurred.

[0004] However, mechanical contact detection often results in material blockage by the time an alarm is triggered, leading to a significant delay in response and an inability to provide early warnings. Furthermore, mechanical components are prone to jamming and damage under conditions of moisture, wear, or sticky materials, reducing the timeliness and environmental adaptability of material blockage detection. Non-contact electrical detection is susceptible to changes in material composition, humidity, and adhesion, resulting in a high false alarm rate. Its fixed installation location also prevents comprehensive monitoring of high-risk areas, and its insensitivity to blockages at non-fixed points reduces the accuracy of material blockage detection. Monocular vision-based detection is extremely sensitive to ambient light; changes in light intensity, shadows, steam, or dust degrade image quality, causing false alarms or missed alarms. Moreover, it only alarms when material is clearly overflowing or accumulating, resulting in delayed warnings and further reducing the timeliness of material blockage detection. Summary of the Invention

[0005] This invention provides a material blockage detection method and system based on multimodal information fusion to solve the problems of low timeliness, low accuracy and low environmental adaptability in material blockage detection.

[0006] According to one aspect of the present invention, a method for detecting material blockage is provided, comprising:

[0007] Simultaneous acquisition of depth and color images using a binocular depth camera;

[0008] Locate and calibrate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0009] When multiple average thickness values ​​calculated from multiple consecutive depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated.

[0010] The target region image corresponding to the key monitoring area in the depth image is extracted from the color image, and the target region image is input into the pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated.

[0011] Based on the detection status of thickness anomaly signals and visual overflow signals, material blockage detection results are generated.

[0012] According to another aspect of the present invention, a material blockage detection device is provided, comprising:

[0013] The synchronous acquisition module is used to simultaneously acquire depth images and color images using a binocular depth camera.

[0014] The positioning and calibration module is used to locate and calibrate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0015] The thickness anomaly module is used to generate a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly.

[0016] The visual overflow module is used to extract the target region image from the color image and the key monitoring area in the depth image, and input the target region image into the pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated.

[0017] The detection results module is used to generate material blockage detection results based on the detection status of thickness anomaly signals and visual overflow signals.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the material blockage detection method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the material blockage detection method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0022] The technical solution of this invention involves simultaneously acquiring depth and color images using a binocular depth camera; locating and calibrating at least one key monitoring area on the depth image; calculating the relative difference between the depth value of each pixel within the key monitoring area and the thickness reference plane; and calculating the arithmetic mean of the thickness of all pixels within the key monitoring area based on each relative difference to obtain an average thickness value; generating a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly; extracting a target area image corresponding to the key monitoring area in the depth image from the color image and inputting the target area image into a pre-trained color image classification model; generating a visual overflow signal when the output result meets the condition of visual overflow; and generating a material blockage detection result based on the detection status of the thickness anomaly signal and the visual overflow signal. By simultaneously acquiring depth and color images using a binocular depth camera, the system can capture the three-dimensional accumulation state of materials based on the depth images. After locating key monitoring areas, by calculating the relative difference between pixels and the thickness reference plane, as well as the average thickness value of the area, a thickness anomaly signal can be generated based on the continuous anomaly condition of the average thickness value of multiple consecutive depth images when subtle anomalies occur in the material thickness. This eliminates the need to wait for material to visibly overflow, advancing the warning time and allowing sufficient time to handle blockages, reducing losses such as production line downtime and equipment damage. Furthermore, the depth information is less affected by differences in lighting and material color, resulting in high anomaly recognition accuracy. The color image can be used to extract the target area corresponding to the key monitoring area and input into a pre-trained classification model to generate a visual overflow signal. This not only helps verify the authenticity of the depth detection results to reduce false alarms but also serves as a redundant detection basis to avoid missed alarms when the depth channel fails. It adapts to complex interferences such as changes in lighting and dust in industrial scenarios, enhancing the system's robustness. Finally, the detection result is formed based on the detection status of the two signals, meeting the early warning needs of high-risk points while also adapting to efficient detection of conventional points, improving the timeliness, accuracy, and environmental adaptability of material blockage detection.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a material blockage detection method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of another material blockage detection method provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a flowchart of another material blockage detection method provided in Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a material blockage detection device according to Embodiment 4 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the material blockage detection method of this invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a material blockage detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to detecting whether material blockage has occurred. The method can be executed by a material blockage detection device, which can be implemented in hardware and / or software and is generally configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110: Simultaneously acquires depth and color images using a binocular depth camera.

[0035] In this embodiment of the invention, the binocular depth camera can be specifically understood as: a camera that mimics the parallax of human eyes, an imaging device with a dual-camera module, which can simultaneously capture images through two lenses with a certain distance, calculate the parallax of the left and right cameras to obtain the distance information of each pixel in the scene, generate a depth image (reflecting the three-dimensional spatial position of the object), and simultaneously output a color image (reflecting the color and texture information of the object), used to simultaneously acquire the two-dimensional visual features and three-dimensional depth information of the scene.

[0036] Synchronous acquisition can be understood as the acquisition of depth and color images maintaining consistency in both time and space, ensuring that scene information from the same viewpoint at the same moment can be matched. A depth image can be understood as an image that represents the distance or depth of each point in the scene to the camera using pixel values, rather than color information, directly reflecting the three-dimensional stacking state of objects (such as material thickness). A color image can be understood as an image that presents the color and texture characteristics of the scene through red, green, and blue pixel values, i.e., an RGB (Red, Green, Blue) image, used to assist in determining the shape of materials (such as whether they have overflowed).

[0037] Specifically, after the system starts running, the dual-modal acquisition function of the binocular depth camera is activated. During operation, the camera simultaneously acquires depth images reflecting the distance from each pixel in the scene to the camera, as well as color images showing the scene's colors and textures, at a fixed frame rate (e.g., 15-30 frames per second, which can be adjusted according to the material flow speed; a higher frame rate is selected for faster flow to ensure continuous capture). During acquisition, the camera hardware's timing control and calibration mechanisms (such as synchronous trigger signals and pixel coordinate mapping) ensure that the depth image and color image are generated at the same time and that each pixel corresponds to the same physical location. This avoids image information mismatch due to acquisition time differences or viewing angle deviations, achieving spatiotemporal alignment of the two images.

[0038] Optionally, based on the above embodiments, the binocular depth camera is installed at a process point prone to blockage on the conveying equipment, and the shooting angle and focal length of the binocular depth camera meet the condition of complete field of view coverage of the key monitoring area.

[0039] Accordingly, based on the above embodiments, before simultaneously acquiring depth images and color images using a binocular depth camera, the following may also be included:

[0040] While the conveying equipment is running unloaded, acquire initial depth images and mark at least one key monitoring area;

[0041] Calculate the average depth value of all pixels within the critical monitoring area, and use it as the thickness reference plane.

[0042] In this embodiment of the invention, the easily clogged process point can be specifically understood as: the critical location in industrial conveying equipment where materials are prone to accumulate and clog, such as downstream of the conveyor belt discharge port and at the corner of the chute. The critical monitoring area can be specifically understood as: the area covered by the easily clogged process point in the depth image, such as the projection of the easily clogged trough or channel cross-section onto the image. The thickness reference plane can be specifically understood as: the average depth value of all pixels in the critical monitoring area under no-load conditions, used as the zero-point reference for subsequent material thickness calculations, such as the empty trough depth.

[0043] Specifically, the binocular depth camera is installed near the easily clogged process points of the conveyor equipment (e.g., vertically above), and its shooting angle and focal length are adjusted to ensure complete coverage of the area to be monitored. When the conveyor equipment is running unloaded, an initial depth image is acquired, and at least one key monitoring area covering the easily clogged process point is delineated within this image. The average depth value of all pixels within this area is calculated, and this value is set as the thickness reference plane (i.e., the zero-point reference for subsequent calculations of the material thickness in this key monitoring area). After system calibration and initialization are completed, depth and color images can be simultaneously acquired using the binocular depth camera for subsequent detection.

[0044] In addition, after setting the thickness reference plane, a pre-trained color image classification model can be loaded into the edge computing server or industrial control computer for subsequent classification and detection of the acquired color images.

[0045] By installing a binocular depth camera at a process point on the conveying equipment that is prone to clogging and adjusting it to fully cover the key monitoring area, and calibrating the area and calculating the thickness reference plane when the equipment is unloaded, the camera can be accurately focused on the area where materials are prone to accumulate. The material thickness change calculated based on the thickness reference plane can directly reflect the degree of three-dimensional accumulation, avoid background interference, improve the early detection accuracy of blockages, and enhance the robustness of the system in complex industrial environments.

[0046] Furthermore, based on the above embodiments, after simultaneously acquiring depth images and color images using a binocular depth camera, the method may further include:

[0047] Perform nonlocal mean filtering on the depth image.

[0048] In this embodiment of the invention, nonlocal mean filtering can be specifically understood as: a denoising algorithm based on image patch similarity weighting. By calculating the similarity between the target pixel and the image patch containing all other pixels in the image, the algorithm assigns higher weights to image patches with high similarity and performs weighted summation to achieve noise suppression. Compared with traditional filtering (such as Gaussian filtering), it can better preserve image details (such as material accumulation edges).

[0049] Specifically, after simultaneously acquiring depth and color images using a binocular depth camera, non-local mean filtering is applied to the depth images to suppress shot noise and time-of-flight noise, thus avoiding deviations in material thickness calculations caused by noise. Shot noise can be understood as noise generated during depth camera acquisition due to random fluctuations in photon counts, manifesting as randomly distributed bright or dark spots in the depth image, causing local pixel depth values ​​to deviate from the true value. Time-of-flight noise can be understood as noise unique to cameras based on the time-of-flight principle, caused by errors in the measurement of the round-trip time of the light signal, resulting in slight fluctuations in depth values, especially noticeable in low-light or highly reflective scenes.

[0050] Nonlocal mean filtering of depth images can specifically include: determining filtering parameters (such as image patch size and search window range). Specifically, the image patch can be set to 7×7 pixels (this can be adjusted according to business needs to balance detail preservation and computational efficiency), and the search window range can be set to 21×21 pixels (to cover a sufficiently large area to find highly similar image patches). Iterate through each pixel v in the depth image, find all pixels w within the 21×21 search window, calculate the Euclidean distance (measuring similarity) between the 7×7 image patches centered on v and w, determine the weight W(v,w) based on the similarity (higher similarity, greater weight), and then apply the formula... Calculate the filtered value of pixel v. Finally, output the filtered depth image. Here, w is the reference pixel within the search window, v is the target pixel (the pixel currently being filtered), I is the search window region, u(w) is the original pixel value of the reference pixel w (the original depth value of w), W(v,w) is the similarity weight between v and w, and NL-Means(v) is the final output value of pixel v after Non-LocalMeans (NL-Means) filtering.

[0051] Understandably, considering that cigarette material particles or filaments are often quite fine, the details of the material accumulation edges in the depth image need to be accurately preserved. If the image block is too large, the edges will be blurred; if it is too small, it will be difficult to effectively capture similarity features. Therefore, the image block size can be determined according to the size specifications of the cigarette material.

[0052] In addition, the weight w(v,w) can be calculated using a Gaussian function: w(v,w)=exp(-d² / σ²) (exp represents the exponential function, d is the Euclidean distance, σ is the weight attenuation coefficient, and σ can be determined iteratively based on the material characteristics of the cigarette material (such as depth fluctuation range, material humidity and feed amount), noise type (such as dust or light interference) and camera parameters (such as resolution and frame rate), to avoid noise residue caused by σ being too small and detail blurring caused by σ being too large.

[0053] By applying nonlocal mean filtering to the depth image, noise in industrial scenarios (such as particle noise caused by dust in cigarette factories and time-of-flight measurement errors) can be suppressed. By weighting high-similarity pixels and reducing the weight of low-similarity noise pixels within the search window, the range of depth value fluctuations can be narrowed, thickness calculation deviations can be reduced, and thickness misjudgments caused by noise can be avoided. This ensures that the system can capture early signs of blockage and enhances its adaptability to complex industrial environments.

[0054] S120. Locate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0055] Specifically, based on the pre-calibrated coordinates of the key monitoring area, the area is located in the depth image. Each pixel within the area is traversed, and the relative difference between the depth value of each pixel and the thickness reference plane is calculated. If the pixel depth value is less than the reference plane, the difference is positive (indicating the presence of material at that point, and the difference represents the thickness). If the depth value is greater than the reference plane (due to noise), it can be considered as having a thickness of 0, thus avoiding the influence of outliers. The sum of all valid thickness values ​​(positive differences) within the area is calculated and then divided by the total number of pixels in the area to obtain the average thickness value of the key monitoring area in the current frame, i.e., the overall thickness index of the area.

[0056] In a specific example, taking the detection of material blockage on a conveyor belt in a cigarette factory as an example, on the depth image, based on the coordinates of the calibrated key monitoring area (such as the 800×400 pixel range corresponding to the belt trough, with coordinates from (200, 150) to (1000, 550)), the area is located using an image coordinate mapping algorithm to ensure that only pixels within the trough are analyzed, avoiding interference from background pixels such as belt edges and supports. The pixels within the key monitoring area are traversed, and thickness calculation is performed for each pixel. The effective thickness values ​​(positive values) of all pixels within the key monitoring area are accumulated to obtain the total thickness value. Then, the total thickness value is divided by the total number of pixels within the key monitoring area (320,000) to calculate the average thickness value of the current frame.

[0057] S130. When multiple average thickness values ​​calculated for multiple continuous depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated.

[0058] In this embodiment of the invention, multiple continuous depth images can be understood as: multiple consecutive depth images acquired by a binocular depth camera in time, reflecting the dynamic changes of material accumulation within a certain time period, rather than single frame static data.

[0059] The continuous abnormal thickness condition can be understood as: the rules for judging whether the material thickness is in an abnormal state for multiple consecutive frames. Specifically, it can include: the average thickness threshold (set based on the equipment's safe conveying capacity, such as 70%-90% of the tank depth) and the number of consecutive frames (set based on the camera frame rate to ensure the time continuity of abnormal judgment and avoid misjudgment due to single frame noise).

[0060] The thickness anomaly signal can be specifically understood as: when the conditions for continuous thickness anomaly are met, the system generates a high-priority signal, which indicates that the material accumulation in the key monitoring area has become continuously abnormal and there is a risk of blockage.

[0061] Specifically, the system receives continuous depth images acquired by a binocular depth camera in chronological order. For each frame, it sequentially performs the process of locating the key monitoring area and calculating the average thickness value to obtain continuous average thickness values. The system monitors these continuous average thickness values ​​in real time to determine whether they meet the conditions for continuous thickness anomalies. For example, the average thickness values ​​of continuous depth images are stored in a queue, and a sliding window (the window size equals the number of consecutive frames) is used to count and determine whether the anomaly conditions are met in real time. The sliding window stores the average thickness value of the most recent image frame, and removes the oldest frame for each new frame. When the average thickness values ​​of all frames in the sliding window meet the conditions for continuous thickness anomalies, it is determined that the material thickness in the key monitoring area is in a continuous abnormal state, indicating that the material accumulation has exceeded the normal range and has not been alleviated, posing a risk of blockage, and a thickness anomaly signal is generated. If the conditions are not met, it is determined to be a temporary fluctuation, no anomaly signal is generated to avoid false alarms, and the image frames in the sliding window are updated to continue monitoring and judgment.

[0062] Optionally, based on the above embodiments, when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly, generating a thickness anomaly signal may include:

[0063] Obtain the average thickness value of the key monitoring area corresponding to multiple depth images acquired within a preset time window;

[0064] When multiple average thickness values ​​are greater than a preset thickness threshold, the condition for continuous thickness anomaly is determined to be met, and a thickness anomaly signal is generated.

[0065] In this embodiment of the invention, the preset time window can be specifically understood as a time interval (e.g., 5 seconds) set based on the dynamic change pattern of material accumulation on the conveying equipment. This interval is used to filter out single-frame noise or temporary fluctuations (e.g., the 1-2 second thickness jump of tobacco shreds in a cigarette factory due to belt vibration; the preset time window size is at least larger than this temporary fluctuation time), ensuring that the judgment is for persistent anomalies. Furthermore, the time window needs to match the camera frame rate. If the camera frame rate is 15 frames / second, a 5-second window corresponds to 75 frames of continuous depth images. The preset thickness threshold can be specifically understood as a critical value set based on the equipment's safe conveying capacity (e.g., 90% of the trough depth). This is the dividing line between normal and abnormal thickness; exceeding this value can easily cause the cigarette material to clog or slip due to excessive accumulation.

[0066] Specifically, the system acquires multiple continuous depth images collected within a preset time window and calculates the average thickness value of the key monitoring area in each image, resulting in a set of average thickness data corresponding to the time sequence. This set of average thickness values ​​is then validated to determine if all values ​​are greater than a preset thickness threshold. If all average thickness values ​​meet the condition of being greater than the preset thickness threshold, the current condition for a continuous thickness anomaly is determined, and a thickness anomaly signal is generated. If any frame's average thickness value does not exceed the preset thickness threshold, it is determined to be a temporary thickness fluctuation, not meeting the continuous anomaly condition, and no thickness anomaly signal is generated. The system continues to update the average thickness values ​​of the continuous depth images in the time window (removing the average thickness value of the oldest frame for each newly added frame) and monitors the image data of the next time window.

[0067] By acquiring the average thickness values ​​of key monitoring areas corresponding to multiple depth images collected within a preset time window, and only determining an anomaly and generating a signal when all these values ​​are greater than a preset thickness threshold, it can effectively filter out instantaneous noise interference in a single frame caused by dust obstruction, belt vibration, and camera measurement errors in industrial scenarios. For example, if the thickness of a single frame is occasionally exceeded due to belt vibration during the transport of tobacco shreds in a cigarette factory, such non-continuous anomalies can be eliminated by verifying multiple consecutive frames within the time window, reducing the false alarm rate and avoiding unnecessary interference to the production process. Furthermore, the preset time window can cover the gradual process of material thickness increasing from normal to exceeding the limit, avoiding the problem of missed images of accumulation and overflow due to frame rate limitations in single-frame judgment. It can identify the gradual accumulation trend of material blockage, reduce the false alarm rate, reserve sufficient time for manual or automatic intervention, and improve the robustness of the system in complex environments.

[0068] S140. Extract the target region image from the color image that corresponds to the key monitoring area in the depth image, and input the target region image into the pre-trained color image classification model. When the output result meets the visual overflow condition, generate a visual overflow signal.

[0069] In this embodiment of the invention, the target area image can be specifically understood as: a sub-image in the color image whose coordinates completely match those of the key monitoring area in the depth image (determined through the coordinate mapping relationship calibrated by the camera), ensuring that the analysis range is consistent with the depth image (e.g., the key monitoring area in the depth image corresponds to the conveyor belt trough, and the target area in the color image also captures the range of that trough). The pre-trained color image classification model can be specifically understood as: a machine learning or deep learning model trained on a large amount of labeled data (e.g., color images of normal conveying, slight overflow, and severe overflow), used to identify whether the material exceeds the equipment boundary (i.e., visual overflow) or the degree of overflow. The input is the target area image, and the output is the classification result (e.g., overflow probability and category label).

[0070] The visual overflow condition can be understood as: the rule for determining whether a material has experienced visual overflow (e.g., when the probability of the overflow category output by the model is greater than or equal to a preset threshold, visual overflow is determined to have occurred), which can be set based on the feature threshold of historical overflow cases. The visual overflow signal can be understood as: the signal generated when the visual overflow condition is met, used to assist in verifying material anomalies from the visual feature dimension, forming a dual-modal cross-validation with the thickness anomaly signal of the depth image.

[0071] Specifically, based on the coordinate mapping relationship between the depth image and the color image (determined through camera calibration), a target area image is extracted from the color image that perfectly matches the location and range of the key monitoring area in the depth image, ensuring that both are analyzing the same physical region. The extracted target area image is input into a pre-trained color image classification model. This model identifies features such as color distribution and material outlines in the image and outputs a judgment result (e.g., category label and probability) regarding the presence of visual overflow. The system compares the model output with preset visual overflow conditions (e.g., whether the overflow category probability meets the threshold). If the conditions are met (e.g., the probability of slight or severe overflow is greater than a preset threshold), visual overflow is determined to exist in the target area, and a visual overflow signal is generated; otherwise, no signal is generated, completing one visual dimension anomaly judgment.

[0072] In a specific example, taking the tobacco conveying scenario in a cigarette factory as an example, based on the internal and external parameters calibrated at the factory of the binocular camera, a pixel coordinate mapping relationship between the depth image and the color image is established (e.g., the upper left corner coordinates (x, y) of the key monitoring area in the depth image correspond to the pixel coordinates in the color image). The lower right corner coordinates (a, b) correspond to This method extracts a target region image from the color image that perfectly matches the key monitoring area of ​​the depth image (e.g., the 800×400 pixel range of the belt conveyor trough), ensuring that the analysis focuses on the distribution of tobacco within and around the trough. This target region image is then input into a pre-trained color image classification model. This model has been trained using labeled samples (including three types of image samples: tobacco being transported normally within the trough, tobacco slightly overflowing the edge of the trough, and tobacco severely overflowing and covering the outside of the belt). The model extracts the color difference between the tobacco and the trough (e.g., the brownish-yellow tobacco and the black trough) and edge contour features (normally, the tobacco edge coincides with the trough boundary; when overflowing, the edge extends beyond the boundary), outputting the probability of the overflow category (e.g., a value between 0 and 1). The system compares the model's output probability with preset visual overflow conditions. If the visual overflow conditions are met, indicating that the tobacco in the target region image does indeed exceed the trough boundary, a visual overflow signal is generated for cross-validation with the thickness anomaly signal in the depth image. If the tobacco does not overflow, no signal is generated, completing the anomaly monitoring in the visual dimension.

[0073] S150. Based on the detection status of thickness anomaly signal and visual overflow signal, generate material blockage detection results.

[0074] Specifically, the system acquires the current detection status of two signals in real time (i.e., whether a thickness anomaly signal and a visual overflow signal are generated), forming four state combinations. For different combinations, scenario-adaptive judgment rules are formulated. For example, if both are generated, it indicates that the material has both three-dimensional thickness exceeding limits and two-dimensional visual overflow, indicating a very high risk of blockage requiring emergency intervention; if only the thickness anomaly signal is generated and the visual overflow signal is not, it indicates that although the material has not overflowed the visible range, its internal accumulation has exceeded the safe thickness, indicating an early blockage warning requiring close monitoring; if only the visual overflow signal is generated and the thickness anomaly signal is not, it indicates that although the material has not formed a large-scale thickness accumulation, boundary overflow has occurred, possibly due to equipment failure (such as material leakage or jamming), indicating a local blockage requiring on-site investigation; if neither is generated, it indicates that the material thickness is normal and there are no visual anomalies, indicating normal conveying without intervention. Finally, the judgment results are converted into executable instructions (such as pushing investigation tasks to the management platform, triggering on-site indicator lights and speakers to issue audible and visual warnings), completing the closed loop from signal monitoring to result output.

[0075] The technical solution of this invention involves simultaneously acquiring depth and color images using a binocular depth camera; locating and calibrating at least one key monitoring area on the depth image; calculating the relative difference between the depth value of each pixel within the key monitoring area and the thickness reference plane; and calculating the arithmetic mean of the thickness of all pixels within the key monitoring area based on each relative difference to obtain an average thickness value; generating a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly; extracting a target area image corresponding to the key monitoring area in the depth image from the color image and inputting the target area image into a pre-trained color image classification model; generating a visual overflow signal when the output result meets the condition of visual overflow; and generating a material blockage detection result based on the detection status of the thickness anomaly signal and the visual overflow signal. By simultaneously acquiring depth and color images using a binocular depth camera, the system can capture the three-dimensional accumulation state of materials based on the depth images. After locating key monitoring areas, by calculating the relative difference between pixels and the thickness reference plane, as well as the average thickness value of the area, a thickness anomaly signal can be generated based on the continuous anomaly condition of the average thickness value of multiple consecutive depth images when subtle anomalies occur in the material thickness. This eliminates the need to wait for material to visibly overflow, advancing the warning time and allowing sufficient time to handle blockages, reducing losses such as production line downtime and equipment damage. Furthermore, the depth information is less affected by differences in lighting and material color, resulting in high anomaly recognition accuracy. The color image can be used to extract the target area corresponding to the key monitoring area and input into a pre-trained classification model to generate a visual overflow signal. This not only helps verify the authenticity of the depth detection results to reduce false alarms but also serves as a redundant detection basis to avoid missed alarms when the depth channel fails. It adapts to complex interferences such as changes in lighting and dust in industrial scenarios, enhancing the system's robustness. Finally, the detection result is formed based on the detection status of the two signals, meeting the early warning needs of high-risk points while also adapting to efficient detection of conventional points, improving the timeliness, accuracy, and environmental adaptability of material blockage detection.

[0076] Example 2

[0077] Figure 2 This is a flowchart of another material blockage detection method provided in Embodiment 2 of the present invention. This embodiment is a refinement of the material blockage detection method in the above embodiments.

[0078] Correspondingly, such as Figure 2 As shown, the method includes:

[0079] S210: Simultaneously acquires depth and color images using a binocular depth camera.

[0080] S220. Locate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0081] S230. When multiple average thickness values ​​calculated for multiple continuous depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated.

[0082] S240. Extract the target area image from the color image that corresponds to the key monitoring area in the depth image.

[0083] S250. Scale the target region image to meet the input size conditions of the pre-trained color image classification model, and input the target region image into the pre-trained color image classification model. When the output result meets the visual overflow condition, generate a visual overflow signal.

[0084] In this embodiment of the invention, the input size condition can be specifically understood as: the neural network model has a fixed requirement for the pixel size of the input image (e.g., 224×224 pixels). The target area image needs to be scaled to this size to ensure that the model can perform feature extraction and inference normally (the model is trained with a fixed-size image as input, and a mismatch in input size will cause feature extraction to fail).

[0085] Specifically, the target area image extracted from the color image is scaled (e.g., by scaling the pixel dimension using bilinear interpolation, and by using weighted averaging to preserve the color difference and edge contours between the cigarette material (e.g., tobacco shreds) and the background (e.g., belt groove) during the scaling process) to ensure that the pixel size of the scaled image matches the input size of the model.

[0086] The scaled image is input into a pre-trained color image classification model (such as a lightweight convolutional neural network binary classification model based on MobileNetV3 architecture, already loaded onto an edge control computer). The model extracts material morphology features from the image (such as whether there are material outlines exceeding the equipment boundaries or whether the color distribution meets the overflow characteristics) through forward inference (extracting low, medium, and high-dimensional features through depthwise separable convolution, compressing features through global average pooling and fully connected layers, and finally outputting probabilities by a Softmax (normalization exponent) classifier). It outputs a classification result of normal or overflow and the corresponding probability. The system compares the model output with preset visual overflow conditions. If the conditions are met, it determines that there is visual overflow in the target area and generates a visual overflow signal.

[0087] Optionally, based on the above embodiments, the target region image is input into a pre-trained color image classification model, and a visual overflow signal is generated when the output result meets the visual overflow condition. This may include:

[0088] The target region image is input into a pre-trained color image classification model, and the overflow confidence corresponding to the target region image is obtained from the output of the pre-trained color image classification model.

[0089] When the overflow confidence level is greater than the preset confidence threshold, the visual overflow condition is determined to be met, and a visual overflow signal is generated.

[0090] In this embodiment of the invention, the overflow confidence level can be specifically understood as: a value between 0 and 1 output by the model, representing the probability that there is a material overflow feature in the input image (the closer the value is to 1, the higher the probability of overflow). The preset confidence threshold can be specifically understood as: a manually set critical value (such as 0.9) used to determine whether the overflow confidence level is sufficient to trigger a visual overflow signal, and needs to be determined by debugging in conjunction with scenario requirements (such as false alarm rate and false alarm rate).

[0091] Specifically, the captured target area image is input into a pre-trained color image classification model. The model outputs the overflow confidence score (i.e., the probability value of material overflow) corresponding to the image through forward inference. The overflow confidence score is compared with a preset confidence threshold. If the overflow confidence score is greater than the preset confidence threshold, it is determined that the visual overflow condition is met and a visual overflow signal is generated. If it is less than or equal to the threshold, no signal is generated, thus completing one visual dimension anomaly detection process.

[0092] Understandably, the humidity of materials will affect their surface color. Pre-trained color image classification models can be trained specifically based on the physical characteristics of cigarette materials (such as color and humidity) and the corresponding data collected in different seasons.

[0093] By inputting the target area image into a pre-trained color image classification model and generating a visual overflow signal based on the overflow confidence, the model can determine material overflow through quantified confidence and a pre-set confidence threshold. The model is trained on multi-condition datasets and can adapt to interference such as light fluctuations and changes in material physical properties in industrial scenarios, ensuring detection stability and reducing the risk of equipment damage and production interruption caused by blockage.

[0094] S260. Based on the detection status of thickness anomaly signal and visual overflow signal, generate material blockage detection results.

[0095] The technical solution of this invention involves simultaneously acquiring depth and color images using a binocular depth camera; locating and calibrating at least one key monitoring region on the depth image; calculating the relative difference between the depth value of each pixel within the key monitoring region and the thickness reference plane; and calculating the arithmetic mean of the thickness of all pixels within the key monitoring region based on each relative difference to obtain the average thickness value; generating a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly; and cropping a target region image corresponding to the key monitoring region in the depth image from the color image, scaling the target region image to meet the input size conditions of the pre-trained color image classification model, so that the pixel arrangement of the target region image is optimized. The column and feature dimensions are kept consistent with the standard input format of the model to avoid model inference failure or chaotic output results due to size mismatch. In industrial scenarios, real-time detection requires coordination with the camera's acquisition frame rate. If the target area image size is too large, it will increase computing power consumption and prolong inference time. Scaling to a smaller size can reduce the number of pixels processed, reduce inference time, and ensure that visual detection and depth detection proceed simultaneously, avoiding missing the best handling opportunity. In addition, a unified input size allows the model to extract features based on the same benchmark, eliminating feature differences of the same material state under different sizes. For example, large sizes dilute edges, and small sizes compress key features, reducing feature recognition bias, improving the stability of model output confidence, and reducing the risk of false positives and false negatives. The target area image is input into a pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated. Based on the detection status of thickness anomaly signal and visual overflow signal, material blockage detection results are generated, improving the timeliness, accuracy, and environmental adaptability of material blockage detection.

[0096] Example 3

[0097] Figure 3 This is a flowchart of another material blockage detection method provided in Embodiment 3 of the present invention. This embodiment is a refinement of the step in the above embodiment of "generating material blockage detection results based on the detection status of thickness anomaly signals and visual overflow signals". Accordingly, as Figure 3 As shown, the method includes:

[0098] S310: Simultaneously acquires depth and color images using a binocular depth camera.

[0099] S320. Locate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0100] S330. When multiple average thickness values ​​calculated for multiple continuous depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated.

[0101] S340. Extract the target region image from the color image that corresponds to the key monitoring area in the depth image, and input the target region image into the pre-trained color image classification model. When the output result meets the visual overflow condition, generate a visual overflow signal.

[0102] S350: When an abnormal thickness signal is detected, an early warning signal is generated.

[0103] In this embodiment of the invention, the early warning signal can be specifically understood as: the highest level alarm triggered by the thickness anomaly signal, used to indicate the earliest stage of blockage risk, with higher priority than the confirmation warning.

[0104] S360. When a visual overflow signal is detected and no thickness abnormality signal is detected within a preset time period prior to the detection of the visual overflow signal, a confirmation warning signal is triggered.

[0105] In this embodiment of the invention, the confirmation warning signal is an alarm triggered only by the visual overflow signal (and there is no thickness abnormality signal within a preset time period before its generation). It is a redundant backup when depth detection fails (such as sensor failure) and is used for risk confirmation in visible overflow scenarios.

[0106] Specifically, the system monitors the status of thickness anomaly signals and visual overflow signals in real time: if a thickness anomaly signal is detected, regardless of whether a visual overflow signal is present, an early warning signal (the highest level warning signal) is immediately generated. This is because continuous thickness exceeding the limit is the fundamental physical cause of blockage, and intervention time needs to be gained through the earliest warning.

[0107] If a visual overflow signal is detected, and no thickness abnormality signal is detected within a preset time period prior to the signal's generation, a confirmation warning signal is triggered as a redundant backup to ensure no risk is overlooked; if neither signal is detected, the material conveying status is determined to be normal.

[0108] In a specific example, a material blockage detection system consists of a binocular depth stereo camera, a data processing unit (industrial edge computing server), and a communication interface unit. The binocular depth stereo camera, with a resolution of 1280×720 and a frame rate of 15, simultaneously acquires depth and RGB images. It is installed 2 meters in front of the drive pulley of the conveyor belt (30 degrees from above) to ensure coverage of the effective working section of the trough belt and provide raw perception data for the system. The data processing unit uses an industrial control computer equipped with a graphics processing unit (GPU) accelerator card for depth analysis (filtering depth images, calculating the average thickness of key monitoring areas, and generating thickness anomaly signals through a 3-second time window), RGB analysis (cropping key monitoring areas of RGB images and scaling them to 224×224 pixels, inputting them into the MobileNetV3 model, and generating visual overflow signals based on overflow confidence), and fusion decision (polling dual signals to generate early warning signals or confirmation warning signals). The communication interface unit uploads alarm information to the upper-level control system through the OPC UA (Open Platform Communications Unified Architecture) protocol, linking audible and visual alarms and equipment control to complete the closed loop.

[0109] By prioritizing responses to depth and thickness anomalies, early warnings can be issued in the early stages of material accumulation before it overflows. Depth information is resistant to light and steam interference, and RGB information verifies the authenticity of depth. Dual-modal fusion reduces false alarm and false negative rates in complex environments. The two detection channels work independently, and the other channel can still detect when one channel fails, improving system availability. It is suitable for edge deployment, has controllable implementation costs, and is easy to promote in industrial scenarios.

[0110] The technical solution of this invention involves simultaneously acquiring depth and color images using a binocular depth camera; locating and calibrating at least one key monitoring area on the depth image; calculating the relative difference between the depth value of each pixel within the key monitoring area and the thickness reference plane; and calculating the arithmetic mean of the thickness of all pixels within the key monitoring area based on each relative difference to obtain an average thickness value; generating a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly; extracting a target area image corresponding to the key monitoring area in the depth image from the color image and inputting the target area image into a pre-trained color image classification model; generating a visual overflow signal when the output result meets the condition of visual overflow; and triggering the generation of an early warning signal when a thickness anomaly signal is detected. When a visual overflow signal is detected and no thickness anomaly signal is detected within a preset time period prior to the detection of the visual overflow signal, a confirmatory warning signal is triggered. The thickness anomaly signal reflects early physical signs of material three-dimensional accumulation, such as excessive thickness inside the tank but no overflow. In this case, an early warning signal is triggered, which can overcome the limitation of traditional detection that requires waiting for material to be visible to overflow. The warning node is brought forward, avoiding the failure from escalating to the point of production line shutdown or equipment damage. The visual overflow signal can serve as a redundant backup for depth detection. When the depth channel fails due to dust obstruction or other malfunctions, if visual overflow is detected and no thickness anomaly is detected within a preset time period, a confirmatory warning signal is triggered. This can cover the extreme scenario of depth failure but material overflow, avoiding the risk of missed detection caused by a single channel failure. At the same time, the hierarchical design of early warning and confirmatory warning can achieve differentiated responses. Early warnings indicate high risk of blockages, allowing staff to prioritize inspections without immediate shutdown, minimizing disruption to production. Confirmed warnings indicate a fault has occurred, requiring emergency shutdown. Furthermore, the absence of thickness anomalies within a preset timeframe eliminates the possibility of repeated alarms, enabling more precise response measures. This balances production continuity with fault safety, improving the timeliness, accuracy, and environmental adaptability of material blockage detection.

[0111] Example 4

[0112] Figure 4 This is a schematic diagram of a material blockage detection device provided in Embodiment 4 of the present invention. Figure 4 As shown, the device includes: a synchronous acquisition module 410, a positioning and calibration module 420, a thickness anomaly module 430, a visual overflow module 440, and a detection result module 450, wherein:

[0113] The synchronous acquisition module 410 is used to synchronously acquire depth images and color images using a binocular depth camera;

[0114] The positioning and calibration module 420 is used to locate and calibrate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0115] The thickness anomaly module 430 is used to generate a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the conditions for continuous thickness anomaly.

[0116] The visual overflow module 440 is used to extract the target area image corresponding to the key monitoring area in the depth image from the color image, and input the target area image into the pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated.

[0117] The detection result module 450 is used to generate material blockage detection results based on the detection status of thickness anomaly signals and visual overflow signals.

[0118] The technical solution of this invention involves simultaneously acquiring depth and color images using a binocular depth camera; locating and calibrating at least one key monitoring area on the depth image; calculating the relative difference between the depth value of each pixel within the key monitoring area and the thickness reference plane; and calculating the arithmetic mean of the thickness of all pixels within the key monitoring area based on each relative difference to obtain an average thickness value; generating a thickness anomaly signal when multiple average thickness values ​​calculated for multiple consecutive depth images meet the condition of continuous thickness anomaly; extracting a target area image corresponding to the key monitoring area in the depth image from the color image and inputting the target area image into a pre-trained color image classification model; generating a visual overflow signal when the output result meets the condition of visual overflow; and generating a material blockage detection result based on the detection status of the thickness anomaly signal and the visual overflow signal. By simultaneously acquiring depth and color images using a binocular depth camera, the system can capture the three-dimensional accumulation state of materials based on the depth images. After locating key monitoring areas, by calculating the relative difference between pixels and the thickness reference plane, as well as the average thickness value of the area, a thickness anomaly signal can be generated based on the continuous anomaly condition of the average thickness value of multiple consecutive depth images when subtle anomalies occur in the material thickness. This eliminates the need to wait for material to visibly overflow, advancing the warning time and allowing sufficient time to handle blockages, reducing losses such as production line downtime and equipment damage. Furthermore, the depth information is less affected by differences in lighting and material color, resulting in high anomaly recognition accuracy. The color image can be used to extract the target area corresponding to the key monitoring area and input into a pre-trained classification model to generate a visual overflow signal. This not only helps verify the authenticity of the depth detection results to reduce false alarms but also serves as a redundant detection basis to avoid missed alarms when the depth channel fails. It adapts to complex interferences such as changes in lighting and dust in industrial scenarios, enhancing the system's robustness. Finally, the detection result is formed based on the detection status of the two signals, meeting the early warning needs of high-risk points while also adapting to efficient detection of conventional points, improving the timeliness, accuracy, and environmental adaptability of material blockage detection.

[0119] Based on the above embodiments, a binocular depth camera is installed at a process point prone to blockage on the conveying equipment, and the shooting angle and focal length of the binocular depth camera meet the condition of complete field of view coverage of the key monitoring area.

[0120] Accordingly, based on the above embodiments, the material blockage detection device may further include: an initial calibration module and a thickness reference module, wherein:

[0121] The initial calibration module is used to acquire initial depth images and calibrate at least one key monitoring area while the conveying equipment is running unloaded, before simultaneously acquiring depth and color images through a binocular depth camera.

[0122] The thickness reference module is used to calculate the average depth value of all pixels in the key monitoring area, which serves as the thickness reference surface.

[0123] Furthermore, based on the above embodiments, the material blockage detection device may further include: a mean filtering module, wherein:

[0124] The mean filtering module is used to perform non-local mean filtering on the depth image after the depth image and color image are acquired simultaneously by the binocular depth camera.

[0125] Based on the above embodiments, the thickness anomaly module 430 is specifically used for:

[0126] Obtain the average thickness value of the key monitoring area corresponding to multiple depth images acquired within a preset time window;

[0127] When multiple average thickness values ​​are greater than a preset thickness threshold, the condition for continuous thickness anomaly is determined to be met, and a thickness anomaly signal is generated.

[0128] Furthermore, based on the above embodiments, the material blockage detection device may further include: an image scaling module, wherein:

[0129] The image scaling module is used to scale the target region image to meet the input size requirements of the pre-trained color image classification model before inputting the target region image into the pre-trained color image classification model.

[0130] Based on the above embodiments, the visual overflow module 440 is specifically used for:

[0131] The target region image is input into a pre-trained color image classification model, and the overflow confidence corresponding to the target region image is obtained from the output of the pre-trained color image classification model.

[0132] When the overflow confidence level is greater than the preset confidence threshold, the visual overflow condition is determined to be met, and a visual overflow signal is generated.

[0133] Based on the above embodiments, the detection result module 450 is specifically used for:

[0134] When an abnormal thickness signal is detected, an early warning signal is generated.

[0135] When a visual overflow signal is detected and no thickness anomaly signal is detected within a preset time period prior to the detection of the visual overflow signal, a confirmation warning signal is triggered.

[0136] The material blockage detection device provided in the embodiments of the present invention can execute the material blockage detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0137] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0138] Example 5

[0139] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0140] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0141] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as material blockage detection methods, i.e.:

[0143] Simultaneous acquisition of depth and color images using a binocular depth camera;

[0144] Locate and calibrate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value.

[0145] When multiple average thickness values ​​calculated from multiple consecutive depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated.

[0146] The target region image corresponding to the key monitoring area in the depth image is extracted from the color image, and the target region image is input into the pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated.

[0147] Based on the detection status of thickness anomaly signals and visual overflow signals, material blockage detection results are generated.

[0148] In some embodiments, the material blockage detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the material blockage detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the material blockage detection method by any other suitable means (e.g., by means of firmware).

[0149] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0154] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0155] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting material blockage, characterized in that, include: Simultaneous acquisition of depth and color images using a binocular depth camera; Locate and calibrate at least one key monitoring area on the depth image, calculate the relative difference between the depth value of each pixel in the key monitoring area and the thickness reference plane, and calculate the arithmetic mean of the thickness of all pixels in the key monitoring area based on each relative difference to obtain the average thickness value. When multiple average thickness values ​​calculated from multiple consecutive depth images meet the condition of continuous thickness anomaly, a thickness anomaly signal is generated. The target region image corresponding to the key monitoring area in the depth image is extracted from the color image, and the target region image is input into the pre-trained color image classification model. When the output result meets the visual overflow condition, a visual overflow signal is generated. Based on the detection status of thickness anomaly signals and visual overflow signals, material blockage detection results are generated.

2. The method according to claim 1, characterized in that, The binocular depth camera is installed at the easily clogged process points of the conveying equipment, and the shooting angle and focal length of the binocular depth camera meet the condition of complete field of view coverage of the key monitoring area. Accordingly, before simultaneously acquiring depth and color images using a binocular depth camera, the following steps are also included: While the conveying equipment is running unloaded, acquire initial depth images and mark at least one key monitoring area; Calculate the average depth value of all pixels within the critical monitoring area, and use it as the thickness reference plane.

3. The method according to claim 1, characterized in that, After simultaneously acquiring depth and color images using a binocular depth camera, the process also includes: Perform nonlocal mean filtering on the depth image.

4. The method according to claim 1, characterized in that, When multiple average thickness values ​​calculated from multiple consecutive depth images satisfy the condition for persistent thickness anomalies, a thickness anomaly signal is generated, including: Obtain the average thickness value of the key monitoring area corresponding to multiple depth images acquired within a preset time window; When multiple average thickness values ​​are greater than a preset thickness threshold, the condition for continuous thickness anomaly is determined to be met, and a thickness anomaly signal is generated.

5. The method according to claim 1, characterized in that, Before inputting the target region image into the pre-trained color image classification model, the following steps are also included: The target region image is scaled to meet the input size requirements of the pre-trained color image classification model.

6. The method according to claim 1, characterized in that, The target region image is input into a pre-trained color image classification model. When the output meets the visual overflow condition, a visual overflow signal is generated, including: The target region image is input into a pre-trained color image classification model, and the overflow confidence corresponding to the target region image is obtained from the output of the pre-trained color image classification model. When the overflow confidence level is greater than the preset confidence threshold, the visual overflow condition is determined to be met, and a visual overflow signal is generated.

7. The method according to claim 1, characterized in that, Based on the detection status of thickness anomaly signals and visual overflow signals, material blockage detection results are generated, including: When an abnormal thickness signal is detected, an early warning signal is generated. When a visual overflow signal is detected and no thickness anomaly signal is detected within a preset time period prior to the detection of the visual overflow signal, a confirmation warning signal is triggered.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the material blockage detection method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the material blockage detection method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the material blockage detection method according to any one of claims 1-7.

Images