Material jam detection method and terminal device

Abstract

This application relates to the field of production monitoring technology, and in particular to a material blockage detection method and terminal equipment. The method includes: acquiring image information of a material falling section to obtain a first image and a second image; the first image and the second image are two consecutive frames captured from the same viewpoint; calculating the optical flow vector of each pixel using the first image and the second image; acquiring a first pixel judgment area and a second pixel judgment area set in the first image or the second image; obtaining a first optical flow velocity and a second optical flow velocity based on the optical flow vector of each pixel within the first pixel judgment area and the second pixel judgment area; and determining whether the material is in a blockage state based on the first optical flow velocity and the second optical flow velocity. Therefore, this application can effectively solve the problems of complex operation and low detection accuracy in existing material blockage detection methods.

Description

Technical Field

[0001] This application relates to the field of production monitoring technology, and in particular to a material blockage detection method and terminal equipment. Background Technology

[0002] Coal unloading and sorting machinery includes a plow-type unloader and a belt conveyor, with the plow-type unloader and belt conveyor connected accordingly. Under normal belt conveyor operation, when the plow-type unloader is raised, the material (usually coal) on the belt is conveyed along with it; when the plow-type unloader is lowered, the material on the belt is diverted to both sides and enters the feeders on either side, thus achieving coal unloading and sorting. However, in actual operation, material blockage (generally called coal blockage) may occur, which can seriously affect the production line, such as reducing efficiency, material leakage, belt damage, damage to unloader components, or affecting equipment lifespan.

[0003] To address the above issues, existing technologies employ the following methods to detect coal blockage:

[0004] The first method involves manually observing the working status of the plow-type unloader. If any abnormalities are found around the unloader, such as blockage at the discharge port or sudden changes in material flow, it indicates a potential coal blockage, requiring the plow to be raised for control. However, this manual method is costly and prone to oversights due to worker fatigue, resulting in low efficiency.

[0005] The second method involves installing sensor devices on the baffle or related components of the plow-type unloader. Commonly used sensors are pressure sensors. These sensors determine if material accumulation has occurred based on pressure values. If material accumulation occurs and the pressure sensor detects a threshold value, an alarm is triggered, requiring plow lifting control. This method requires additional hardware installation and adjustments to alarm parameters based on different materials and transmission speeds, making it cumbersome. Furthermore, the harsh on-site environment can affect sensor sensitivity and lifespan over time, ultimately impacting detection accuracy.

[0006] The third method involves checking the motor current and voltage of the plow-type unloader. Abnormal motor current and voltage indicate potential coal blockage. However, while abnormal motor current and voltage confirm coal blockage, requiring machine shutdown and inspection, in practice, voltage and current fluctuations are not always indicative of coal blockage, and their correlation is weak. Therefore, this method of coal blockage detection is inaccurate. Summary of the Invention

[0007] In view of this, the embodiments of this application provide a material blockage detection method and terminal equipment, which can effectively solve the problems of complex operation and low detection accuracy in the existing material blockage detection methods.

[0008] In a first aspect, embodiments of this application provide a method for detecting material blockage, including:

[0009] Image information of the material falling section is acquired to obtain a first image and a second image; the first image and the second image are two consecutive frames captured from the same viewpoint.

[0010] The optical flow vector of each pixel is calculated using the first image and the second image;

[0011] Obtain a first pixel judgment region and a second pixel judgment region set in the first image or the second image; the first pixel judgment region and the second pixel judgment region satisfy that they are located in the same image in the material falling section and are perpendicular to the material flow direction.

[0012] Based on the optical flow vectors of each pixel in the first pixel judgment area and the second pixel judgment area, the first optical flow velocity and the second optical flow velocity are obtained;

[0013] Based on the first optical flow velocity and the second optical flow velocity, it is determined whether the material is in a blocked state.

[0014] In some embodiments, the first pixel determination region and the second pixel determination region are two parallel line segments, which correspond to the first pixel determination line segment and the second pixel determination line segment, respectively.

[0015] The step of obtaining the first optical flow velocity and the second optical flow velocity based on the optical flow vectors of each pixel in the first pixel judgment region and the second pixel judgment region includes:

[0016] Determine the perpendicular line of the first pixel judgment line segment or the second pixel judgment line segment to obtain the judgment line segment perpendicular line;

[0017] In the optical flow vectors of each pixel, the optical flow vectors of the pixels on the first pixel judgment line segment and the second pixel judgment line segment are selected respectively to obtain the first set of optical flow vectors and the second set of optical flow vectors.

[0018] The projections of each optical flow vector in the first set of optical flow vectors and the second set of optical flow vectors onto the perpendicular line of the judgment line segment are calculated respectively to obtain the corresponding projection values. The corresponding projection values ​​are used as the optical flow velocity of the corresponding pixel, thereby obtaining the first set of optical flow velocity and the second set of optical flow velocity.

[0019] The first optical flow velocity is obtained based on the first set of optical flow velocities, and the second optical flow velocity is obtained based on the second set of optical flow velocities.

[0020] In some embodiments, obtaining the first optical flow velocity based on the first set of optical flow velocities and obtaining the second optical flow velocity based on the second set of optical flow velocities includes:

[0021] The average optical flow velocity is obtained by averaging the optical flow velocities in the first set of optical flow velocities, and the average optical flow velocity is used as the first optical flow velocity.

[0022] The average optical flow velocity is obtained by averaging the optical flow velocities in the second set of optical flow velocities, and the average optical flow velocity is used as the second optical flow velocity.

[0023] In some embodiments, the sizes of the first image and the second image are determined based on the outer boundaries of the first pixel determination region and the second pixel determination region.

[0024] In some embodiments, determining whether the material is in a blocked state based on the first optical flow velocity and the second optical flow velocity includes:

[0025] Determine whether the absolute value of the difference between the first optical flow velocity and the second optical flow velocity is greater than a set threshold.

[0026] If the absolute value of the difference between the first optical flow velocity and the second optical flow velocity is greater than a set threshold, then the material is determined to be in a blocked state; otherwise, the material is determined not to be blocked.

[0027] In some embodiments, the set threshold is obtained using the following formula:

[0028] theta = max(a, bV1)

[0029] Where theta is the set threshold, a is the minimum difference empirical value, b is the set percentage, and V1 is the first optical flow velocity.

[0030] In some embodiments, before obtaining the first optical flow velocity and the second optical flow velocity, the method further includes:

[0031] Obtain the optical flow vector for each pixel;

[0032] The optical flow vector of each pixel is preprocessed to remove interference data; wherein, the preprocessing includes removing data with an optical flow velocity of 0, and removing data whose optical flow vector direction differs from the perpendicular line of the judgment line segment by a preset difference.

[0033] In some embodiments, the RAFT algorithm is used to calculate the optical flow vector of each pixel based on the first image and the second image.

[0034] Secondly, embodiments of this application provide a terminal device, which includes a camera, a processor, and a memory. The camera is used to capture image information of a material falling section; the memory stores a computer program, and the processor is used to execute the computer program to implement a material blockage detection method provided in the first aspect of this application.

[0035] In some embodiments, the terminal device is an inspection robot.

[0036] The embodiments of this application have the following beneficial effects:

[0037] This application uses two optical flow velocities from different regions perpendicular to the material direction in the same image to determine whether coal blockage has occurred. The method is simple, unaffected by conveyor belt speed, and offers high detection accuracy. Furthermore, to reduce computation, this application uses the velocity difference between two parallel line segments to determine if coal blockage has occurred. If the velocity at the lower end of the blockage is slower, the velocity difference is larger, thus detecting the blockage. To improve accuracy, this application uses the average velocity on the first and second pixel judgment line segments for calculation, ensuring precision. Moreover, using the projection of the optical flow vector onto the perpendicular line of the judgment line segment as the optical flow velocity fully considers the main direction of the material, further improving detection accuracy. Therefore, this application effectively solves the problems of complex operation and low detection accuracy in existing material blockage detection methods. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart of a material blockage detection method according to an embodiment of this application is shown;

[0040] Figure 2 This illustration shows a schematic diagram of a pixel-based line segment determination method in a material blockage detection method according to an embodiment of this application.

[0041] Figure 3 This illustration shows a schematic diagram of determining the perpendicularity of a line segment in the material blockage detection method according to an embodiment of this application;

[0042] Figure 4 Another flowchart of the material blockage detection method according to an embodiment of this application is shown;

[0043] Figure 5A schematic diagram of a material blockage detection device according to an embodiment of this application is shown.

[0044] Explanation of key component symbols:

[0045] 510 - Image acquisition module; 520 - Optical flow vector calculation module; 530 - Judgment region acquisition module; 540 - Optical flow velocity calculation module; 550 - Blockage status judgment module. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0047] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0048] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0049] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0050] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0051] In existing technologies, plow-type unloaders detect coal blockage during operation by manually measuring the pressing force using pressure sensors or by monitoring the generator's operating current and voltage. However, some of these methods require parameter adjustments based on different materials and speeds, while others are not strongly correlated with coal blockage. Overall, existing technologies are complex to operate and have low detection accuracy. Therefore, this application proposes a material blockage detection method and terminal equipment, which can effectively solve the problems of complex operation and low detection accuracy in existing material blockage detection methods.

[0052] The following examples illustrate the method for detecting material blockage.

[0053] Figure 1 A flowchart of a material blockage detection method according to an embodiment of this application is shown. Exemplarily, the material blockage detection method includes the following steps:

[0054] S10: Acquire image information of the material falling section to obtain a first image and a second image. The first image and the second image are two consecutive frames captured from the same viewpoint. To ensure real-time performance and accuracy, this embodiment processes each consecutive pair of images frame by frame.

[0055] In this embodiment, a camera is used to acquire image information of the material as it leaves the conveyor belt and falls into the discharge port. This embodiment acquires real-time image information at a shooting rate of 25 frames per second to obtain video data. Then, the video data from the discharge port is analyzed frame by frame to determine whether coal blockage has occurred.

[0056] The camera can be mounted independently or integrated into a rail-mounted inspection robot. Rail-mounted inspection robots already exist and are used for inspection tasks other than material blockage detection. Generally, the camera is positioned diagonally above the leakage port to clearly see its details. If lighting is insufficient, additional lights can be added. Furthermore, if using a camera from an inspection robot, a stationary observation point (i.e., one that cannot detect material blockage while in motion) needs to be set up above the leakage port, adjusting the viewing angle and focus appropriately, and activating supplementary lighting if necessary. The video is in 1080p, H.264 / H.264 encoding format.

[0057] In other words, this method can be used for rail-mounted inspection robots, reusing their existing cameras to achieve material blockage detection at multiple material inlets in a rotating manner. Alternatively, fixed cameras can be used to detect material blockage at a fixed material inlet. Regarding hardware costs, coal blockage detection is merely an additional feature of rail-mounted inspection robots; if the robot already has it, no additional hardware cost is considered necessary. If fixed cameras are used, the cost is negligible because the material inlets themselves also require video monitoring. In this case, an edge computing box is recommended, which can simultaneously support 4-8 channels of real-time analysis, with an average hardware cost of less than 1000 yuan per channel.

[0058] S20, the optical flow vector of each pixel is calculated using the first image and the second image.

[0059] The optical flow vector of each pixel is calculated using two consecutively captured images, the first and the second. This optical flow vector includes both the optical flow velocity and direction. The calculation of the optical flow vector can be performed using existing methods, and no restrictions are imposed here.

[0060] Furthermore, to improve detection accuracy, this embodiment employs the RAFT (Recurrent All-PairsField Transforms) algorithm to calculate the optical flow vector of each pixel based on the first and second images. The RAFT algorithm extracts features at full resolution and then constructs a multi-scale 4D correlation space. Although this step is computationally intensive, the cost volume (matching cost) can be obtained later through a lookup table, eliminating the need for further calculation. RAFT iteratively updates the optical flow field using GRU recurrent units, thus simulating the optimization iteration process in traditional methods. The introduction of GRU innovatively yields excellent results. This is fundamentally different from the previous coarse-to-fine strategy. Raft continuously iterates and updates at high resolution, rather than estimating the optical flow at low resolution first, then passing it layer by layer to high resolution for final optimization. The advantages of this approach are that it avoids the influence of incorrect estimations at low resolution, does not miss information about fast-moving small objects, and, due to the fixed weights, has a smaller overall parameter set, resulting in faster convergence and requiring fewer training iterations.

[0061] Specifically, RAFT can be divided into the following layers:

[0062] Feature Extractor (with weight sharing) & Context Extractor (semantic feature extraction module).

[0063] The feature extraction module extracts features from two images pixel by pixel, while the semantic feature extraction module has the same architecture as the feature extraction module, but only extracts features from one of the modules.

[0064] The Visual Similarity Calculator constructs a 4D correlation space by calculating the dot product between pairwise feature vectors of two images, thus obtaining 4D association information. It's important to note that W*H*W*H here does not refer to the full resolution of the image, but rather to the full resolution of the features. For example, if the feature vectors are constructed at 1 / 8 resolution, then the so-called full resolution becomes 1 / 8 resolution.

[0065] The Updator module uses the current optical flow estimation results to search for 4D correlation information based on the GRU cyclic update operator, and then iteratively updates the optical flow field with an initial value of 0.

[0066] RAFT obtains the optical flow vector using the following method:

[0067] 1) Grayscale processing is performed to obtain the first grayscale image and the second grayscale image.

[0068] The first image I1 and the second image I2 are converted into grayscale images to obtain a first grayscale image and a second grayscale image. The grayscale value range is (0, 255), and the standardized grayscale value is obtained using the following formula:

[0069]

[0070] Where g represents the grayscale value, and the value of g ranges from (0, 255). f G represents the standardized grayscale value. f The value range is (-1.0, 1.0).

[0071] 2) The feature extraction module extracts feature maps of the first and second grayscale images. The module uses two CNN architectures with shared weights, each containing a 6-layer residual network, to calculate the feature maps f(I1) and f(I2) of the first and second grayscale images, respectively. The resolution of the feature maps is 1 / 8 of the original image.

[0072] 3) Correlation Calculation. The similarity calculation module uses the dot product method to calculate the correlation between any two pixels, resulting in a four-dimensional tensor called the correlation volume. This four-dimensional tensor provides key information about pixel displacement, also known as 4D correlation information, for use in the correlation lookup table.

[0073] 4) Iteratively calculate and update the optical flow vector using the update iteration module. Set the initial optical flow graph to all zeros, and update the optical flow through a series of GRUs. The iterative process includes the following steps:

[0074] (1) Initialize the optical flow information to 0: f0 = 0, and estimate the optical flow information {f1, ..., f n}

[0075] (2) In each iteration, the direction of optical flow update is calculated as Δf.

[0076] (3) Update the current optical flow information flow: f k +1=Δf+f k .

[0077] The process for calculating the optical flow update direction Δf in each iteration is as follows: using the current optical flow information f k Using the 4D correlation information obtained in step 3), a correlation matrix is ​​interpolated. For each pixel in the first image I1, this matrix includes the correlation of all potential locations in the second image I2. A GRU (Gate Recurrent Unit, a type of RNN) is used to calculate the optical flow update direction Δf for this iteration, as well as the hidden status of the GRU. This hidden status serves as the input for the next iteration. Based on the optical flow update direction Δf calculated in this iteration, the current optical flow information flow is updated to obtain the optical flow information f for this iteration. The resolution of the irrigation information flow remains 1 / 8 of the original image.

[0078] 5) Upsample the optical flow information to obtain the optical flow at the original resolution. The 8x upsampling process can be simply described as follows: each pixel is expanded into 8*8 pixels. Specifically, each expanded pixel is obtained by weighting the original pixel and its 8 neighboring pixels (a total of 9 pixels), and the weights are generated by the network. This yields the optical flow amount (optical flow information) at the original resolution.

[0079] S30, obtain the first pixel judgment region and the second pixel judgment region set in the first image or the second image. The first pixel judgment region and the second pixel judgment region satisfy the condition that they are located in the material falling segment in the same image and are perpendicular to the material flow direction. The first pixel judgment region and the second pixel judgment region can be set manually or obtained by an adaptive method.

[0080] In one implementation, since the RAFT algorithm involves a large amount of computation, to reduce this computational load, this embodiment determines the size of the first and second images based on the outer boundaries of the first and second pixel judgment regions. That is, the first image is designed to include both the first and second pixel judgment regions. Similarly, the second image also includes both the first and second pixel judgment regions. Specifically, based on the image captured by the camera, cropping is performed so that the outer boundaries of the first and second pixel judgment regions in the image are used as the outer boundaries of the first image. Similarly, the second image is obtained.

[0081] S40: Based on the optical flow vectors of each pixel in the first pixel judgment area and the second pixel judgment area, the first optical flow velocity and the second optical flow velocity are obtained.

[0082] In one implementation, such as Figure 2 As shown, the first pixel judgment region and the second pixel judgment region are two parallel line segments, corresponding to the first pixel judgment line segment L1 and the second pixel judgment line segment L2, respectively. The minimum bounding rectangle of the first pixel judgment line segment L1 and the second pixel judgment line segment L2 is set as the processing region. It can be understood that the image regions of the first image and the second image are this processing region. It can be understood that the first pixel judgment line segment L1 and the second pixel judgment line segment L2 can be obtained by manual calibration or by an adaptive method. For example, by manually marking the start and end points and recording the pixel information of the start and end points, the first pixel judgment line segment L1 can be determined based on the start and end points.

[0083] Step S40, based on the optical flow vectors of each pixel within the first and second pixel judgment regions, yields the first optical flow velocity and the second optical flow velocity, including:

[0084] Determine the perpendicular line to either the first pixel's judgment segment L1 or the second pixel's judgment segment L2, and obtain the perpendicular line P of the judgment segment, such as... Figure 3 As shown;

[0085] In the optical flow vectors of each pixel, the optical flow vectors of each pixel on the first pixel judgment line segment L1 are selected to obtain the first set of optical flow vectors. That is, the set of pixels common to the first pixel judgment line segment L1 and the first image is obtained, or the set of pixels covered by the first pixel judgment line segment L1. The optical flow vectors of each pixel in this set of pixels are selected to obtain the first set of optical flow vectors.

[0086] In the optical flow vector of each pixel, the second pixel is selected to determine the optical flow vector of each pixel on line segment L2, and the corresponding second set of optical flow vectors is obtained.

[0087] The projections of each optical flow vector in the first set of optical flow vectors and the second set of optical flow vectors onto the perpendicular line P of the judgment line segment are calculated respectively to obtain the corresponding projection values. The corresponding projection values ​​are used as the optical flow velocities of the corresponding pixels, thereby obtaining the optical flow velocities of the first set and the second set.

[0088] The first optical flow velocity is obtained based on the first set of optical flow velocities, and the second optical flow velocity is obtained based on the second set of optical flow velocities.

[0089] Furthermore, to improve detection accuracy, remove noise, and enhance robustness, a first optical flow velocity is obtained based on a first set of optical flow velocities, and a second optical flow velocity is obtained based on a second set of optical flow velocities, including:

[0090] The average optical flow velocity is obtained by averaging the optical flow velocities in the first set of optical flow velocities, and this average optical flow velocity is used as the first optical flow velocity.

[0091] The average optical flow velocity is obtained by averaging the optical flow velocities in the second set of optical flow velocities, and this average optical flow velocity is used as the second optical flow velocity.

[0092] S50 determines whether the material is blocked based on the first optical flow velocity and the second optical flow velocity.

[0093] In this step, the material can be determined to be in a blocked state based on the ratio or difference between the first and second optical flow velocities.

[0094] In one embodiment, the blockage state of the material is determined by the difference between a first optical flow velocity and a second optical flow velocity. Specifically, it is determined whether the absolute value of the difference between the first and second optical flow velocities is greater than a set threshold. If the absolute value of the difference between the first and second optical flow velocities is greater than the set threshold, the material is determined to be in a blockage state; otherwise, the material is determined not to be blocked.

[0095] In one implementation, the threshold is set using the following formula:

[0096] theta = max(a, bV1)

[0097] Where theta is a set threshold, a is an empirical value for the minimum difference, b is a set percentage, and V1 is the first optical flow velocity. Preferably, a = 2 and b = 20%.

[0098] Furthermore, when material blockage is detected, the method also outputs an alarm message. In this embodiment, the alarm message can be output to the platform or directly linked to the unloader and output to the unloader.

[0099] Furthermore, to improve detection accuracy, before obtaining the first optical flow velocity and the second optical flow velocity, the following steps are also included:

[0100] The optical flow vector of each pixel is obtained and preprocessed to remove interfering data. This preprocessing includes removing data with an optical flow velocity of 0, and removing data whose optical flow vector direction differs from the perpendicular line of the judgment segment by a value greater than a preset difference. In other words, data with large directional differences and data with a velocity of 0 are removed. The preset difference can be determined through multiple tests in specific experiments, for example, 45°. The specific method involves traversing all pixels and calculating the optical flow vector, which contains both a direction and a velocity magnitude. Theoretically, the direction should be perpendicular to the calibrated perpendicular line P of the judgment segment, and the velocity should be greater than 0.

[0101] The following describes a material blockage detection method of this application based on a specific application scenario. In this scenario, the material is coal, the unloader is a plow-type unloader, and the conveyor belt delivers the transported coal into the plow-type unloader. Figure 4 As shown, it includes the following steps:

[0102] S210: After receiving the falling information emitted by the plow unloader during operation, the camera begins to capture image information.

[0103] The control of plow-type unloaders is generally manual, but with the development of intelligent technology, more intelligent auxiliary controls are being incorporated. In this embodiment, the plow-type unloader and the material conveying are controlled by a fixed lifting and lowering control program. When the plow-type unloader is working, it sends down-falling information. To reduce data storage and computational load, the camera starts capturing image information after receiving the down-falling information.

[0104] S220: Acquire image information of the process from coal leaving the conveyor belt to falling into the plow unloader until it stops, and obtain the first and second base images of consecutive frames.

[0105] S230: Mark two parallel line segments on the first base image, namely the first pixel judgment line segment L1 and the second pixel judgment line segment L2, and mark the first pixel judgment line segment L1 and the second pixel judgment line segment L2 on the second base image. This step can be done using drawing software by marking the starting point and recording the two pixel points to obtain the line segments.

[0106] S240, based on the outer boundaries of the first pixel judgment line segment L1 and the second pixel judgment line segment L2, crop the first base image and the second base image so that the image only contains the first pixel judgment line segment L1 and the second pixel judgment line segment L2, as well as the pixels between the first pixel judgment line segment L1 and the second pixel judgment line segment L2, and obtain the first image and the second image accordingly.

[0107] S250 uses the RAFT algorithm to calculate the optical flow vector of each pixel using the first and second images.

[0108] S260, acquire the optical flow vector of each pixel; preprocess the optical flow vector of each pixel to remove interference data.

[0109] S270, obtain each pixel point on the first pixel judgment line segment L1 and the second pixel judgment line segment L2 respectively, obtain the optical flow vector of each pixel point, and obtain the first set optical flow vector and the second set optical flow vector accordingly.

[0110] S280, calculate the projection of the optical flow vector onto the perpendicular line P of the judgment line segment to obtain the optical flow velocity.

[0111] Determine the perpendicular line of the first pixel judgment line segment L1 to obtain the judgment line segment perpendicular line P. Calculate the projection of each optical flow vector in the first set of optical flow vectors and the second set of optical flow vectors onto the judgment line segment perpendicular line P to obtain the corresponding projection value. Use the corresponding projection value as the optical flow velocity of the corresponding pixel, and then obtain the first set of optical flow velocity and the second set of optical flow velocity.

[0112] S290, calculate the average optical flow velocity to obtain the first optical flow velocity V1 and the second optical flow velocity V2.

[0113] The average optical flow velocity is obtained by averaging the optical flow velocities in the first set of optical flow velocities, and the average optical flow velocity is taken as the first optical flow velocity V1.

[0114] The average optical flow velocity is obtained by averaging the optical flow velocities in the second set of optical flow velocities. This average optical flow velocity is then used as the second optical flow velocity V2.

[0115] S300 determines whether the material is blocked based on the values ​​of V1-V2 and theta.

[0116] Determine the value of V1-V2 relative to theta, where theta = max(2, V1 * 20%). If V1-V2 > theta, the material is definitely blocked. If V1-V2 ≤ theta, the material is definitely not blocked.

[0117] This application enables coal blockage detection based on a standard camera at the discharge port, providing fully automated processing without manual intervention. Furthermore, this application is unaffected by belt conveyor speed and exhibits high robustness. As an auxiliary control for plow-type unloaders, it can effectively detect coal blockage, thereby reducing safety hazards in the production environment.

[0118] Figure 5A schematic diagram of a material blockage detection device according to an embodiment of this application is shown. Exemplarily, the material blockage detection device includes: an image acquisition module 510, an optical flow vector calculation module 520, a judgment region acquisition module 530, an optical flow velocity calculation module 540, and a blockage state judgment module 550.

[0119] The image acquisition module 510 is used to acquire image information of the material falling section and obtain a first image and a second image; the first image and the second image are two consecutive frames captured from the same viewpoint.

[0120] The optical flow vector calculation module 520 is used to calculate the optical flow vector of each pixel using the first image and the second image.

[0121] The judgment area acquisition module 530 is used to acquire the first pixel judgment area and the second pixel judgment area set in the first image or the second image; the first pixel judgment area and the second pixel judgment area satisfy that they are located in the material falling section in the same image and are perpendicular to the material flow direction.

[0122] The optical flow velocity calculation module 540 is used to obtain the first optical flow velocity and the second optical flow velocity based on the optical flow vector of each pixel in the first pixel judgment area and the second pixel judgment area.

[0123] The blockage state determination module 550 is used to determine whether the material is in a blockage state based on the first optical flow velocity and the second optical flow velocity.

[0124] It is understood that the device in this embodiment corresponds to the material blockage detection method in the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0125] This application also provides a terminal device, exemplary of which includes a camera, a processor, and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described material blockage detection method or the above-described material blockage detection device.

[0126] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0127] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0128] In one implementation, the terminal device is an inspection robot. The inspection robot includes a camera and a computing unit.

[0129] The camera is used to capture images. The computing unit is used to execute the functions of each module in the material blockage detection method or the material blockage detection device described above. The computing unit can be a server or an embedded computing box. Embedded computing boxes are low-cost and can simultaneously support material blockage detection at 4-8 leakage ports. In this embodiment, the computing unit can also be an embedded computing chip embedded in the inspection robot.

[0130] This application also provides a readable storage medium for storing the computer program used in the aforementioned terminal device.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0132] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0133] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for detecting material blockage, characterized in that, include: Image information of the material falling section is acquired to obtain a first image and a second image; the first image and the second image are two consecutive frames captured from the same viewpoint. The optical flow vector of each pixel is calculated using the first image and the second image; Obtain a first pixel judgment region and a second pixel judgment region set in the first image or the second image; the first pixel judgment region and the second pixel judgment region satisfy that they are located in the same image in the material falling section and are perpendicular to the material flow direction. Based on the optical flow vectors of each pixel in the first pixel judgment area and the second pixel judgment area, the first optical flow velocity and the second optical flow velocity are obtained; Based on the first optical flow velocity and the second optical flow velocity, determine whether the material is in a blocked state; The first pixel judgment area and the second pixel judgment area are two parallel line segments, which correspond to the first pixel judgment line segment and the second pixel judgment line segment, respectively. The step of obtaining the first optical flow velocity and the second optical flow velocity based on the optical flow vectors of each pixel in the first pixel judgment region and the second pixel judgment region includes: Determine the perpendicular line of the first pixel judgment line segment or the second pixel judgment line segment to obtain the judgment line segment perpendicular line; In the optical flow vectors of each pixel, the optical flow vectors of the pixels on the first pixel judgment line segment and the second pixel judgment line segment are selected respectively to obtain the first set of optical flow vectors and the second set of optical flow vectors. The projections of each optical flow vector in the first set of optical flow vectors and the second set of optical flow vectors onto the perpendicular line of the judgment line segment are calculated respectively to obtain the corresponding projection values. The corresponding projection values ​​are used as the optical flow velocity of the corresponding pixel, thereby obtaining the first set of optical flow velocity and the second set of optical flow velocity. The first optical flow velocity is obtained based on the first set of optical flow velocities, and the second optical flow velocity is obtained based on the second set of optical flow velocities.

2. The material blockage detection method according to claim 1, characterized in that, The step of obtaining the first optical flow velocity based on the first set of optical flow velocities and obtaining the second optical flow velocity based on the second set of optical flow velocities includes: The average optical flow velocity is obtained by averaging the optical flow velocities in the first set of optical flow velocities, and the average optical flow velocity is used as the first optical flow velocity. The average optical flow velocity is obtained by averaging the optical flow velocities in the second set of optical flow velocities, and the average optical flow velocity is used as the second optical flow velocity.

3. The material blockage detection method according to claim 1, characterized in that, The sizes of the first image and the second image are determined based on the outer boundaries of the first pixel determination region and the second pixel determination region.

4. The material blockage detection method according to claim 1, characterized in that, The step of determining whether the material is in a blocked state based on the first optical flow velocity and the second optical flow velocity includes: Determine whether the absolute value of the difference between the first optical flow velocity and the second optical flow velocity is greater than a set threshold. If the absolute value of the difference between the first optical flow velocity and the second optical flow velocity is greater than a set threshold, then the material is determined to be in a blocked state; otherwise, the material is determined not to be blocked.

5. The material blockage detection method according to claim 4, characterized in that, The set threshold is obtained using the following formula: Where theta is a set threshold. b is the minimum difference empirical value, and b is the set percentage. This is the first optical flow velocity.

6. The material blockage detection method according to claim 1, characterized in that, Before obtaining the first optical flow velocity and the second optical flow velocity, the method further includes: Obtain the optical flow vector for each pixel; The optical flow vector of each pixel is preprocessed to remove interference data; wherein, the preprocessing includes removing data with an optical flow velocity of 0, and removing data whose optical flow vector direction differs from the perpendicular line of the judgment line segment by a preset difference.

7. The material blockage detection method according to any one of claims 1 to 6, characterized in that, The RAFT algorithm is used to calculate the optical flow vector of each pixel based on the first image and the second image.

8. A terminal device, characterized in that, The terminal device includes a camera, a processor, and a memory. The camera is used to capture image information of the material falling section. The memory stores a computer program, and the processor is used to execute the computer program to implement the material blockage detection method according to any one of claims 1-7.

9. The terminal device according to claim 8, characterized in that, The terminal device is an inspection robot.

Images