Real-time detection method and system for coal flow blockage state based on monocular depth estimation

Abstract

The application provides a coal flow blockage state real-time detection method and system based on monocular depth estimation. The method comprises the following steps: collecting continuous video frame sequences of a coal conveying device and a coal flow area, and extracting a coal flow dynamic area and a static area of each frame image in the video frame sequences; inputting each frame image in the continuous video frame sequences into a pre-trained monocular depth estimation model to obtain a relative depth map of each frame image and form a relative depth map sequence; performing scale alignment on the relative depth map sequence based on depth information of the static area of each frame image to obtain a depth map sequence; extracting a depth time sequence of the coal flow dynamic area based on the depth map sequence; and judging whether the coal flow is in a blockage state according to the depth time sequence. The technical scheme provided by the application does not require additional hardware and labeled data, is suitable for various coal conveying devices, has high robustness to complex underground environments, and realizes low-cost and popularized real-time detection of coal flow blockage.

Description

Technical Field

[0001] This application relates to the fields of intelligent coal mining and computer vision technology, and in particular to a method and system for real-time detection of coal flow blockage based on monocular depth estimation. Background Technology

[0002] In the process of intelligent coal mining, real-time monitoring and early warning of anomalies in coal conveying devices such as scraper conveyors and belt conveyors are crucial. If coal flow blockage is not detected in time, it can easily lead to equipment overload, shutdown, or even major safety accidents. Therefore, developing efficient and reliable coal flow blockage detection technology has significant engineering implications.

[0003] Currently, coal flow status detection methods are mainly divided into two categories: The first is visual methods based on image processing or deep learning. These methods typically perform feature analysis on images in monitoring videos, such as edge detection, texture classification, or image recognition based on convolutional neural networks. However, uneven lighting, dust, equipment vibration, and frequent obstructions in the underground environment severely affect image quality, leading to insufficient robustness of these methods. More importantly, most existing visual methods rely on two-dimensional image information, making it difficult to effectively model the three-dimensional accumulation characteristics of coal flow. This can easily lead to misjudging large pieces of coal passing normally as blockages or missing local accumulations, resulting in high false alarm and false alarm rates. The second category is measurement methods based on physical sensors such as laser ranging, ultrasonic waves, or radar. These methods measure the distance or volume of the coal flow surface by actively transmitting and receiving signals, offering relatively high accuracy. However, their disadvantages are also significant: they require the installation of dedicated sensors, resulting in high deployment and maintenance costs; they are easily damaged in the harsh underground environment and require frequent calibration; and they are difficult to integrate with the existing widely deployed video monitoring systems in coal mines, making modification and promotion challenging. Furthermore, existing methods are mostly designed for specific conveying equipment (such as belt conveyors) and rely on the prior position of the equipment or a fixed viewing angle. However, scraper conveyors in fully mechanized mining faces undergo displacement and posture changes during operation, leading to dynamic changes in the viewing angle and coal flow distribution. This makes existing fixed-parameter detection models difficult to adapt to, resulting in poor versatility. Therefore, there is an urgent need for a real-time coal flow blockage detection solution that utilizes existing video surveillance resources in coal mines to reliably and automatically identify the coal flow blockage status of various coal conveying devices without increasing hardware costs or requiring a large amount of scene annotation data. Summary of the Invention

[0004] This application provides a method and system for real-time detection of coal flow blockage based on monocular depth estimation, in order to at least solve the technical problems of existing technologies that are difficult to balance generalization, robustness and low cost, and generally suffer from lack of three-dimensional accumulation modeling, poor adaptability to the underground environment, reliance on labeled data or dedicated sensors, and difficulty in adapting to the dynamic changes of different types of coal conveying devices.

[0005] The first aspect of this application proposes a real-time detection method for coal flow blockage state based on monocular depth estimation, the method comprising: A continuous video frame sequence of the coal conveying device and the coal flow area is acquired using a monocular camera arranged in the coal conveying device area, and the dynamic and static areas of the coal flow in each frame of the video frame sequence are extracted. Each frame of the continuous video frame sequence is input into a pre-trained monocular depth estimation model to obtain the relative depth map of each frame, and a relative depth map sequence is constructed based on the relative depth maps of each frame. The relative depth map sequence is scale-aligned based on the depth information of the static regions of each frame image to obtain a depth map sequence with consistent scale. Based on the scale-consistent depth map sequence, the depth time series sequence of the dynamic region of coal flow is extracted; The depth time sequence is used to determine whether the coal flow is blocked.

[0006] Preferably, the extraction of the dynamic and static regions of coal flow in each frame of the video frame sequence includes: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

[0007] Furthermore, the process of scaling the relative depth map sequence based on the depth information of static regions in each frame image to obtain a scale-consistent depth map sequence includes: The first frame of the continuous video frame sequence is used as the reference frame, and the depth values ​​of the static regions of each frame image and the depth values ​​of the static regions of the reference frame are extracted based on the relative depth maps of each frame image. The scale factor of each frame image is determined based on the depth value of the static region of the reference frame and the depth value of the static region of each frame image, respectively. The relative depth maps in the relative depth map sequence are scaled according to the scale factor of each frame image to obtain the relative depth map of each frame image, and a depth map sequence with consistent scale is constructed based on the relative depth maps of each frame image.

[0008] Furthermore, the extraction of the depth time series sequence of the dynamic region of coal flow based on the scale-consistent depth map sequence includes: Obtain the relative depth values ​​of pixels in the dynamic region of coal flow in each frame of the image; The average depth value of the dynamic region of the coal flow in each frame image is determined based on the dynamic region mask of the coal flow and the relative depth value of the pixels of the dynamic region of the coal flow in each frame image. The depth time series is constructed based on the average depth value of the dynamic region of coal flow in each frame of the image.

[0009] Furthermore, the formula for calculating the average depth value of the dynamic region of coal flow in each frame image is as follows:

[0010] In the formula, Let be the average depth value of the image at time t. Let be the relative depth value of the pixel in the i-th row and j-th column of the relative depth values ​​of the dynamic region of the coal flow in the image at time t. This is the dynamic region mask for the coal flow of the pixel corresponding to the relative depth value in the i-th row and j-th column of the relative depth values ​​of the pixels. This is the dynamic region mask for the coal flow.

[0011] Furthermore, determining whether the coal flow is blocked based on the depth time sequence includes: Linear regression is performed on the depth time series to calculate the slope of the trend of the depth time series changing over time; When the slope of the trend is greater than a preset slope threshold, the coal flow is determined to be in the clearing process state; When the slope of the trend is less than the negative value of the preset slope threshold, it is determined that the coal flow is in a blocked or accumulated state. When the trend slope is greater than or equal to the negative value of the preset slope threshold and less than or equal to the preset slope threshold, the coal flow is determined to be in a normal conveying state. Wherein, the preset slope threshold is greater than zero; An early warning is issued when the coal flow is blocked or accumulated.

[0012] A second aspect of this application proposes a real-time coal flow blockage detection system based on monocular depth estimation, comprising: The acquisition module is used to acquire a continuous video frame sequence of the coal conveying device and the coal flow area using a monocular camera arranged in the coal conveying device area, and to extract the dynamic and static areas of the coal flow in each frame of the video frame sequence. The first construction module is used to input each frame image in the continuous video frame sequence into a pre-trained monocular depth estimation model to obtain the relative depth map of each frame image, and to construct a relative depth map sequence based on the relative depth maps of each frame image. An alignment module is used to perform scale alignment on the relative depth map sequence based on the depth information of the static regions of each frame image to obtain a depth map sequence with consistent scale. The extraction module is used to extract the depth time series sequence of the coal flow dynamic region based on the depth map sequence with consistent scale; The judgment module is used to determine whether the coal flow is blocked based on the depth time sequence.

[0013] Furthermore, the acquisition module is also used for: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

[0014] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method described in the first aspect embodiment.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0016] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: This application proposes a method and system for real-time detection of coal flow blockage based on monocular depth estimation. The method includes: acquiring a continuous video frame sequence of the coal conveying device and the coal flow area using a monocular camera deployed in the coal conveying device area, and extracting the dynamic and static regions of the coal flow in each frame of the video frame sequence; inputting each frame of the continuous video frame sequence into a pre-trained monocular depth estimation model to obtain a relative depth map of each frame, and constructing a relative depth map sequence based on the relative depth maps of each frame; performing scale alignment on the relative depth map sequence based on the depth information of the static region of each frame to obtain a scale-consistent depth map sequence; extracting a depth time-series sequence of the dynamic region of the coal flow based on the scale-consistent depth map sequence; and determining whether the coal flow is in a blockage state based on the depth time-series sequence. The technical solution proposed in this application requires no additional hardware or labeled data, is adaptable to various coal conveying devices, has high robustness to complex underground environments, and achieves low-cost, scalable real-time detection of coal flow blockage.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a real-time detection method for coal flow blockage state based on monocular depth estimation according to an embodiment of this application; Figure 2 This is a structural diagram of a real-time coal flow blockage detection system based on monocular depth estimation, according to an embodiment of this application. Detailed Implementation

[0019] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0020] This application proposes a method and system for real-time detection of coal flow blockage based on monocular depth estimation. The method includes: acquiring a continuous video frame sequence of the coal conveying device and the coal flow area using a monocular camera deployed in the coal conveying device area, and extracting the dynamic and static regions of the coal flow in each frame of the video frame sequence; inputting each frame of the continuous video frame sequence into a pre-trained monocular depth estimation model to obtain a relative depth map of each frame, and constructing a relative depth map sequence based on the relative depth maps of each frame; performing scale alignment on the relative depth map sequence based on the depth information of the static region of each frame to obtain a scale-consistent depth map sequence; extracting a depth time-series sequence of the dynamic region of the coal flow based on the scale-consistent depth map sequence; and determining whether the coal flow is in a blockage state based on the depth time-series sequence. The technical solution proposed in this application requires no additional hardware or labeled data, is adaptable to various coal conveying devices, has high robustness to complex underground environments, and achieves low-cost, scalable real-time detection of coal flow blockage.

[0021] The following describes, with reference to the accompanying drawings, a method and system for real-time detection of coal flow blockage based on monocular depth estimation, according to embodiments of this application.

[0022] Example 1 Figure 1 This is a flowchart illustrating a real-time detection method for coal flow blockage state based on monocular depth estimation according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes: Step 1: Use a monocular camera placed in the coal conveying device area to collect a continuous video frame sequence of the coal conveying device and the coal flow area, and extract the dynamic and static areas of the coal flow in each frame of the video frame sequence. In this embodiment of the disclosure, the extraction of the dynamic and static regions of coal flow in each frame of the video frame sequence includes: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

[0023] It should be noted that, to extract the dynamic and static regions of the coal flow, known coal flow region identification methods (such as optical flow detection, semantic segmentation, or motion region detection) are used to generate two types of binary masks from each frame of the image: The dynamic region mask for coal flow is denoted as Pixels with a value of 1 belong to the coal flow; The static region mask is denoted as This corresponds to structures that do not change over time, such as conveyor frames and tunnel walls. This step does not rely on a fixed position and can adapt to the dynamic displacement of the coal conveying device in the fully mechanized mining face.

[0024] Step 2: Input each frame image in the continuous video frame sequence into the pre-trained monocular depth estimation model to obtain the relative depth map of each frame image, and construct a relative depth map sequence based on the relative depth maps of each frame image; It should be noted that each frame of the image The input is a pre-trained monocular depth estimation model, which is the general monocular depth estimation base model. Output the corresponding relative depth map: in This is a pre-trained monocular depth estimation base model that does not require fine-tuning for coal mine scenarios.

[0025] Step 3: Based on the depth information of the static regions of each frame image, scale-align the relative depth map sequence to obtain a depth map sequence with consistent scale; It should be noted that, based on the depth information of relatively stable static regions in each frame of the image, the relative depth map sequence is scale-aligned so that the depth maps of different frames maintain relative consistency under a unified scale reference, thereby obtaining a scale-aligned depth map sequence.

[0026] In this embodiment of the disclosure, step 3 specifically includes: The first frame of the continuous video frame sequence is used as the reference frame, and the depth values ​​of the static regions of each frame image and the depth values ​​of the static regions of the reference frame are extracted based on the relative depth maps of each frame image. The scale factor of each frame image is determined based on the depth value of the static region of the reference frame and the depth value of the static region of each frame image, respectively. The relative depth maps in the relative depth map sequence are scaled according to the scale factor of each frame image to obtain the relative depth map of each frame image, and a depth map sequence with consistent scale is constructed based on the relative depth maps of each frame image.

[0027] The scale factor of each frame image can be calculated using the following formula:

[0028] In the formula, Let be the scale factor of the image at time t. The depth value of the static region of the reference frame. This is the static region mask value. Let t be the depth value of the static region of the image at time t.

[0029] It should be noted that the method for obtaining the scale factor between different frames is not limited to this one.

[0030] It should be noted that because the model outputs relative depth, there is a scale inconsistency issue between different frames. Taking the first frame of a video within a certain time period as a baseline, and spanning T frames, scale normalization is performed on the entire depth sequence. Specifically, for each frame... Using static region mask Extract the depth value and calculate its scale factor relative to the reference frame: Thus, the scale factor sequence is obtained. .

[0031] For the original depth sequence Scale the image frame by frame to obtain the scale-aligned depth sequence: .

[0032] Step 4: Extract the depth time series sequence of the coal flow dynamic region based on the scale-consistent depth map sequence; In this embodiment of the disclosure, step 4 specifically includes: Obtain the relative depth values ​​of pixels in the dynamic region of coal flow in each frame of the image; The average depth value of the dynamic region of the coal flow in each frame image is determined based on the dynamic region mask of the coal flow and the relative depth value of the pixels of the dynamic region of the coal flow in each frame image. The depth time series is constructed based on the average depth value of the dynamic region of coal flow in each frame of the image.

[0033] The formula for calculating the average depth of the dynamic region of coal flow in each frame image is as follows:

[0034] In the formula, Let be the average depth value of the image at time t. Let be the relative depth value of the pixel in the i-th row and j-th column of the relative depth values ​​of the dynamic region of the coal flow in the image at time t. This is the dynamic region mask for the coal flow of the pixel corresponding to the relative depth value in the i-th row and j-th column of the relative depth values ​​of the pixels. This is the dynamic region mask for the coal flow.

[0035] It should be noted that in the aligned depth map In this process, the average depth value of the dynamic region of the coal flow is extracted for each frame, forming a depth time series that characterizes the change in the distance between the coal flow and the camera.

[0036] Specifically, for each frame ,calculate: Thus, a deep time series sequence is obtained. .

[0037] Step 5: Determine whether the coal flow is blocked based on the depth time sequence.

[0038] In this embodiment of the disclosure, step 5 specifically includes: Linear regression is performed on the depth time series to calculate the slope of the trend of the depth time series changing over time; When the slope of the trend is greater than a preset slope threshold, the coal flow is determined to be in the clearing process state; When the slope of the trend is less than the negative value of the preset slope threshold, it is determined that the coal flow is in a blocked or accumulated state. When the trend slope is greater than or equal to the negative value of the preset slope threshold and less than or equal to the preset slope threshold, the coal flow is determined to be in a normal conveying state. Wherein, the preset slope threshold is greater than zero; An early warning is issued when the coal flow is blocked or accumulated.

[0039] It should be noted that for deep time series... Perform a univariate linear regression to fit its overall trend. Let the time mean be... The average depth is Then the trend slope for: ; Set threshold According to the slope Determine the state of the coal flow: like The situation is determined to be a blockage or accumulation (coal flow is approaching the camera); like It was determined to be a dredging process; Otherwise, it is determined to be in normal transport status. When a congestion is detected, the system automatically triggers a warning signal.

[0040] The method proposed in this embodiment has the following advantages: 1. No additional hardware required: Utilizes only existing monocular video surveillance systems, resulting in low deployment costs; 2. High versatility: Applicable to all types of coal conveying equipment, regardless of equipment type or installation location; 3. High robustness: By modeling coal flow accumulation with deep geometric information, it has strong resistance to interference such as dust and obstruction; 4. Zero-shot adaptation: It adopts a general monocular depth estimation base model, which does not require labeled data for coal mine scenes; In summary, the real-time detection method for coal flow blockage based on monocular depth estimation proposed in this embodiment does not require differentiation of coal conveying device type, does not rely on prior equipment location, and does not require additional sensors or a large amount of labeled data. It can achieve highly robust, low-cost, and scalable automatic identification of blockage status by simply extracting the relative depth temporal change of the coal flow area from monocular video using a general depth estimation model.

[0041] Example 2 Figure 2 This is a structural diagram of a real-time coal flow blockage detection system based on monocular depth estimation according to an embodiment of this application, as shown below. Figure 2 As shown, the system includes: The acquisition module 100 is used to acquire a continuous video frame sequence of the coal conveying device and the coal flow area using a monocular camera arranged in the coal conveying device area, and to extract the dynamic and static areas of the coal flow in each frame of the video frame sequence. The first construction module 200 is used to input each frame image in the continuous video frame sequence into a pre-trained monocular depth estimation model to obtain the relative depth map of each frame image, and to construct a relative depth map sequence based on the relative depth maps of each frame image. Alignment module 300 is used to perform scale alignment on the relative depth map sequence based on the depth information of the static regions of each frame image to obtain a depth map sequence with consistent scale. Extraction module 400 is used to extract the depth time series sequence of the coal flow dynamic region based on the depth map sequence with consistent scale; The judgment module 500 is used to determine whether the coal flow is in a blocked state based on the depth time sequence.

[0042] In this embodiment of the disclosure, the acquisition module 100 is further configured to: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

[0043] In this embodiment of the disclosure, the alignment module 300 is further configured to: take the first frame of the continuous video frame sequence as a reference frame, and extract the depth value of the static region of each frame image and the depth value of the static region of the reference frame based on the relative depth map of each frame image; The scale factor of each frame image is determined based on the depth value of the static region of the reference frame and the depth value of the static region of each frame image, respectively. The relative depth maps in the relative depth map sequence are scaled according to the scale factor of each frame image to obtain the relative depth map of each frame image, and a depth map sequence with consistent scale is constructed based on the relative depth maps of each frame image.

[0044] The scale factor of each frame image is calculated using the following formula:

[0045] In the formula, Let be the scale factor of the image at time t. The depth value of the static region of the reference frame. The static region mask value of the reference frame. This is the static region mask value. Let t be the depth value of the static region of the image at time t.

[0046] In this embodiment of the disclosure, the extraction module 400 is further configured to: Obtain the relative depth values ​​of pixels in the dynamic region of coal flow in each frame of the image; The average depth value of the dynamic region of the coal flow in each frame image is determined based on the dynamic region mask of the coal flow and the relative depth value of the pixels of the dynamic region of the coal flow in each frame image. The depth time series is constructed based on the average depth value of the dynamic region of coal flow in each frame of the image.

[0047] The formula for calculating the average depth of the dynamic region of coal flow in each frame image is as follows:

[0048] In the formula, Let be the average depth value of the image at time t. Let be the relative depth value of the pixel in the i-th row and j-th column of the relative depth values ​​of the dynamic region of the coal flow in the image at time t. This is the dynamic region mask for the coal flow of the pixel corresponding to the relative depth value in the i-th row and j-th column of the relative depth values ​​of the pixels. This is the dynamic region mask for the coal flow.

[0049] In this embodiment of the disclosure, the determination module 500 is further configured to: Linear regression is performed on the depth time series to calculate the slope of the trend of the depth time series changing over time; When the slope of the trend is greater than a preset slope threshold, the coal flow is determined to be in the clearing process state; When the slope of the trend is less than the negative value of the preset slope threshold, it is determined that the coal flow is in a blocked or accumulated state. When the trend slope is greater than or equal to the negative value of the preset slope threshold and less than or equal to the preset slope threshold, the coal flow is determined to be in a normal conveying state. Wherein, the preset slope threshold is greater than zero; An early warning is issued when the coal flow is blocked or accumulated.

[0050] In summary, the real-time coal flow blockage detection system proposed in this embodiment does not require differentiation of coal conveying device type, does not rely on prior equipment location, and does not require additional sensors or a large amount of labeled data. It can achieve highly robust, low-cost, and scalable automatic identification of blockage status by simply extracting the relative depth temporal changes of the coal flow area from monocular video using a general depth estimation model.

[0051] Example 3 To implement the above embodiments, this disclosure also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method described in Embodiment 1.

[0052] Example 4 To implement the above embodiments, this disclosure also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0053] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0054] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0055] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for real-time detection of coal flow blockage state based on monocular depth estimation, characterized in that, The method includes: A continuous video frame sequence of the coal conveying device and the coal flow area is acquired using a monocular camera arranged in the coal conveying device area, and the dynamic and static areas of the coal flow in each frame of the video frame sequence are extracted. Each frame of the continuous video frame sequence is input into a pre-trained monocular depth estimation model to obtain the relative depth map of each frame, and a relative depth map sequence is constructed based on the relative depth maps of each frame. The relative depth map sequence is scale-aligned based on the depth information of the static regions of each frame image to obtain a depth map sequence with consistent scale. Based on the scale-consistent depth map sequence, the depth time series sequence of the dynamic region of coal flow is extracted; The depth time sequence is used to determine whether the coal flow is blocked.

2. The method as described in claim 1, characterized in that, The extraction of the dynamic and static regions of coal flow from each frame in the video frame sequence includes: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

3. The method as described in claim 2, characterized in that, The relative depth map sequence is scale-aligned based on the depth information of static regions in each frame image to obtain a scale-consistent depth map sequence, including: The first frame of the continuous video frame sequence is used as the reference frame, and the depth values ​​of the static regions of each frame image and the depth values ​​of the static regions of the reference frame are extracted based on the relative depth maps of each frame image. The scale factor of each frame image is determined based on the depth value of the static region of the reference frame and the depth value of the static region of each frame image, respectively. The relative depth maps in the relative depth map sequence are scaled according to the scale factor of each frame image to obtain the relative depth map of each frame image, and a depth map sequence with consistent scale is constructed based on the relative depth maps of each frame image.

4. The method as described in claim 3, characterized in that, The extraction of the depth time series sequence of the dynamic region of coal flow based on the scale-consistent depth map sequence includes: Obtain the relative depth values ​​of pixels in the dynamic region of coal flow in each frame of the image; The average depth value of the dynamic region of the coal flow in each frame image is determined based on the dynamic region mask of the coal flow and the relative depth value of the pixels of the dynamic region of the coal flow in each frame image. The depth time series is constructed based on the average depth value of the dynamic region of coal flow in each frame of the image.

5. The method as described in claim 4, characterized in that, The formula for calculating the average depth of the dynamic region of the coal flow in each frame image is as follows: In the formula, Let be the average depth value of the image at time t. Let be the relative depth value of the pixel in the i-th row and j-th column of the relative depth values ​​of the dynamic region of the coal flow in the image at time t. This is the dynamic region mask for the coal flow of the pixel corresponding to the relative depth value in the i-th row and j-th column of the relative depth values ​​of the pixels. This is the dynamic region mask for the coal flow.

6. The method as described in claim 5, characterized in that, The step of determining whether the coal flow is blocked based on the depth time series includes: Linear regression is performed on the depth time series to calculate the slope of the trend of the depth time series changing over time; When the slope of the trend is greater than a preset slope threshold, the coal flow is determined to be in the clearing process state; When the slope of the trend is less than the negative value of the preset slope threshold, it is determined that the coal flow is in a blocked or accumulated state. When the trend slope is greater than or equal to the negative value of the preset slope threshold and less than or equal to the preset slope threshold, the coal flow is determined to be in a normal conveying state. Wherein, the preset slope threshold is greater than zero; An early warning is issued when the coal flow is blocked or accumulated.

7. A real-time detection system for coal flow blockage status based on monocular depth estimation, characterized in that, The system includes: The acquisition module is used to acquire a continuous video frame sequence of the coal conveying device and the coal flow area using a monocular camera arranged in the coal conveying device area, and to extract the dynamic and static areas of the coal flow in each frame of the video frame sequence. The first construction module is used to input each frame image in the continuous video frame sequence into a pre-trained monocular depth estimation model to obtain the relative depth map of each frame image, and to construct a relative depth map sequence based on the relative depth maps of each frame image. An alignment module is used to perform scale alignment on the relative depth map sequence based on the depth information of the static regions of each frame image to obtain a depth map sequence with consistent scale. The extraction module is used to extract the depth time series sequence of the coal flow dynamic region based on the depth map sequence with consistent scale; The judgment module is used to determine whether the coal flow is blocked based on the depth time sequence.

8. The system as described in claim 7, characterized in that, The acquisition module is also used for: A coal flow region identification method is used to identify each frame of the image, generating dynamic and static region masks for the coal flow. The dynamic region mask of the coal flow is used to identify the moving coal flow portion in the image, and the static region mask is used to identify the background structure portion in the image that does not change over time. The coal flow region identification algorithm includes: optical flow detection method, semantic segmentation method, or motion region detection method.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images