Belt volume analysis method and device based on image recognition

Abstract

The invention discloses a belt volume analysis method and device based on image recognition, and the method comprises the steps: marking two labels, namely a coal label and a belt label, in an image collected by a coal flow belt video through a marking tool; training the marked image by adopting an instance segmentation algorithm to obtain an instance segmentation model for identifying the coal flow; marking a line segment on an image acquired by a coal flow belt video, calculating a pixel length and a real physical length of the line segment, determining a unit pixel length, carrying out perspective restoration on the belt, calculating a zoom ratio at a longitudinal pixel, and obtaining a coal flow width; and performing instance segmentation on the video frame through the instance segmentation model, calculating the mass of the coal in unit time by combining the frame time difference according to the measured average coal density, the estimated coal flow speed and the estimated coal flow sectional area, and performing accumulation to obtain the preset granularity coal quantity. According to the invention, non-contact traffic volume estimation is realized through image recognition, artificial counterfeiting is avoided, and the data accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of coal quantity analysis, and particularly relates to a belt conveying quantity analysis method and device based on image recognition. BACKGROUND

[0002] In the process of coal mine production and operation, coal mining enterprises generally exist in the situation of concealing or underreporting raw coal and gross coal production in order to improve profits. The real production data of some coal mines are derived from the advancing footage reported by each shift, and the production value is calculated. This kind of artificial reporting and calculation mode makes it difficult for managers and supervision units at all levels to obtain the real production data of coal mining enterprises and accurately control the production situation.

[0003] At present, the main ways to obtain coal production in the industry are enterprise self-reporting and belt scale measurement. However, both of these two ways have obvious disadvantages: the data reported by enterprises have subjective motives for fraud, and there is a lack of objective and reliable verification methods; belt scale measurement is a contact type measurement, and the equipment is easy to be modified or disturbed by human, with extremely low fraud cost, and the measurement error will accumulate with the use time, which cannot be used as an effective basis for truly reflecting the coal production. These two ways together lead to a significant supervision blind spot in the process of coal production supervision. SUMMARY

[0004] Therefore, the present application provides a belt conveying quantity analysis method and device based on image recognition to solve the problem of low fraud cost and data distortion in the existing coal production acquisition method, and the existence of a supervision blind spot.

[0005] In order to achieve the above purpose, the present application provides the following technical scheme: a belt conveying quantity analysis method based on image recognition, comprising the following steps:

[0006] Using a labeling tool to label two labels of "coal" and "belt" in the image collected by the coal flow belt video;

[0007] Using an instance segmentation algorithm to train the labeled image to obtain an instance segmentation model for identifying coal flow;

[0008] Marking a line segment on the image collected by the coal flow belt video, calculating the pixel length and real physical length of the line segment, determining the unit pixel length, and performing perspective restoration on the belt to calculate the scaling ratio at the vertical pixel to obtain the coal flow width;

[0009] Classifying the coal through the classification model, and determining the natural angle of repose according to the coal flow width and the classification result;

[0010] Approximating the cross-sectional area of the coal flow as a trapezoid and an arc, and calculating the cross-sectional area of the coal flow in combination with the natural angle of repose;

[0011] The instance segmentation model is used for instance segmentation on the video frame, and the mass of the coal in a unit of time is calculated according to the measured average density of the coal, the estimated speed of the coal flow and the cross-sectional area of the coal flow, combined with the frame time difference, and the preset granularity of the coal flow is accumulated to obtain the coal flow.

[0012] As an optimal solution of the belt capacity analysis method based on image recognition, when labeling the two labels of "coal" and "belt", the following operations are performed on the coal flow belt video frame image using the labeling tool:

[0013] Pixel-level contour labeling is performed on the edge of the coal flow area and the edge of the belt body.

[0014] A mutually exclusive label system is established, and the labeled pixel set is divided into two non-intersecting "coal" and "belt" categories, wherein the "belt" label is used as a negative sample set of the "coal" label.

[0015] As an optimal solution of the belt capacity analysis method based on image recognition, the labeled image is trained using an instance segmentation algorithm to obtain an instance segmentation model for identifying the coal flow, specifically including:

[0016] A training sample containing the coal flow target bounding box and mask is constructed using the labeled image dataset;

[0017] Based on the Anchor-Free architecture of YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, a deep learning model with pixel-level segmentation capability of the coal flow area is trained as an instance segmentation model by optimizing the CIoU loss function and the binary cross-entropy loss function of the mask branch, to realize instance-level recognition and boundary positioning of the coal flow target in the belt video image.

[0018] As an optimal solution of the belt capacity analysis method based on image recognition, when calculating the width of the coal flow, a line segment EF is marked on the image collected by the coal flow belt video, and the pixel length l k and the real physical length l m of the line segment EF are calculated, to obtain the unit pixel length l m / l k .

[0019] The four coordinates of the belt edge are labeled to perform perspective restoration, calculate the scaling ratio r i at the vertical pixel y i , and obtain the real length l (i) of the coal flow = r i ×dx i ×l m / l k ; wherein dx i is the pixel length of the coal at the vertical pixel y i .

[0020] As a preferred scheme of the image recognition-based belt load analysis method, the coal flow cross-sectional area is approximated as a trapezoid and an arc, and the natural angle of repose is used to calculate the cross-sectional area of the coal flow:

[0021] The area of the trapezoidal part is calculated by the difference between the areas of two isosceles triangles, and the formula is:

[0022]

[0023]

[0024] wherein S (l) is the area of an isosceles triangle with the coal flow width l as the base and the angle α between the rollers as the base angle; is the area of an isosceles triangle with the corrected width l b as the base and the angle α between the rollers as the base angle; l is the coal flow width identified by image recognition, α is the angle between the rollers and the horizontal direction, l m is the standard width of the belt corresponding to the top of the rollers, and h is the height from the top of the rollers to the bottom of the belt;

[0025] The area of the arc part is calculated by the difference between the area of a sector and the area of a triangle, and the formula is:

[0026]

[0027] The cross-sectional area of the coal flow is:

[0028] S = S 梯形 + S 弓形

[0029] wherein S 扇形 is the area of a sector corresponding to the arc part of the coal flow, which is the minuend when calculating the area of the arc; S Δ represents the area of a triangle corresponding to the arc part of the coal flow, which is the subtrahend when calculating the area of the arc; β is the natural angle of repose; and S represents the cross-sectional area of the coal flow.

[0030] The preset rule of the natural angle of repose β is:

[0031] When l ≤ 0.2l m and the coal type is “low”, β = 20°;

[0032] When 0.2l m < l < 0.8l m and the coal type is “low”, β = 30°;

[0033] When l ≥ 0.8l m or the coal type is “high”, β = 40°.

[0034] As an optimal scheme of the belt load analysis method based on image recognition, a formula for calculating the mass of coal per unit time in combination with the frame time difference is:

[0035] M=S*Delta t*v*rho

[0036] In the formula, Delta t is the time difference between the current frame and the previous frame, v is the coal flow speed estimated by the feature point detection algorithm, and rho is the average density of the coal measured.

[0037] The application also provides a belt load analysis device based on image recognition, comprising:

[0038] A label labeling module is configured to label two labels of "coal" and "belt" in the image collected by the coal flow belt video using a labeling tool.

[0039] A model training module is configured to train the labeled image using an instance segmentation algorithm to obtain an instance segmentation model for identifying the coal flow.

[0040] A width calculation module is configured to mark a line segment on the image collected by the coal flow belt video, calculate the pixel length and the real physical length of the line segment, determine the unit pixel length, perform perspective restoration on the belt, calculate the scaling ratio at the longitudinal pixel, and obtain the coal flow width.

[0041] A classification and parameter determination module is configured to classify the coal by the classification model, and determine the natural angle of repose according to the coal flow width and the classification result.

[0042] A cross-sectional area calculation module is configured to approximate the cross-sectional area of the coal flow as a trapezoid and an arc, and calculate the cross-sectional area of the coal flow in combination with the natural angle of repose.

[0043] A load calculation module is configured to perform instance segmentation on the video frame by the instance segmentation model, estimate the mass of coal per unit time in combination with the frame time difference according to the measured average density of the coal, the estimated coal flow speed, and the cross-sectional area of the coal flow, and accumulate the coal quantity of a preset granularity.

[0044] As an optimal scheme of the belt load analysis device based on image recognition, in the label labeling module:

[0045] The pixel-level contour labeling is performed on the edges of the coal flow area and the edges of the belt body.

[0046] A mutual exclusion label system is established, and the labeled pixel set is divided into two types of "coal" and "belt" which are disjointed, wherein the "belt" label is used as the negative sample set of the "coal" label.

[0047] As an optimal scheme of the belt load analysis device based on image recognition, in the model training module:

[0048] Use the labeled image dataset to construct training samples containing coal flow target bounding boxes and masks;

[0049] Based on the Anchor-Free architecture of the YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, by optimizing the CIoU loss function and the binary cross entropy loss function of the mask branch, a deep learning model with the ability to segment the coal flow area at the pixel level is trained as an instance segmentation model to perform instance-level recognition and boundary positioning of coal flow targets in belt video images.

[0050] As a preferred solution of the belt transport volume analysis device based on image recognition, in the width calculation module:

[0051] By marking the line segment EF on the image collected by the coal flow belt video, the pixel length l of the line segment EF is calculated. k and the real physical length l m , the unit pixel length is l m / l k ;

[0052] And restore the perspective by marking the four coordinates of the belt edge, and calculate the vertical pixel y i The scaling ratio r i ,, get the real length of coal flow l (i) =r i ×dx i ×l m / l k ; Among them, dx i is the vertical pixel y i The pixel length of the coal.

[0053] As a preferred solution of the belt transport volume analysis device based on image recognition, in the cross-sectional area calculation module:

[0054] The area of ​​the trapezoidal part is calculated by the difference in the areas of the two isosceles triangles, using the formula:

[0055]

[0056] Where S (l) is the area of ​​an isosceles triangle with the coal flow width l as the base and the roller angle α as the base angle; To correct the width l b is the area of ​​an isosceles triangle with the base side and the roller angle α as the base angle; l is the coal flow width identified by the image, α is the angle between the roller and the horizontal direction, l m is the standard width of the belt corresponding to the top of the roller, and h is the height from the top of the roller to the bottom of the belt;

[0057] The area of ​​the arcuate part is calculated by the difference between the area of ​​the sector and the area of ​​the triangle, and the formula is:

[0058]

[0059] The cross-sectional area of the coal flow is:

[0060] S = S 梯形 + S 弓形

[0061] In the formula, S 扇形 refers to the sector area corresponding to the arc-shaped part of the coal flow, which is used as the minuend when calculating the arc-shaped area; S Δ represents the triangular area corresponding to the arc-shaped part of the coal flow, which is used as the subtrahend when calculating the arc-shaped area; β is the natural angle of repose; and S represents the cross-sectional area of the coal flow.

[0062] In the classification and parameter determination module, the preset rule of the natural angle of repose β is:

[0063] When l≤0.2l m , and the coal type is "low", β = 20°.

[0064] When 0.2l m < l < 0.8l m , and the coal type is "low", β = 30°.

[0065] When l≥0.8l m , or the coal type is "high", β = 40°.

[0066] In the transport volume calculation module, the mass of coal in unit time is calculated according to the time difference between frames, and the formula is:

[0067] M = S * Δt * v * ρ

[0068] In the formula, Δt is the time difference between the current frame and the previous frame, v is the estimated speed of the coal flow through the feature point detection algorithm, and ρ is the average density of the coal measured.

[0069] The beneficial effects of the present application are as follows:

[0070] First, by collecting images and combining image recognition technology to estimate the transport volume, the defect that the traditional belt scale contact measurement is easily modified by human is avoided, the possibility of data falsification is eliminated from a technical means, and the yield data is ensured to be real and reliable. Without complex weighing equipment, only a camera and image algorithm are used to realize transport volume analysis, the equipment investment and maintenance cost are low, it is suitable for large-scale deployment in coal mine sites, and the economic efficiency of the scheme is improved.

[0071] Second, through the instance segmentation model, the coal flow edge is accurately identified, the perspective restoration and the geometric modeling are combined to calculate the cross-sectional area, in the pilot, the hourly accuracy is more than 80%, the daily accuracy is more than 90%, and reliable data support is provided for supervision. The video frame is calculated in real time, and the coal quantity is accumulated, so that the minute, hour and other granularities are realized, the lag of manual reporting is replaced, and the digital upgrading of production management is assisted.

[0072] Third, through the design of mutually exclusive label training model and roller position fixed measurement, the influence of interference factors such as belt deviation and perspective distortion is reduced, and the measurement stability in complex industrial environment is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be derived from the provided drawings without creative labor.

[0074] The structures, proportions, sizes and the like shown in the specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the limiting conditions of the embodiments of the present application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose of the present application, should still fall within the scope of the technical content disclosed by the present application.

[0075] Figure 1 The image recognition-based belt load analysis method flowchart provided by the embodiment of the present application is shown in the figure;

[0076] Figure 2 The "coal" and "belt" operation interface provided by the embodiment of the present application is shown in the figure;

[0077] Figure 3 The instance segmentation model for identifying coal flow provided by the embodiment of the present application is shown in the figure;

[0078] Figure 4 The coal flow width calculation interface provided by the embodiment of the present application is shown in the figure;

[0079] Figure 5 The perspective restoration diagram of marking the coordinates of the four points of the belt edge on the figure provided by the embodiment of the present application is shown in the figure;

[0080] Figure 6 The cross-sectional area estimation diagram provided by the embodiment of the present application is shown in the figure;

[0081] Figure 7A trapezoidal part area calculation schematic diagram provided for an embodiment of the present application is shown in FIG. 1.

[0082] Figure 8 An arcuate part area calculation schematic diagram provided for an embodiment of the present application is shown in FIG. 2.

[0083] Figure 9 A classification model classification interface schematic diagram provided for an embodiment of the present application is shown in FIG. 3.

[0084] Figure 10 A belt carrying capacity analysis device architecture schematic diagram based on image recognition provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0085] The embodiments of the present application will be described in detail by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0086] Embodiment 1

[0087] Referring to Figure 1 , the embodiment 1 of the present application provides a belt carrying capacity analysis method based on image recognition, comprising the following steps:

[0088] S1, using a labeling tool to label two labels of "coal" and "belt" in the image collected by the coal flow belt video; the region boundary of the coal flow and the belt in the image is determined by manual labeling, and accurate sample data is provided for subsequent model training. The labeled label is used as supervision information to guide the model to learn the characteristics of the coal flow and the belt, so that the model can accurately distinguish between the two in subsequent image recognition;

[0089] S2, training the labeled image using an instance segmentation algorithm to obtain an instance segmentation model for identifying the coal flow; the instance segmentation algorithm can accurately perform pixel-level segmentation on each target instance in the image. By training the labeled image, the instance segmentation algorithm learns the shape, texture and other characteristics of the coal flow, so that the trained model can accurately identify the coal flow instance in the image and segment its contour;

[0090] S3, marking a line segment on the image collected by the coal flow belt video, calculating the pixel length and the real physical length of the line segment, determining the unit pixel length, and performing perspective restoration on the belt to calculate the scaling ratio at the longitudinal pixel to obtain the coal flow width; due to the perspective distortion in camera shooting, the pixel length of the object in the image and the real physical length are not a simple linear relationship. By marking a line segment with a known real physical length, the pixel length can be calculated to obtain the real length represented by the unit pixel. Perspective restoration is performed on the belt to eliminate the influence of perspective distortion, so that the pixels at each position in the image can accurately reflect the real physical size, and then the pixel width of the coal flow in the image is converted into the real physical width by calculating the scaling ratio at the longitudinal pixel.

[0091] S4, classifying the coal through a classification model, and determining the natural angle of repose according to the coal flow width and the classification result; different coal accumulation states have different natural angles of repose. The classification model classifies the accumulation state of the coal, and in combination with the coal flow width, the appropriate natural angle of repose parameter can be determined. The natural angle of repose is used for subsequent calculation of the arcuate part in the coal flow cross-sectional area, and different natural angles of repose will affect the shape and area of the arcuate part, thereby affecting the calculation of the coal flow cross-sectional area.

[0092] S5, approximating the coal flow cross-sectional area as a trapezoid and an arcuate shape, and calculating the coal flow cross-sectional area in combination with the natural angle of repose; the cross-sectional shape of the coal flow on the belt can be approximately regarded as being composed of a lower trapezoid and an upper arcuate shape. The trapezoidal part is formed due to the supporting action of the roller, and the arcuate part is formed due to the natural accumulation of the coal. In combination with the natural angle of repose and other parameters, the areas of the trapezoid and the arcuate shape are calculated through corresponding geometric formulas, and the sum of the two areas is the cross-sectional area of the coal flow.

[0093] S6, performing instance segmentation on the video frame through the instance segmentation model, calculating the mass of coal in a unit time in combination with the frame time difference according to the measured average density of the coal, the estimated speed of the coal flow and the cross-sectional area of the coal flow, and accumulating to obtain the coal quantity of a preset granularity; according to the physical formula, the mass is equal to the volume multiplied by the density. The volume of the coal flow can be obtained by multiplying the cross-sectional area by the speed and then by the time. Through the instance segmentation model, the cross-sectional area of the coal flow is obtained, and in combination with the estimated speed of the coal flow and the frame time difference, the volume of coal in a unit time can be calculated, and then multiplied by the average density of the coal to obtain the mass of coal in a unit time. The masses of coal in each unit time are accumulated to obtain the coal quantity of a preset granularity (such as minutes, hours, days, etc.).

[0094] Reference Figure 2 In the embodiment, in step S1, when labeling the two labels of “coal” and “belt”, the labeling tool is used to perform the following operations on the coal flow belt video frame image:

[0095] The pixel-level contour labeling is performed on the edge of the coal flow area and the edge of the belt body. The pixel-level contour labeling can more accurately define the boundaries of the coal flow and the belt, so that the model can learn more detailed features during training, thereby improving the accuracy of model recognition. If the labeled contour is not accurate enough, the model may misrecognize the edge area of the coal flow and the belt, thereby affecting the subsequent analysis results.

[0096] A mutual exclusion label system is established to divide the labeled pixel set into two non-intersecting classes of "coal" and "belt", wherein the "belt" label is used as a negative sample set of the "coal" label. The mutual exclusion label system ensures that each pixel can only belong to one of the coal or belt, avoiding label confusion. Taking the belt as the negative sample set of the coal can enable the model to better learn the features of the coal flow and distinguish the coal flow from the background (belt), thereby improving the recognition accuracy of the model for the coal flow. If the labels are not mutually exclusive, the model may be confused during training, resulting in a decrease in recognition effect.

[0097] Referring to Figure 3 In this embodiment, at step S2, an instance segmentation algorithm is used to train the labeled image to obtain an instance segmentation model for identifying the coal flow, specifically including:

[0098] A training sample containing a coal flow target bounding box and a mask is constructed using the labeled image dataset. The bounding box is used to locate the position of the coal flow target in the image, and the mask is used to accurately segment the pixel area of the coal flow target. Constructing a training sample containing this information can provide more comprehensive supervision information for the instance segmentation algorithm, so that the algorithm can better learn the position and shape features of the coal flow.

[0099] Based on the Anchor-Free architecture of YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, a deep learning model with pixel-level segmentation capability of the coal flow area is trained as an instance segmentation model by optimizing the CIoU loss function and the binary cross-entropy loss function of the mask branch, to realize instance-level recognition and boundary positioning of the coal flow target in the belt video image.

[0100] Specifically, the Anchor-Free architecture of YOLOv8 avoids the trouble of manually designing anchor boxes and can automatically learn the scale and position features of the target. The PathAggregationNetwork feature pyramid structure can fuse features at different levels to improve the model's ability to recognize coal flow targets of different sizes. Optimizing the CIoU loss function can improve the accuracy of bounding box regression, enabling the model to more accurately locate the coal flow target. The binary cross-entropy loss function of the mask branch is used to optimize the pixel-level segmentation results of the coal flow, enabling the model to more accurately segment the outline of the coal flow. Through the combination of these technical means, the trained model can perform instance-level recognition and boundary positioning on the coal flow target in the belt video image.

[0101] In this embodiment, in step S3, when calculating the width of the coal flow, the length l k of the line segment EF is calculated by marking the line segment EF on the image collected by the coal flow belt video m , and the real physical length l m of the line segment EF is known, and the ratio of the pixel length l k of the line segment EF in the image to the real physical length l m is the real length represented by a unit pixel. This unit pixel length is used to convert the pixel length of other objects in the image into real physical length.

[0102] And by labeling the four coordinates of the belt edge, the scaling ratio r i at the vertical pixel y (i) is calculated, and the real length l i of the coal flow is obtained as l i = r m × d x k × l i / l i ; wherein d x i is the pixel length of the coal at the vertical pixel y i .

[0103] Referring to Figure 4 and Figure 5 , specifically, the four coordinates of the belt edge are labeled, and the principle of perspective transformation is used to restore the perspective distorted image to an orthographic image, thereby eliminating the influence of perspective distortion. In the perspective restored image, the relationship between the pixels at each position and the real physical size is more consistent. The scaling ratio r i at the vertical pixel y m is calculated, which is used to adjust the proportional relationship between the pixel length and the real physical length at different vertical positions. Through the scaling ratio r k , the unit pixel length l i / l i , and the pixel length of the coal at the vertical pixel yi Pixel length dx at the position i The real physical length of the coal flow at the position can be calculated, and then the width of the coal flow is obtained.

[0104] Since the belt often deviates, it is recommended to select the position of the idler as the position for calculating the width of the coal flow, and other positions that can identify more accurate positions after perspective transformation can also be selected as the position for calculating the width of the coal flow.

[0105] In this embodiment, in step S5, the cross-sectional area of the coal flow is approximated as a trapezoid and a segment of a circle, and the natural angle of repose is combined to calculate the cross-sectional area of the coal flow.

[0106] The area of the trapezoidal part is calculated by the difference between the areas of two isosceles triangles, and the formula is:

[0107]

[0108]

[0109] In the formula, S (l) is the area of an isosceles triangle with the width l of the coal flow as the base and the included angle α of the idler as the base angle; is the area of an isosceles triangle with the corrected width l b as the base and the included angle α of the idler as the base angle; l is the width of the coal flow identified by image recognition, α is the included angle of the idler with the horizontal direction, l m is the standard width of the belt corresponding to the top end of the idler, and h is the height from the top end of the idler to the bottom of the belt;

[0110] Referring to Figure 6 , Figure 7 and Figure 8 , specifically, the support of the idler causes the belt to form a certain shape, and the lower cross section of the coal flow on the belt can be approximated as a trapezoid. The area of this trapezoid can be calculated by the difference between the areas of two isosceles triangles. The area S (l) of an isosceles triangle with the width l of the coal flow as the base represents the area including the coal flow and the support part of the idler, and the area S b of an isosceles triangle with the corrected width l (lb) as the base represents the area of the support part of the idler, and the difference between the two is the area of the trapezoidal part. The calculation of the corrected width l b takes into account the height h from the top end of the idler to the bottom of the belt and the included angle α of the idler, reflecting the influence of the idler on the shape of the belt.

[0111] wherein the area of the segment of a circle is calculated by the difference between the area of a sector and the area of a triangle, and the formula is:

[0112]

[0113] The cross-sectional area of the coal flow is:

[0114] S = S 梯形 + S 弓形

[0115] In the formula, S 扇形 refers to the sector area corresponding to the arcuate part of the coal flow, which is used as the minuend when calculating the arcuate area; S Δ represents the triangular area corresponding to the arcuate part of the coal flow, which is used as the subtrahend when calculating the arcuate area; β is the natural accumulation angle; and S represents the cross-sectional area of the coal flow.

[0116] Specifically, when the coal is naturally accumulated, an arcuate shape is formed, and the area of the arcuate shape can be calculated by the difference between the sector area and the triangular area. The sector area S 扇形 is determined by the natural accumulation angle β and the width l of the coal flow, and the triangular area S Δ is also determined by the natural accumulation angle β and the width l of the coal flow. The difference between the two is the area of the arcuate part. The cross-sectional area of the coal flow can be obtained by adding the area of the trapezoidal part and the area of the arcuate part.

[0117] In step S4, the preset rule of the natural accumulation angle β is:

[0118] when l≤0.2l m and the coal type is "low", β = 20°;

[0119] when 0.2l m <0.8l m and the coal type is "low", β = 30°;

[0120] when l≥0.8l m or the coal type is "high", β = 40°.

[0121] Referring to Figure 9 , specifically, the natural accumulation angle β is related to the accumulation state of the coal and the width of the coal flow. When the width of the coal flow is small and the coal type is "low", the accumulation height of the coal is low, and the natural accumulation angle is small; when the width of the coal flow is moderate and the coal type is "low", the accumulation height of the coal is moderate, and the natural accumulation angle is moderate; when the width of the coal flow is large or the coal type is "high", the accumulation height of the coal is high, and the natural accumulation angle is large. Through this preset rule, a suitable natural accumulation angle can be selected according to the actual situation, and the accuracy of the calculation of the cross-sectional area of the coal flow is improved.

[0122] In this embodiment, in step S6, the formula for calculating the mass of the coal per unit time in combination with the frame time difference is:

[0123] M = S × Δt × v × ρ

[0124] In the formula, Δt is the time difference between the current frame and the previous frame, v is the speed of the coal flow estimated by the feature point detection algorithm, and ρ is the average density of the coal measured.

[0125] Specifically, according to a physical formula, mass is equal to volume multiplied by density. The volume of the coal flow can be regarded as the cross-sectional area S multiplied by the coal flow speed v and then multiplied by the time Δt, that is, V = S × v × Δt. Therefore, the mass M of the coal per unit time is equal to the volume V multiplied by the average density ρ of the coal, that is, M = S × v × Δt × ρ. Through this formula, the mass of the coal per unit time can be calculated according to the cross-sectional area of the coal flow, the coal flow speed, the frame time difference and the average density of the coal, and the amount of coal of a preset granularity can be obtained by accumulation.

[0126] In the process of coal mine production, when the raw coal is transported by the belt conveyor after being mined, a high-definition camera is installed above the belt to collect real-time video images of the belt coal flow. Using the image recognition-based volume analysis method of the application, the coal flow in the video image is segmented at the pixel level, the width is calculated, and the cross-sectional area is modeled. Combined with the speed and density parameters of the coal flow, the automatic calculation and accumulation of the coal volume per unit time are realized. This application scenario can help coal mining enterprises and regulatory units to master real production data in real time, solve the problem of easy fraud in traditional belt scale measurement and data distortion in manual reporting, and provide accurate quantitative basis for production management, yield statistics and regulatory verification. The practical application process of the application is as follows:

[0127] Preparation and deployment

[0128] Equipment installation: In the coal mine belt transportation scene, a high-definition camera is installed at a suitable position above the belt to ensure that the belt coal flow video can be clearly captured. The installation position of the camera needs to ensure stable shooting angle to avoid frequent changes.

[0129] Labeling tool and environment building: Deploy the labeling tool and the deep learning framework (such as the environment required by YOLOv8) on the computer to provide software support for subsequent image labeling and model training.

[0130] Data collection and model training

[0131] Image collection and labeling: Collect belt coal flow video through the camera, and extract multiple frames of images from the video as samples. Use the labeling tool to label the pixels of "coal" and "belt" in each frame of image, establish a mutually exclusive label system, and ensure that the labeled pixel sets of "coal" and "belt" are disjointed. "Belt" is used as a negative sample of "coal".

[0132] Model training: Use the labeled image dataset to construct training samples containing coal flow target bounding boxes and masks. Based on the Anchor-Free architecture of YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, optimize the CIoU loss function and the binary cross-entropy loss function of the mask branch, and train an instance segmentation model that can segment the coal flow at the pixel level.

[0133] Real-time calculation of coal flow parameters

[0134] Coal flow width calculation: mark a line segment EF with known real physical length on the real-time collected video image, calculate its pixel length l k and real physical length l m , determine the unit pixel length l m / l k . At the same time, mark the four-point coordinates of the belt edge for perspective restoration, calculate the scaling ratio r i at each longitudinal pixel to convert the pixel width of the coal flow into the real physical width.

[0135] Natural angle of repose determination and cross-sectional area calculation: classify the coal through the trained classification model (“high” or “low”), combine the coal flow width l with the standard width of the belt corresponding to the top of the roller l m , and determine the natural angle of repose β according to the preset rules. The cross-sectional area of the coal flow is approximated as a trapezoid and an arc, and the parameters such as the roller included angle α and the height h from the top of the roller to the bottom of the belt are used to calculate the areas of the trapezoid and the arc through the formula, and the sum of the two areas is the cross-sectional area S of the coal flow.

[0136] Quantity calculation and accumulation: estimate the coal flow speed v through the feature point detection algorithm and measure the average density ρ of the coal. Use the instance segmentation model to segment the coal flow in each frame of the video and calculate the cross-sectional area, combine the frame time difference Δt, and calculate the mass of the coal per unit time through the formula M = S × Δt × v × ρ, and accumulate the total coal quantity according to the preset granularity such as minutes, hours, etc.

[0137] System optimization and maintenance

[0138] Model updating: as the coal quality in coal mines, belt running state and other factors change, new video images are collected regularly and relabeled, and the instance segmentation model and classification model are fine-tuned or retrained to maintain the recognition accuracy of the model.

[0139] Parameter calibration: regularly check physical parameters such as the roller included angle α and the height h from the top of the roller to the bottom of the belt, and if there is belt wear, roller position change, etc., update the parameters in time to ensure the accuracy of the cross-sectional area calculation.

[0140] Abnormal handling: set up a system monitoring mechanism, when there are belt deviation, camera obstruction and other abnormal situations, automatically alarm and pause the quantity calculation, after the exception is eliminated, resume operation, ensure the reliability of the data.

[0141] The scheme of the present application is applied in a pilot coal mine, and the hourly average accuracy rate is over 80% and the daily accuracy rate is over 90% in three months, which has obtained good comments from coal mining enterprises.

[0142] Example 2

[0143] Referring to Figure 10 Embodiment 2 of the present application provides a belt carrying capacity analysis device based on image recognition, comprising:

[0144] A label labeling module 100 is configured to label two labels of "coal" and "belt" in images collected by a coal flow belt video by using a labeling tool;

[0145] A model training module 200 is configured to train the labeled images by using an instance segmentation algorithm to obtain an instance segmentation model for identifying coal flow;

[0146] A width calculation module 300 is configured to mark a line segment on the images collected by the coal flow belt video, calculate the pixel length and the real physical length of the line segment, determine the unit pixel length, and perform perspective restoration on the belt to calculate the scaling ratio at the longitudinal pixel to obtain the width of the coal flow;

[0147] A classification and parameter determination module 400 is configured to classify the coal by using a classification model and determine the natural angle of repose according to the width of the coal flow and the classification result;

[0148] A cross-sectional area calculation module 500 is configured to approximate the cross-sectional area of the coal flow as a trapezoidal shape and an arc shape, and calculate the cross-sectional area of the coal flow in combination with the natural angle of repose;

[0149] A carrying capacity calculation module 600 is configured to perform instance segmentation on the video frames by using the instance segmentation model, estimate the mass of coal in a unit time in combination with the frame time difference according to the measured average density of coal, the estimated speed of the coal flow, and the cross-sectional area of the coal flow, and accumulate to obtain the coal quantity of a preset granularity.

[0150] In the present embodiment, the label labeling module 100 comprises:

[0151] The pixel-level contour labeling is performed on the edges of the coal flow area and the edges of the belt body;

[0152] A mutual exclusion label system is established, and the labeled pixel set is divided into two types of "coal" and "belt" which are disjointed, wherein the "belt" label is used as a negative sample set of the "coal" label.

[0153] In the present embodiment, the model training module 200 comprises:

[0154] The labeled image data set is used to construct a training sample containing a coal flow target bounding box and a mask;

[0155] Based on the Anchor-Free architecture of the YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, by optimizing the CIoU loss function and the binary cross entropy loss function of the mask branch, a deep learning model with the ability to segment the coal flow area at the pixel level is trained as an instance segmentation model to perform instance-level recognition and boundary positioning of coal flow targets in belt video images.

[0156] In this embodiment, in the width calculation module 300:

[0157] By marking the line segment EF on the image collected by the coal flow belt video, the pixel length l of the line segment EF is calculated. k and the real physical length l m , the unit pixel length is l m / l k ;

[0158] And restore the perspective by marking the four coordinates of the belt edge, and calculate the vertical pixel y i The scaling ratio r i ,, get the real length of coal flow l (i) =r i ×dx i ×l m / l k ; Among them, dx i is the vertical pixel y i The pixel length of the coal.

[0159] In this embodiment, in the cross-sectional area calculation module 500:

[0160] The area of ​​the trapezoidal part is calculated by the difference in the areas of the two isosceles triangles, using the formula:

[0161]

[0162]

[0163] Where S (l) is the area of ​​an isosceles triangle with the coal flow width l as the base and the roller angle α as the base angle; To correct the width l b is the area of ​​an isosceles triangle with the base side and the roller angle α as the base angle; l is the coal flow width identified by the image, α is the angle between the roller and the horizontal direction, l m is the standard width of the belt corresponding to the top of the roller, and h is the height from the top of the roller to the bottom of the belt;

[0164] The area of ​​the arcuate part is calculated by the difference between the area of ​​the sector and the area of ​​the triangle, and the formula is:

[0165]

[0166] The cross-sectional area of the coal flow is:

[0167] S=S 梯形 +S 弓形

[0168] In the formula, S 扇形 represents a sector area corresponding to the arcuate portion of the coal flow, which is used as a subtractive number when calculating the arcuate area; S Δ represents a triangular area corresponding to the arcuate portion of the coal flow, which is subtracted when calculating the arcuate area; β is a natural angle of repose; and S represents the cross-sectional area of the coal flow.

[0169] In the classification and parameter determination module 400, the preset rule of the natural angle of repose β is:

[0170] When l≤0.2l m and the coal type is "low", β=20°.

[0171] When 0.2l m <0.8l m and the coal type is "low", β=30°.

[0172] When l≥0.8l m or the coal type is "high", β=40°.

[0173] In the traffic volume calculation module 600 in the embodiment, the mass of coal in a unit time is calculated according to the frame time difference, and the formula is:

[0174] M=S*Δt*v*ρ

[0175] In the formula, Δt is the time difference between the current frame and the previous frame, v is the coal flow speed estimated by the feature point detection algorithm, and ρ is the average density of the coal measured.

[0176] It should be noted that the information interaction and execution process between the modules of the device described above are based on the same concept as the method embodiment in Embodiment 1 of the present application, and the technical effects brought by the method embodiment are the same as those of the method embodiment of the present application. For specific content, refer to the description of the method embodiment in the foregoing description of the method embodiment, which will not be repeated here.

[0177] Embodiment 3

[0178] Embodiment 3 of the present application provides a non-transitory computer readable storage medium, which stores a program code of a belt traffic volume analysis method based on image recognition, and the program code includes instructions for executing the belt traffic volume analysis method based on image recognition of Embodiment 1 or any possible implementation manner thereof.

[0179] The computer-readable storage medium can be any available media or a set of one or more available media that is accessible by a computer, data center, server, and the like, or data storage device integrated with the same. The available media can be a magnetic medium, (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), and the like.

[0180] Embodiment 4

[0181] Embodiment 4 of the present application provides an electronic device, comprising a memory and a processor;

[0182] The processor and the memory complete the communication between each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calling the program instructions can execute the image recognition-based belt load analysis method of embodiment 1 or any possible implementation manner thereof.

[0183] Specifically, the processor can be implemented by hardware or software, when implemented by hardware, the processor can be a logic circuit, an integrated circuit, and the like; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software codes stored in a memory, the memory can be integrated in the processor or located outside the processor and independently exist.

[0184] In the above embodiments, all or part of them can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0185] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with a general purpose computing device, which can be centralized on a single computing device or distributed across a network of multiple computing devices, and optionally implemented with program code executable by a computing device, which can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be performed in a different order than shown or described, or made into individual integrated circuit modules or multiple modules or steps made into a single integrated circuit module. Thus, the application is not limited to any particular combination of hardware and software.

[0186] While the application has been described in detail and with reference to specific embodiments thereof, it will be apparent to one skilled in the art that various modifications or changes can be made therein without departing from the spirit and scope of the application. Accordingly, it is intended that all such modifications and changes be included within the scope of the application as claimed.

Claims

1. A belt transport volume analysis method based on image recognition, characterized in that: The following steps are involved: Use the annotation tool to label the images captured by the coal flow belt video with the labels "coal" and "belt"; The labeled images are trained using an instance segmentation algorithm to obtain an instance segmentation model for identifying coal flow. Mark line segments on the image captured by the coal flow belt video, calculate the pixel length and real physical length of the line segments, determine the unit pixel length, perform perspective restoration on the belt, calculate the scaling ratio at the longitudinal pixel, and obtain the coal flow width; Classify the coal using a classification model and determine the natural accumulation angle based on the coal flow width and the classification results; The cross-sectional area of ​​the coal flow is approximated as a trapezoid and an arch, and the cross-sectional area of ​​the coal flow is calculated in combination with the natural stacking angle; The video frames are segmented by the instance segmentation model, and the mass of coal per unit time is calculated based on the measured average coal density, the estimated coal flow velocity and the coal flow cross-sectional area, combined with the frame time difference, and the preset particle size coal amount is accumulated.

2. The belt transport volume analysis method based on image recognition according to claim 1, characterized in that: When labeling "coal" and "belt," use the labeling tool to perform the following operations on the coal flow belt video frame image: Pixel-level contour annotation of the coal flow area edge and the belt body edge; A mutually exclusive label system is established to divide the labeled pixel set into two disjoint categories: "coal" and "belt", where the "belt" label serves as a negative sample set for the "coal" label.

3. The belt transport volume analysis method based on image recognition according to claim 1, characterized in that: The labeled images are trained using an instance segmentation algorithm to obtain an instance segmentation model for identifying coal flow, which includes: Use the labeled image dataset to construct training samples containing coal flow target bounding boxes and masks; Based on the Anchor-Free architecture of the YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, by optimizing the CIoU loss function and the binary cross entropy loss function of the mask branch, a deep learning model with the ability to segment the coal flow area at the pixel level is trained as an instance segmentation model to perform instance-level recognition and boundary positioning of coal flow targets in belt video images.

4. The belt transport volume analysis method based on image recognition according to claim 1, characterized in that: When calculating the coal flow width, the line segment EF is marked on the image collected by the coal flow belt video, and the pixel length of the line segment EF is calculated. k and the real physical length l m , the unit pixel length is l m / l k ; And restore the perspective by marking the four coordinates of the belt edge, and calculate the vertical pixel y i The scaling ratio r i ,, get the real length of coal flow l (i) =r i ×dx i ×l m / l k ; Among them, dx i is the vertical pixel y i The pixel length of the coal.

5. The belt transport volume analysis method based on image recognition according to claim 1, characterized in that: The cross-sectional area of ​​the coal flow is approximated as a trapezoid and an arch, and the cross-sectional area of ​​the coal flow is calculated in combination with the natural stacking angle: The area of ​​the trapezoidal part is calculated by the difference in the areas of the two isosceles triangles, using the formula: Where S (l) is the area of ​​an isosceles triangle with the coal flow width l as the base and the roller angle α as the base angle; To correct the width l b is the area of ​​an isosceles triangle with the base side and the roller angle α as the base angle; l is the coal flow width identified by the image, α is the angle between the roller and the horizontal direction, l m is the standard width of the belt corresponding to the top of the roller, and h is the height from the top of the roller to the bottom of the belt; The area of ​​the arcuate part is calculated by the difference between the area of ​​the sector and the area of ​​the triangle, and the formula is: The cross-sectional area of ​​coal flow is: S=S 梯形 +S 弓形 Where S 扇形 Refers to the sector area corresponding to the arched part of the coal flow, which is used as the minuend when calculating the arched area; S Δ Represents the area of ​​the triangle corresponding to the bow portion of the coal flow, which is subtracted when calculating the bow area; β is the natural stacking angle; S represents the cross-sectional area of ​​the coal flow; The preset rule of the natural stacking angle β is: When l≤0.2l m When the coal type is "low", β = 20°; When 0.2l m <l<0.8l m When the coal type is "low", β = 30°; When l≥0.8l m Or when the coal type is "high", β=40°.

6. The belt transport volume analysis method based on image recognition according to claim 5, characterized in that: The formula for calculating the mass of coal per unit time by combining the frame time difference is: M=S×Δt×v×ρ Where Δt is the time difference between the current frame and the previous frame, v is the coal flow velocity estimated by the feature point detection algorithm, and ρ is the average density of the coal obtained by measurement.

7. A belt transport volume analysis device based on image recognition, characterized in that: include: The labeling module is used to use the labeling tool to label the images captured by the coal flow belt video with the two labels "coal" and "belt"; A model training module is used to train the labeled images using an instance segmentation algorithm to obtain an instance segmentation model for identifying coal flow; The width calculation module is used to mark line segments on the image captured by the coal flow belt video, calculate the pixel length and real physical length of the line segments, determine the unit pixel length, restore the belt perspective, calculate the scaling ratio at the longitudinal pixel, and obtain the coal flow width; A classification and parameter determination module is used to classify coal using a classification model and determine the natural accumulation angle based on the coal flow width and the classification results; Cross-sectional area calculation module, used to approximate the cross-sectional area of ​​the coal flow into a trapezoid and an arc, and calculate the cross-sectional area of ​​the coal flow in combination with the natural stacking angle; The transport volume calculation module is used to perform instance segmentation on the video frame through the instance segmentation model, calculate the mass of coal per unit time based on the measured average density of coal, the estimated coal flow velocity and the cross-sectional area of ​​the coal flow, combined with the frame time difference, and accumulate the preset particle size coal volume.

8. The belt transport volume analysis device based on image recognition according to claim 7, characterized in that: In the label marking module: Pixel-level contour annotation of the coal flow area edge and the belt body edge; A mutually exclusive label system is established to divide the labeled pixel set into two disjoint categories: "coal" and "belt", where the "belt" label serves as a negative sample set for the "coal" label.

9. The belt transport volume analysis device based on image recognition according to claim 7, characterized in that: In the model training module: Use the labeled image dataset to construct training samples containing coal flow target bounding boxes and masks; Based on the Anchor-Free architecture of the YOLOv8 algorithm and the PathAggregationNetwork feature pyramid structure, by optimizing the CIoU loss function and the binary cross-entropy loss function of the mask branch, a deep learning model capable of pixel-level segmentation of coal flow areas was trained as an instance segmentation model to identify and locate the boundaries of coal flow targets in belt conveyor video images at the instance level. In the width calculation module: By marking the line segment EF on the image collected by the coal flow belt video, the pixel length l of the line segment EF is calculated. k and the real physical length l m , the unit pixel length is l m / l k ; And restore the perspective by marking the four coordinates of the belt edge, and calculate the vertical pixel y i The scaling ratio r i ,, get the real length of coal flow l (i) =r i ×dx i ×l m / l k ; Among them, dx i is the vertical pixel y i The pixel length of the coal.

10. The belt transport volume analysis device based on image recognition according to claim 7, characterized in that: In the cross-sectional area calculation module: The area of ​​the trapezoidal part is calculated by the difference in the areas of the two isosceles triangles, using the formula: Where S (l) is the area of ​​an isosceles triangle with the coal flow width l as the base and the roller angle α as the base angle; To correct the width l b is the area of ​​an isosceles triangle with the base side and the roller angle α as the base angle; l is the coal flow width identified by the image, α is the angle between the roller and the horizontal direction, l m is the standard width of the belt corresponding to the top of the roller, and h is the height from the top of the roller to the bottom of the belt; The area of ​​the arcuate part is calculated by the difference between the area of ​​the sector and the area of ​​the triangle, and the formula is: The cross-sectional area of ​​coal flow is: S=S 梯形 +S 弓形 Where S 扇形 Refers to the sector area corresponding to the arched part of the coal flow, which is used as the minuend when calculating the arched area; S Δ Represents the area of ​​the triangle corresponding to the bow portion of the coal flow, which is subtracted when calculating the bow area; β is the natural stacking angle; S represents the cross-sectional area of ​​the coal flow; In the classification and parameter determination module, the preset rule of the natural stacking angle β is: When l≤0.2l m When the coal type is "low", β = 20°; When 0.2l m <l<0.8l m When the coal type is "low", β = 30°; When l≥0.8l m Or when the coal type is "high", β=40°; In the transportation volume calculation module, the formula for calculating the mass of coal per unit time in combination with the frame time difference is: M=S×Δt×v×ρ Where Δt is the time difference between the current frame and the previous frame, v is the coal flow velocity estimated by the feature point detection algorithm, and ρ is the average density of the coal obtained by measurement.

Images