Coal flow monitoring system and method based on laser scanner

Abstract

Disclosed are a coal flow monitoring system based on a laser scanner and a method thereof, which adopts image processing technology and artificial intelligence technology to determine the volume of the transported coal based on a three-dimensional model of the transported coal in a unit time, and to perform spatial domain feature filtering based on a convolution kernel on a monitoring image representing the apparent state of the coal through a convolutional neural network to capture explicit and implicit mode features within a local neighborhood in the monitoring image, and then to accurately estimate the density value of the coal. In this way, an accurate mapping model of the apparent state features of the coal and the density value of the coal is constructed, so that a more accurate coal flow value is obtained based on the density estimated value and the volume of the transported coal.

Description

Technical Field

[0001] This application relates to the field of coal flow monitoring technology, and more specifically, to a coal flow monitoring system and method based on a laser scanner. Background Technology

[0002] Before stacker-reclaimer systems became unmanned, operators manually observed and controlled the coal flow rate. With unmanned operation, human observation is no longer necessary; the technology must keep pace.

[0003] After stacker-reclaimer equipment achieves intelligent and unmanned operation, automatic coal flow control technology is an important technology for the intelligent operation of bucket wheel excavators. Its purpose is to prevent belt overload, and sometimes it is necessary for coal blending ratio. Some bucket wheel excavators are equipped with electronic belt scales on the cantilever belts, but due to the constant changes in the pitch angle of the cantilever belt and the lack of calibration methods, the accuracy of these scales is extremely poor in practical use, and they cannot be used for measuring the coal flow control of bucket wheel excavators.

[0004] To achieve a relatively accurate and easy-to-maintain method for measuring coal flow, a laser scanner can be installed above the cantilever conveyor belt. The volume is calculated by scanning the surface shape of the coal flow, and the density is estimated to convert it into flow rate. The advantage of this method is that it has less error drift, is relatively stable, and does not require frequent calibration. Its disadvantage is that the density of the coal requires an empirical value.

[0005] Therefore, an optimized coal flow monitoring solution based on laser scanners is needed. Summary of the Invention

[0006] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a coal flow monitoring system and method based on a laser scanner. This system employs image processing and artificial intelligence technologies to determine the volume of the transported coal based on a three-dimensional model of the coal transported per unit time. Furthermore, it uses a convolutional neural network to perform spatial domain feature filtering on the monitoring image representing the apparent state of the coal, based on convolutional kernels, to capture explicit and implicit pattern features within the local neighborhood of the monitoring image, thereby accurately estimating the coal density value. In this way, a precise mapping model between the apparent state characteristics of coal and the coal density value is constructed, thereby obtaining a more accurate coal flow value based on the density estimate and the volume of the transported coal.

[0007] Accordingly, according to one aspect of this application, a coal flow monitoring system based on a laser scanner is provided, comprising:

[0008] The laser scanning module is used to acquire a three-dimensional model of the transported coal per unit time, collected by a laser scanner.

[0009] A volume calculation module is used to determine the volume of the transported coal based on a three-dimensional model of the transported coal within the unit time.

[0010] A coal image acquisition module is used to acquire monitoring images of the transmitted coal captured by a camera;

[0011] A coal feature extraction module is used to process the monitoring image through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model.

[0012] A feature fusion module is used to fuse the shallow feature map and the deep feature map to obtain a decoded feature map;

[0013] The feature discrimination enhancement module is used to enhance the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map.

[0014] A density value decoding module is used to decode and regress the enhanced decoding feature map through a decoder to obtain a decoded value, which represents the density value of the transported coal; and

[0015] The coal flow monitoring result module is used to determine the coal flow value based on the decoded value and the volume of the transmitted coal.

[0016] In the aforementioned coal flow monitoring system based on a laser scanner, the shallow layers of the convolutional neural network model are the first to sixth layers of the convolutional neural network model, and the deep layers of the convolutional neural network model are the last layer of the convolutional neural network model.

[0017] In the aforementioned coal flow monitoring system based on a laser scanner, the convolutional neural network model used as the feature extractor is a deep residual network model.

[0018] In the aforementioned coal flow monitoring system based on a laser scanner, the feature fusion module is further configured to: fuse the shallow feature map and the deep feature map using the following formula to obtain a decoded feature map; wherein, the formula is:

[0019] X = Concat[F1, F2]

[0020] Where F1 represents the shallow feature map, F2 represents the deep feature map, Concat[·,·] represents the cascade function, and X represents the decoded feature map.

[0021] In the aforementioned coal flow monitoring system based on a laser scanner, the feature discrimination enhancement module is further configured to: enhance the feature value discrimination of the decoded feature map using the following formula to obtain an enhanced decoded feature map; wherein, the formula is:

[0022]

[0023]

[0024]

[0025] Where F represents the decoded feature map, and a and b are predetermined hyperparameters. and The addition and subtraction of the feature map by position are represented, the division represents the feature value at each position of the feature map divided by the response value, and Cov(·) represents the convolution operation through a single convolutional layer, and F′ represents the enhanced decoded feature map.

[0026] In the aforementioned coal flow monitoring system based on a laser scanner, the density value decoding module is further configured to: use the decoder to perform decoding regression on the enhanced decoding feature map using the following formula to obtain the decoded value; wherein, the formula is: Where X is each enhanced decoding feature matrix in the enhanced decoding feature map, Y is the decoded value, and W is the weight matrix. This represents matrix multiplication.

[0027] According to another aspect of this application, a method for monitoring coal flow based on a laser scanner is also provided, comprising:

[0028] Obtain a 3D model of the transported coal per unit time, captured by a laser scanner;

[0029] Based on the three-dimensional model of the coal being transported per unit time, the volume of the coal being transported is determined.

[0030] Acquire monitoring images of the transmitted coal captured by the camera;

[0031] The surveillance image is passed through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model.

[0032] The shallow feature map and the deep feature map are fused to obtain a decoded feature map;

[0033] The decoded feature map is enhanced with feature value discrimination to obtain an enhanced decoded feature map;

[0034] The enhanced decoded feature map is decoded and regressed using a decoder to obtain a decoded value, which represents the density value of the transported coal; and

[0035] Based on the decoded value and the volume of the transmitted coal, the coal flow rate is determined.

[0036] In the above-mentioned coal flow monitoring method based on laser scanner, the shallow layers of the convolutional neural network model are the first to sixth layers of the convolutional neural network model, and the deep layers of the convolutional neural network model are the last layer of the convolutional neural network model.

[0037] In the above-mentioned coal flow monitoring method based on laser scanner, the convolutional neural network model used as the feature extractor is a deep residual network model.

[0038] In the aforementioned method for monitoring coal flow based on a laser scanner, the step of fusing the shallow feature map and the deep feature map to obtain a decoded feature map includes: fusing the shallow feature map and the deep feature map using the following formula to obtain the decoded feature map; wherein, the formula is:

[0039] X = Concat[F1, F2]

[0040] Where F1 represents the shallow feature map, F2 represents the deep feature map, Concat[·,·] represents the cascade function, and X represents the decoded feature map.

[0041] In the aforementioned coal flow monitoring method based on a laser scanner, the step of enhancing the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map includes: enhancing the feature value discrimination of the decoded feature map using the following formula to obtain an enhanced decoded feature map; wherein, the formula is:

[0042]

[0043]

[0044]

[0045] Where F represents the decoded feature map, and a and b are predetermined hyperparameters. and The addition and subtraction of the feature map by position are represented, the division represents the feature value at each position of the feature map divided by the response value, and Cov(·) represents the convolution operation through a single convolutional layer, and F′ represents the enhanced decoded feature map.

[0046] In the aforementioned method for monitoring coal flow based on a laser scanner, the step of decoding and regressing the enhanced decoding feature map using a decoder to obtain a decoded value, wherein the decoded value represents the density value of the transported coal, includes: using the decoder to perform decoding and regressing on the enhanced decoding feature map using the following formula to obtain the decoded value; wherein the formula is: Where X is each enhanced decoding feature matrix in the enhanced decoding feature map, Y is the decoded value, and W is the weight matrix. This represents matrix multiplication.

[0047] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions, which, when executed by the processor, cause the processor to perform the coal flow monitoring method based on a laser scanner as described above.

[0048] According to another aspect of this application, a computer-readable medium is provided having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the coal flow monitoring method based on a laser scanner as described above.

[0049] Compared with existing technologies, the coal flow monitoring system and method based on a laser scanner provided in this application employs image processing and artificial intelligence technologies. It determines the volume of the transported coal based on a three-dimensional model of the coal transported per unit time, and uses a convolutional neural network to perform spatial domain feature filtering on the monitoring image representing the apparent state of the coal, capturing explicit and implicit pattern features within the local neighborhood of the monitoring image, thereby accurately estimating the coal density value. In this way, a precise mapping model between the apparent state characteristics of coal and the coal density value is constructed, thus obtaining a more accurate coal flow value based on the density estimate and the volume of the transported coal. Attached Figure Description

[0050] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0051] Figure 1 This is an application scenario diagram of a coal flow monitoring system based on a laser scanner according to an embodiment of this application.

[0052] Figure 2 This is a block diagram of a coal flow monitoring system based on a laser scanner according to an embodiment of this application.

[0053] Figure 3 This is a schematic diagram of the architecture of a coal flow monitoring system based on a laser scanner according to an embodiment of this application.

[0054] Figure 4This is a flowchart of a coal flow monitoring method based on a laser scanner according to an embodiment of this application.

[0055] Figure 5 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0057] Application Overview

[0058] As mentioned above, by installing a laser scanner above the cantilever conveyor belt, the volume is calculated by scanning the surface shape of the coal flow, and then the density is estimated and converted into flow rate. The advantage of this method is that it has less error drift, is relatively stable, and does not require frequent calibration. The disadvantage is that the coal density requires an empirical value. In other words, improving the accuracy of coal density estimation can improve the accuracy of coal flow rate monitoring.

[0059] In the technical solution of this application, the density value of coal is related to the type of coal and its physical properties (for example, if the coal is wet, its density value will increase). Both the type of coal and its physical properties are reflected in the appearance of the coal, including its color and texture. Therefore, if an accurate mapping model between the appearance characteristics of coal and its density value can be constructed, the estimation accuracy of the coal density value can be significantly improved, thereby improving the accuracy of coal flow monitoring.

[0060] In recent years, deep learning and neural networks have been widely applied in fields such as computer vision, natural language processing, and text signal processing. Furthermore, deep learning and neural networks have demonstrated near-human or even surpassed human-level performance in areas such as image classification, object detection, semantic segmentation, and text translation. The development of deep learning and neural networks has provided new ideas and solutions for constructing a precise mapping model between the apparent state characteristics of coal and its density values.

[0061] Specifically, in the technical solution of this application, a three-dimensional model of the transported coal per unit time is first obtained by a laser scanner. Then, based on the three-dimensional model of the transported coal per unit time, the volume of the transported coal is determined. For example, in a specific example of this application, the volume is calculated by scanning the surface shape of the coal flow. Of course, the volume can also be calculated directly based on the three-dimensional model; this is not a limitation of this application.

[0062] Simultaneously, monitoring images of the transported coal are acquired via a camera. That is, while a three-dimensional model of the transported coal is acquired using a laser scanner, monitoring images of the transported coal are also acquired via a camera. As mentioned earlier, the density of coal is related to its type and physical properties. Both the type and physical properties of coal are reflected in its appearance, including its color and texture. By acquiring these monitoring images, image processing and artificial intelligence technologies can be used to analyze them and obtain the apparent characteristics of the transported coal.

[0063] Specifically, the monitoring image is first processed by a convolutional neural network model, which acts as a feature extractor, to extract shallow feature maps from the shallow layers and deep feature maps from the deep layers of the convolutional neural network model. That is, in this application's technical solution, a deep learning-based deep convolutional neural network model is used as a feature extractor to perform spatial domain feature filtering based on convolutional kernels on the monitoring image to capture explicit and implicit pattern features within the local neighborhood of the monitoring image. Here, during the encoding process of the convolutional neural network model, as the encoding depth increases, shallow features are gradually weakened or even submerged. In this application's technical solution, shallow features include shape, texture, and color, etc. If these shallow features can be effectively preserved, the accuracy of the representation of the apparent features of the transported coal will be significantly improved.

[0064] Based on this, the technical solution of this application extracts shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model. Here, the shallow layers of the convolutional neural network model are the first to sixth layers, and the deep layers are the last few layers, for example, the last layer. In a specific example of this application, the ratio between the deep layers and the shallow layers of the convolutional neural network model is 10-15.

[0065] After obtaining the shallow feature map and the deep feature map, they are fused to obtain a decoded feature map. To fully utilize the deep and shallow features expressed by the deep and shallow feature maps, the decoded feature map is preferably obtained by directly cascading the deep and shallow feature maps along the channel dimension. Next, the decoded feature map is decoded and regressed using a decoder to obtain a decoded value, which represents the density of the transported coal. That is, the decoder performs decoded regression on the decoded feature map using a weight matrix with learnable weight parameters to obtain the decoded value. Furthermore, based on the decoded value and the volume of the transported coal, the coal flow rate is determined; that is, the product between the decoded value and the volume of the transported coal is calculated to obtain the coal flow rate.

[0066] Specifically, in the technical solution of this application, in order to fully utilize the deep and shallow features expressed by the deep feature map and the shallow feature map, the decoded feature map is preferably obtained by directly cascading the deep feature map and the shallow feature map in the channel dimension. Therefore, since the deep feature map is obtained by further feature extraction based on the shallow feature map through convolution based on the spatial distribution of feature values ​​of the shallow feature map, both the deep and shallow feature maps contain local feature values ​​with more significant spatial importance. Thus, if the feature values ​​of the fused decoded feature map can be effectively distinguished during decoding, the training speed of the decoder and the accuracy of the decoding results can obviously be improved.

[0067] Therefore, the applicant of this application performs interactive enhancement on the decoded feature map, for example denoted as F, based on distinguishable physical stimuli, as follows:

[0068]

[0069]

[0070]

[0071] Where a and b are predefined hyperparameters. and The addition and subtraction of the feature map by position are represented, the division represents dividing each position of the feature map by the response value, and Cov(·) represents the convolution operation through a single convolutional layer.

[0072] Here, the interaction enhancement based on discriminability physical incentives is used to improve the interaction between the feature space and the solution space of the decoding regression problem during the backpropagation process through gradient descent. It extracts and mimics actionable features in a manner similar to physical incentives. Thus, a general-purpose, low-dimensional, differentiable physical incentive is used to obtain the physical representation of actionable features with gradient discriminability, thereby strengthening the active parts in the decoding feature map F during training, so as to improve the training speed of the decoder and the accuracy of the decoding results of the decoded features after training.

[0073] Based on this, this application provides a coal flow monitoring system based on a laser scanner, comprising: a laser scanning module for acquiring a three-dimensional model of the coal being transported per unit time, collected by the laser scanner; a volume calculation module for determining the volume of the coal being transported based on the three-dimensional model of the coal being transported per unit time; a coal image acquisition module for acquiring a monitoring image of the coal being transported, collected by a camera; a coal feature extraction module for extracting shallow feature maps from the shallow layers and deep feature maps from the deep layers of the convolutional neural network model by passing the monitoring image through a convolutional neural network model as a feature extractor; a feature fusion module for fusing the shallow feature maps and the deep feature maps to obtain a decoded feature map; a feature discrimination enhancement module for enhancing the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map; a density value decoding module for decoding and regressing the enhanced decoded feature map through a decoder to obtain a decoded value, the decoded value representing the density value of the coal being transported; and a coal flow monitoring result module for determining a coal flow value based on the decoded value and the volume of the coal being transported.

[0074] Figure 1 This is an application scenario diagram of a coal flow monitoring system based on a laser scanner according to an embodiment of this application. Figure 1 As shown, in this application scenario, the data is first obtained from a laser scanner (e.g., such as...). Figure 1 The Sc shown in the figure represents the amount of coal transported per unit time (e.g., as shown in the figure). Figure 1 The three-dimensional model of Co (as shown) and the camera (e.g., such as Figure 1 The monitoring image of the transported coal is acquired by Ca as shown in the figure. Then, the 3D model of the transported coal and the monitoring image of the transported coal are input to a server deployed with a coal flow monitoring algorithm based on a laser scanner (e.g., such as...). Figure 1In the illustrated S), the server is able to process the three-dimensional model of the transported coal and the monitoring image of the transported coal based on the laser scanner-based coal flow monitoring algorithm to obtain a decoded value, the decoded value being used to represent the density value of the transported coal, and the coal flow value being determined based on the decoded value and the volume of the transported coal.

[0075] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0076] Exemplary System

[0077] Figure 2 This is a block diagram of a coal flow monitoring system based on a laser scanner according to an embodiment of this application. Figure 2 As shown, a coal flow monitoring system 100 based on a laser scanner according to an embodiment of this application includes: a laser scanning module 110, used to acquire a three-dimensional model of the coal being transported per unit time by a laser scanner; a volume calculation module 120, used to determine the volume of the coal being transported based on the three-dimensional model of the coal being transported per unit time; a coal image acquisition module 130, used to acquire a monitoring image of the coal being transported captured by a camera; and a coal feature extraction module 140, used to pass the monitoring image through a convolutional neural network model as a feature extractor to extract shallow features from the shallow layers of the convolutional neural network model. The system includes: a shallow feature map and a deep feature map extracted from the deep layers of the convolutional neural network model; a feature fusion module 150 for fusing the shallow feature map and the deep feature map to obtain a decoded feature map; a feature discrimination enhancement module 160 for enhancing the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map; a density value decoding module 170 for decoding and regressing the enhanced decoded feature map through a decoder to obtain a decoded value, the decoded value representing the density value of the transported coal; and a coal flow monitoring result module 180 for determining the coal flow rate value based on the decoded value and the volume of the transported coal.

[0078] Figure 3 This is a schematic diagram of the architecture of a coal flow monitoring system based on a laser scanner according to an embodiment of this application. Figure 3As shown, firstly, a three-dimensional model of the transported coal within a unit time period is acquired by a laser scanner; then, based on the three-dimensional model of the transported coal within the unit time period, the volume of the transported coal is determined, and simultaneously, a monitoring image of the transported coal is acquired by a camera; next, the monitoring image is processed through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model; subsequently, the shallow feature maps and the deep feature maps are fused to obtain a decoded feature map; then, the feature value discrimination enhancement is performed on the decoded feature map to obtain an enhanced decoded feature map; next, the enhanced decoded feature map is decoded and regressed through a decoder to obtain a decoded value, which is used to represent the density value of the transported coal; finally, based on the decoded value and the volume of the transported coal, the coal flow rate is determined.

[0079] As mentioned above, by installing a laser scanner above the cantilever conveyor belt, the volume is calculated by scanning the surface shape of the coal flow, and then the density is estimated and converted into flow rate. The advantage of this method is that it has less error drift, is relatively stable, and does not require frequent calibration. The disadvantage is that the coal density requires an empirical value. In other words, improving the accuracy of coal density estimation can improve the accuracy of coal flow rate monitoring.

[0080] In the technical solution of this application, the density value of coal is related to the type of coal and its physical properties (for example, if the coal is wet, its density value will increase). Both the type of coal and its physical properties are reflected in the appearance of the coal, including its color and texture. Therefore, if an accurate mapping model between the appearance characteristics of coal and its density value can be constructed, the estimation accuracy of the coal density value can be significantly improved, thereby improving the accuracy of coal flow monitoring.

[0081] In recent years, deep learning and neural networks have been widely applied in fields such as computer vision, natural language processing, and text signal processing. Furthermore, deep learning and neural networks have demonstrated near-human or even surpassed human-level performance in areas such as image classification, object detection, semantic segmentation, and text translation. The development of deep learning and neural networks has provided new ideas and solutions for constructing a precise mapping model between the apparent state characteristics of coal and its density values.

[0082] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the laser scanning module 110 and the volume calculation module 120 are used to acquire a three-dimensional model of the transported coal within a unit time period collected by the laser scanner, and to determine the volume of the transported coal based on the three-dimensional model of the transported coal within the unit time period. For example, in a specific example of this application, the volume is calculated by scanning the surface shape of the coal flow; of course, the volume can also be calculated directly based on the three-dimensional model, and this is not limited to this application.

[0083] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the coal image acquisition module 130 is used to acquire monitoring images of the transported coal captured by a camera. That is, while acquiring a three-dimensional model of the transported coal using a laser scanner, a monitoring image of the transported coal is simultaneously acquired using a camera. As mentioned earlier, the density value of coal is related to its type and physical properties. Both the type and physical properties of coal are reflected in its appearance, including its color and texture. By acquiring the monitoring images, image processing and artificial intelligence technologies can be used to analyze the images to obtain the appearance characteristics of the transported coal.

[0084] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the coal feature extraction module 140 is used to process the monitoring image through a convolutional neural network model, which acts as a feature extractor, to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model. The convolutional neural network model acting as the feature extractor is a deep residual network model.

[0085] In other words, in the technical solution of this application, a deep convolutional neural network model based on deep learning is used as a feature extractor to perform spatial domain feature filtering based on convolution kernels on the monitoring image to capture explicit and implicit pattern features in the local neighborhood of the monitoring image. Here, in the encoding process of the convolutional neural network model, as the encoding depth increases, shallow features will gradually be weakened or even submerged. In the technical solution of this application, shallow features include shape, texture, and color, etc. If these shallow features can be effectively preserved, the accuracy of the representation of the appearance features of the transported coal will be significantly improved.

[0086] Based on this, the technical solution of this application extracts shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model. Here, the shallow layers of the convolutional neural network model are the first to sixth layers, and the deep layers are the last few layers, for example, the last layer. In a specific example of this application, the ratio between the deep layers and the shallow layers of the convolutional neural network model is 10-15.

[0087] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the feature fusion module 150 is used to fuse the shallow feature map and the deep feature map to obtain a decoded feature map. To fully utilize the deep and shallow features expressed by the deep and shallow feature maps, the decoded feature map is preferably obtained by directly cascading the deep and shallow feature maps along the channel dimension.

[0088] Specifically, in this embodiment of the application, the shallow feature map and the deep feature map are fused using the following formula to obtain the decoded feature map; wherein, the formula is:

[0089] X = Concat[F1, F2]

[0090] Where F1 represents the shallow feature map, F2 represents the deep feature map, Concat[·,·] represents the cascade function, and X represents the decoded feature map.

[0091] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the feature discrimination enhancement module 160 is used to enhance the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map. Specifically, in the technical solution of this application, in order to fully utilize the deep and shallow features expressed by the deep and shallow feature maps, it is preferable to obtain the decoded feature map by directly cascading the deep and shallow feature maps along the channel dimension. Therefore, since the deep feature map is obtained by further feature extraction based on the shallow feature map through convolution of the spatial distribution of feature values ​​from the shallow feature map, both the deep and shallow feature maps contain local feature values ​​with more significant spatial importance. Thus, if the feature values ​​of the fused decoded feature map can be effectively distinguished during decoding, the training speed of the decoder and the accuracy of the decoding results can obviously be improved.

[0092] Therefore, the applicant of this application performs interactive enhancement on the decoded feature map, for example denoted as F, based on distinguishable physical stimuli, as follows:

[0093]

[0094]

[0095]

[0096] Where F represents the decoded feature map, and a and b are predetermined hyperparameters. and The addition and subtraction of the feature map by position are represented, the division represents the feature value at each position of the feature map divided by the response value, and Cov(·) represents the convolution operation through a single convolutional layer, and F′ represents the enhanced decoded feature map.

[0097] Here, the interaction enhancement based on discriminability physical incentives is used to improve the interaction between the feature space and the solution space of the decoding regression problem during the backpropagation process through gradient descent. It extracts and mimics actionable features in a manner similar to physical incentives. Thus, a general-purpose, low-dimensional, differentiable physical incentive is used to obtain the physical representation of actionable features with gradient discriminability, thereby strengthening the active parts in the decoding feature map F during training, so as to improve the training speed of the decoder and the accuracy of the decoding results of the decoded features after training.

[0098] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the density value decoding module 170 is used to decode and regress the enhanced decoding feature map through a decoder to obtain a decoded value, which represents the density value of the transmitted coal. That is, the decoder uses a weight matrix with learnable weight parameters to perform decoding and regression on the decoding feature map to obtain the decoded value.

[0099] Specifically, in this embodiment, the decoder is used to perform decoding regression on the enhanced decoding feature map using the following formula to obtain the decoding value; wherein, the formula is: Where X is each enhanced decoding feature matrix in the enhanced decoding feature map, Y is the decoded value, and W is the weight matrix. This represents matrix multiplication.

[0100] In the aforementioned coal flow monitoring system 100 based on a laser scanner, the coal flow monitoring result module 180 is used to determine the coal flow value based on the decoded value and the volume of the transported coal. That is, the product between the decoded value and the volume of the transported coal is calculated to obtain the coal flow value.

[0101] In summary, the coal flow monitoring system 100 based on a laser scanner, according to embodiments of this application, employs image processing and artificial intelligence technologies to determine the volume of the transported coal based on a three-dimensional model of the coal transported per unit time. It also uses a convolutional neural network to perform spatial domain feature filtering on the monitoring image representing the apparent state of the coal, based on convolutional kernels, to capture explicit and implicit pattern features within the local neighborhood of the monitoring image, thereby accurately estimating the coal density value. In this way, a precise mapping model between the apparent state characteristics of coal and the coal density value is constructed, thereby obtaining a more accurate coal flow value based on the density estimate and the volume of the transported coal.

[0102] As described above, the laser scanner-based coal flow monitoring system 100 according to the embodiments of this application can be implemented in various terminal devices, such as servers for laser scanner-based coal flow monitoring. In one example, the laser scanner-based coal flow monitoring system 100 according to the embodiments of this application can be integrated into a terminal device as a software module and / or a hardware module. For example, the laser scanner-based coal flow monitoring system 100 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the laser scanner-based coal flow monitoring system 100 can also be one of many hardware modules of the terminal device.

[0103] Alternatively, in another example, the laser scanner-based coal flow monitoring system 100 and the terminal device can also be separate devices, and the laser scanner-based coal flow monitoring system 100 can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0104] Exemplary methods

[0105] Figure 4 This is a flowchart of a coal flow monitoring method based on a laser scanner according to an embodiment of this application. Figure 4As shown, the coal flow monitoring method based on a laser scanner according to an embodiment of this application includes: S110, acquiring a three-dimensional model of the coal being transported per unit time, collected by a laser scanner; S120, determining the volume of the coal being transported based on the three-dimensional model of the coal being transported per unit time; S130, acquiring a monitoring image of the coal being transported, collected by a camera; S140, passing the monitoring image through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model; S150, fusing the shallow feature maps and the deep feature maps to obtain a decoded feature map; S160, enhancing the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map; S170, performing decoding regression on the enhanced decoded feature map through a decoder to obtain a decoded value, the decoded value being used to represent the density value of the coal being transported; and S180, determining a coal flow value based on the decoded value and the volume of the coal being transported.

[0106] In one example, in the above-mentioned coal flow monitoring method based on laser scanner, the shallow layers of the convolutional neural network model are the first to sixth layers of the convolutional neural network model, and the deep layers of the convolutional neural network model are the last layer of the convolutional neural network model.

[0107] In one example, in the above-mentioned coal flow monitoring method based on laser scanner, the convolutional neural network model that serves as the feature extractor is a deep residual network model.

[0108] In one example, in the above-described method for monitoring coal flow based on a laser scanner, fusing the shallow feature map and the deep feature map to obtain a decoded feature map includes: fusing the shallow feature map and the deep feature map using the following formula to obtain the decoded feature map; wherein, the formula is:

[0109] X = Concat[F1, F2]

[0110] Where F1 represents the shallow feature map, F2 represents the deep feature map, Concat[·,·] represents the cascade function, and X represents the decoded feature map.

[0111] In one example, in the above-described method for monitoring coal flow based on a laser scanner, the step of enhancing the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map includes: enhancing the feature value discrimination of the decoded feature map using the following formula to obtain an enhanced decoded feature map; wherein, the formula is:

[0112]

[0113]

[0114]

[0115] Where F represents the decoded feature map, and a and b are predetermined hyperparameters. and The addition and subtraction of the feature map by position are represented, the division represents the feature value at each position of the feature map divided by the response value, and Cov(·) represents the convolution operation through a single convolutional layer, and F′ represents the enhanced decoded feature map.

[0116] In one example, in the above-described laser scanner-based coal flow monitoring method, the step of performing decoding regression on the enhanced decoding feature map using a decoder to obtain a decoded value, the decoded value representing the density value of the transported coal, includes: using the decoder to perform decoding regression on the enhanced decoding feature map using the following formula to obtain the decoded value; wherein, the formula is: Where X is each enhanced decoding feature matrix in the enhanced decoding feature map, Y is the decoded value, and W is the weight matrix. This represents matrix multiplication.

[0117] In summary, the coal flow monitoring method based on a laser scanner according to embodiments of this application is explained. It employs image processing and artificial intelligence technologies to determine the volume of the transported coal based on a three-dimensional model of the coal transported per unit time. Furthermore, it uses a convolutional neural network to perform spatial domain feature filtering on the monitoring image representing the apparent state of the coal, capturing explicit and implicit pattern features within the local neighborhood of the monitoring image, thereby accurately estimating the coal density value. In this way, a precise mapping model between the apparent state characteristics of coal and the coal density value is constructed, thereby obtaining a more accurate coal flow value based on the density estimate and the volume of the transported coal.

[0118] Exemplary electronic devices

[0119] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present application. Figure 5 This is a block diagram of an electronic device according to an embodiment of this application. (e.g.) Figure 5 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0120] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0121] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the functions and / or other desired functions in the laser scanner-based coal flow monitoring method of the various embodiments of this application described above. Various content such as 3D models and monitoring images may also be stored in the computer-readable storage medium.

[0122] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0123] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0124] The output device 14 can output various information to the outside, including classification results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0125] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0126] Exemplary computer program products and computer-readable storage media

[0127] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the functions in the laser scanner-based coal flow monitoring method according to various embodiments of this application described in the "Exemplary Methods" section of this specification.

[0128] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0129] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the function in the laser scanner-based coal flow monitoring method according to various embodiments of this application described in the "Exemplary Methods" section above.

[0130] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0131] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0132] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0133] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0134] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0135] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A coal flow monitoring system based on a laser scanner, characterized in that, include: The laser scanning module is used to acquire a three-dimensional model of the transported coal per unit time, collected by a laser scanner. A volume calculation module is used to determine the volume of the transported coal based on a three-dimensional model of the transported coal within the unit time. A coal image acquisition module is used to acquire monitoring images of the transmitted coal captured by a camera; A coal feature extraction module is used to process the monitoring image through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model. A feature fusion module is used to fuse the shallow feature map and the deep feature map to obtain a decoded feature map; The feature discrimination enhancement module is used to enhance the feature value discrimination of the decoded feature map to obtain an enhanced decoded feature map. The density value decoding module is used to decode and regress the enhanced decoding feature map through the decoder to obtain a decoded value, which is used to represent the density value of the transmitted coal. as well as The coal flow monitoring result module is used to determine the coal flow value based on the decoded value and the volume of the transmitted coal; The feature discrimination enhancement module is further used for: The decoded feature map is enhanced by performing feature value discrimination enhancement using the following formula to obtain an enhanced decoded feature map; The formula is as follows: in The decoded feature map, and It is a pre-defined hyperparameter. and This represents positional addition and subtraction of the feature map. Division represents dividing each positional feature value of the feature map by twice a predetermined hyperparameter. This indicates a convolution operation performed through a single convolutional layer. This represents the enhanced decoding feature map.

2. The coal flow monitoring system based on a laser scanner according to claim 1, characterized in that, The shallow layers of the convolutional neural network model are the first to sixth layers of the convolutional neural network model, and the deep layers of the convolutional neural network model are the last layer of the convolutional neural network model.

3. The coal flow monitoring system based on a laser scanner according to claim 2, characterized in that, The convolutional neural network model used as the feature extractor is a deep residual network model.

4. The coal flow monitoring system based on a laser scanner according to claim 3, characterized in that, The feature fusion module is further used for: The shallow feature map and the deep feature map are fused using the following formula to obtain the decoded feature map; The formula is as follows: in, This represents the shallow feature map. This represents the deep feature map. Represents a cascade function. This represents the decoded feature map.

5. The coal flow monitoring system based on a laser scanner according to claim 4, characterized in that, The density value decoding module is further used for: The decoder is used to perform decoding regression on the enhanced decoding feature map using the following formula to obtain the decoded value; wherein the formula is: ,in These are the various enhanced decoding feature matrices in the enhanced decoding feature map. It is the decoded value, It is a weight matrix. This represents matrix multiplication.

6. A method for monitoring coal flow based on a laser scanner, characterized in that, include: Obtain a 3D model of the transported coal per unit time, captured by a laser scanner; Based on the three-dimensional model of the coal being transported per unit time, the volume of the coal being transported is determined. Acquire monitoring images of the transmitted coal captured by the camera; The surveillance image is passed through a convolutional neural network model as a feature extractor to extract shallow feature maps from the shallow layers of the convolutional neural network model and deep feature maps from the deep layers of the convolutional neural network model. The shallow feature map and the deep feature map are fused to obtain a decoded feature map; The decoded feature map is enhanced with feature value discrimination to obtain an enhanced decoded feature map; The enhanced decoded feature map is decoded and regressed using a decoder to obtain a decoded value, which represents the density value of the transported coal; and Based on the decoded value and the volume of the transmitted coal, the coal flow rate is determined; The decoded feature map is enhanced by feature value discrimination using the following formula to obtain an enhanced decoded feature map; The formula is as follows: in The decoded feature map, and It is a pre-defined hyperparameter. and This represents positional addition and subtraction of the feature map. Division represents dividing each positional feature value of the feature map by twice a predetermined hyperparameter. This indicates a convolution operation performed through a single convolutional layer. This represents the enhanced decoding feature map.

7. The coal flow monitoring method based on a laser scanner according to claim 6, characterized in that, The shallow layers of the convolutional neural network model are the first to sixth layers of the convolutional neural network model, and the deep layers of the convolutional neural network model are the last layer of the convolutional neural network model.

8. The coal flow monitoring method based on a laser scanner according to claim 7, characterized in that, The convolutional neural network model used as the feature extractor is a deep residual network model.

9. The coal flow monitoring method based on a laser scanner according to claim 8, characterized in that, The process of fusing the shallow feature map and the deep feature map to obtain the decoded feature map includes: The shallow feature map and the deep feature map are fused using the following formula to obtain the decoded feature map; The formula is as follows: in, This represents the shallow feature map. This represents the deep feature map. Represents a cascade function. This represents the decoded feature map.

Images