Multi-modal data fusion coal ash nondestructive testing method and system and storage medium

Abstract

The invention relates to a coal ash content nondestructive testing method and system based on multi-modal data fusion and a storage medium, and relates to the field of coal ash content detection.The coal ash content nondestructive testing method comprises the steps that coal contour and image quality recognition is conducted on a plurality of coal images, and a plurality of coal contours and a plurality of image quality coefficients are obtained; performing point cloud fitting in the plurality of point cloud data according to the plurality of coal contours to obtain a plurality of coal point clouds and a plurality of fitting accuracy coefficients; constructing a coal three-dimensional model according to the coal point cloud, processing to obtain the coal volume, and calculating to obtain the coal density according to the coal volume and the coal mass; calculating and configuring ash content analysis precision according to the fitting accuracy coefficient and the image quality coefficient, and performing integrated ash content analysis on the coal density to obtain ash content information. According to the method, the technical problems that the ash content cannot be quickly and effectively identified by a traditional method, and the detection timeliness and accuracy are insufficient can be solved; more efficient and more accurate coal ash detection can be realized, and the applicability, the efficiency and the accuracy of ash content detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of coal ash detection, and in particular to a multi-modal data fusion coal ash non-destructive detection method, system and storage medium. Background Art

[0002] Coal is one of the world's main energy sources, and its quality directly affects energy utilization efficiency and environmental protection levels.

[0003] In the process of coal production, transportation and combustion, the ash content of coal is one of the important indicators to measure the quality of coal. Too high ash content will lead to reduced coal combustion efficiency and increased pollutant emissions, which will have a negative impact on energy utilization in industries such as electricity and steel. Therefore, accurate and rapid determination of coal ash content is crucial to optimize coal utilization, improve combustion efficiency and reduce environmental pollution.

[0004] However, traditional coal ash detection methods usually rely on laboratory methods such as chemical analysis, X-ray fluorescence, infrared spectroscopy, etc. These methods cannot quickly and effectively identify the ash content, especially in real-time monitoring and large-scale production processes, and it is difficult to meet the needs of rapid coal ash detection. Summary of the invention

[0005] The present invention aims to solve the technical problems that traditional coal ash detection methods cannot quickly and effectively identify the ash content, have insufficient detection timeliness and accuracy, and are difficult to meet the needs of real-time ash monitoring. A coal ash non-destructive detection method, system and storage medium with multimodal data fusion are provided to solve the problem.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: In a first aspect, the present invention provides a method for nondestructive detection of coal ash with multimodal data fusion, comprising: performing multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition on coal to obtain multiple coal images, multiple point cloud data and coal quality, performing coal contour recognition and image quality recognition on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients, wherein the multiple angles include angles on both sides of the coal; performing iterative fitting of coal point clouds in the multiple point cloud data according to the multiple coal contours, to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein the multiple coal contours are adjusted according to the multiple image quality coefficients. Iterative fitting of point clouds; constructing a three-dimensional coal model based on the multiple coal point clouds, processing to obtain coal volume information, and calculating the coal density based on the coal volume information and the coal quality; calculating the configuration ash analysis accuracy based on the multiple fitting accuracy coefficients and the multiple image quality coefficients, performing integrated ash analysis on the coal density, and obtaining the ash information of the coal.

[0007] In a second aspect, the present invention further provides an electronic device, comprising: At least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can perform the steps of any one of the methods described in the first aspect above.

[0008] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the steps of the method described in any one of the first aspects are implemented.

[0009] The beneficial effects of the present invention are as follows: by performing multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition on coal, a plurality of coal images, a plurality of point cloud data and coal quality are obtained, wherein the multi-angle includes the angles on both sides of the coal; then the plurality of coal images are subjected to coal contour recognition and image quality recognition, and a plurality of coal contours and a plurality of image quality coefficients are obtained; then the coal point cloud is iteratively fitted in the plurality of point cloud data according to the plurality of coal contours, and a plurality of coal point clouds and a plurality of fitting accuracy coefficients are obtained, wherein the plurality of coal contours are adjusted to perform iterative fitting of the point clouds respectively according to the plurality of image quality coefficients; further a three-dimensional coal model is constructed according to the plurality of coal point clouds, and the coal volume information is obtained by processing, and the coal density is obtained by calculating the coal volume information and the coal quality; finally, the ash analysis accuracy is calculated and configured according to the plurality of fitting accuracy coefficients and the plurality of image quality coefficients, and the coal density is integrated ash analysis is performed on the coal density to obtain the ash information of the coal; that is, through multimodal data fusion and integrated learning analysis, more efficient and more accurate coal ash detection can be achieved, and the applicability, efficiency and accuracy of ash content detection can be significantly improved, thereby meeting the real-time monitoring requirements of ash in the coal production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of a flow chart of a nondestructive detection method for coal ash using multimodal data fusion provided by the present invention; Figure 2 A schematic diagram of the structure of an electronic device provided by the present invention; Figure 3 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

[0011] In the accompanying drawings, the components represented by the reference numerals are described as follows: Electronic device 500 , memory 510 , processor 520 , first computer program 511 , computer-readable storage medium 600 , second computer program 611 . DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0013] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0014] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0015] Embodiment 1, as Figure 1 As shown, the embodiment of the present invention provides a method for nondestructive detection of coal ash by multimodal data fusion, which specifically includes the following steps: S100: Perform multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition on coal to obtain multiple coal images, multiple point cloud data and coal quality, perform coal contour recognition and image quality recognition on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients, wherein the multiple angles include angles on both sides of the coal.

[0016] Specifically, firstly, multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition are carried out on the coal on the conveyor belt. Among them, the multi-angle includes the angles on both sides of the coal (the left and right sides of the conveyor belt), that is, the coal is transmitted on the conveyor belt, and image acquisition devices are installed on both sides of the conveyor belt. Multi-angle shooting ensures that most areas of the coal surface can be covered. The image obtained by each camera will include information from different angles, which helps to improve the recognition accuracy of the coal surface contour; laser scanners are installed on both sides of the coal conveyor belt, that is, in order to avoid the coal being blocked or missed due to angle problems, the point cloud acquisition system scans the coal surface from two different angles at the same time to generate three-dimensional point cloud data covering the entire coal surface. These data can accurately reflect the morphological characteristics and spatial distribution of the coal; quality acquisition refers to the real-time acquisition of the weight of the coal through high-precision sensors or weighing devices to obtain coal weight data; multiple coal images, multiple point cloud data and coal quality are obtained through data acquisition.

[0017] Through multi-angle images, point cloud data collection and coal quality data collection, multi-dimensional information of coal can be obtained comprehensively and accurately. These data provide strong support for subsequent ash detection and analysis.

[0018] Furthermore, step S100 of the present invention further includes: S110: Pre-training a coal image recognition channel, wherein the coal image recognition channel includes a coal contour recognition path and a noise point recognition branch.

[0019] Furthermore, step S110 of the present invention further includes: S111: Based on the coal detection data in the historical time, a set of sample coal images is collected, and the proportion of coal contours and noise pixels in each sample coal image is marked to obtain a set of sample coal contours and a set of sample noise ratios; S112: The set of sample coal images is used as input data, and the set of sample coal contours and the set of sample noise ratios are used as supervision data respectively, and based on the convolutional neural network, the coal contour recognition path and the noise recognition branch are supervised and trained respectively; S113: The coal contour recognition path and the noise recognition branch that have been completed through combination training are combined to obtain a coal image recognition channel.

[0020] Specifically, coal detection data within a historical period (such as within the last month) is obtained, and sample coal images at different times and under different production conditions are collected based on the coal detection data to obtain a set of sample coal images. Next, the coal contour and the ratio of noise pixels in each sample coal image are annotated. For example, an image processing algorithm (such as edge detection) is used to identify the actual contour of the coal in each sample coal image, and the contour information of the coal surface is accurately extracted; noise pixels are usually abnormal points in the image, which appear as points that do not conform to the shape of the coal or background characteristics. By setting a threshold or using an image classification model based on machine learning, the ratio of noise can be effectively identified; a set of sample coal contours and a set of sample noise ratios are obtained, in which the sample coal images, sample coal contours, and sample noise ratios correspond one to one.

[0021] Convolutional neural network (CNN) is a type of deep learning model that is widely used to process multi-dimensional data such as images and videos. It automatically extracts features from input data through convolution operations and can learn more complex and efficient feature representations. Then, a coal contour recognition path and a noise point recognition branch are constructed based on the convolutional neural network. The coal contour recognition path and the noise point recognition branch both include an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. The convolution layer is responsible for extracting local features in the image through convolution operations; the pooling layer is used to reduce the spatial resolution of the feature map, but retain the most important feature information; the fully connected layer is used to fuse these features through the fully connected layer based on the features extracted by the convolution layer and the pooling layer and output the final result, that is, mapping high-dimensional features to the output space. The input data of the input layer of the coal contour recognition path is the coal image, and the output data of the output layer is the coal contour; the input data of the input layer of the noise point recognition branch is the coal image, and the output data of the output layer is the noise point ratio.

[0022] Then, the sample coal image is used as input, the sample coal contour is used as supervision, and the sample coal image set and the sample coal contour set are used to supervise the coal contour recognition path. First, forward propagation is performed through the input coal image to calculate the output of each convolution layer, and finally the prediction result of the network (i.e., the predicted coal contour) is generated; then, the loss value is calculated through the loss function according to the difference between the predicted coal contour and the true label; further, the gradient of the loss function relative to each weight and bias is calculated, the gradient is back-propagated and the model parameters are updated, and the weights and biases of the network are updated according to the optimization algorithm (such as Adam, SGD, etc.); the sample data is used for iterative optimization until the loss function meets the expected convergence conditions, and the trained coal contour recognition path is obtained. On the other hand, the sample coal image is used as input, the sample noise ratio is used as supervision, and the sample coal image set and the sample noise ratio set are used as training data to supervise the noise recognition branch until convergence, and the trained noise recognition branch is obtained, wherein the training method of the noise recognition branch is the same as the training method of the coal contour recognition path, which is not described in detail here. By constructing a coal contour recognition path and noise point recognition branch based on a convolutional neural network, the accuracy and efficiency of coal ash detection can be greatly improved, thus providing support for real-time, automated and accurate monitoring of coal ash content.

[0023] Finally, the trained coal contour recognition path and noise point recognition branch are combined to obtain a coal image recognition channel, that is, the input of the coal image recognition channel is the coal image, and the output is the coal contour and noise point ratio.

[0024] S120: Input the multiple coal images into the coal image recognition channel respectively to identify and obtain multiple coal contours and multiple noise point ratios; S130: Subtract the multiple noise point ratios from 1 to obtain multiple image quality coefficients.

[0025] Specifically, the multiple coal images are then input into the coal image recognition channel for recognition, and multiple coal contours and multiple noise point ratios are output; then, the multiple noise point ratios are subtracted from 1, and the difference between 1 and the noise ratio is set as the image quality coefficient to obtain multiple image quality coefficients, wherein a smaller image quality coefficient indicates poor image quality; a larger image quality coefficient indicates good image quality. By calculating the noise point ratio and obtaining the image quality coefficient, the quality of each coal image can be quantified, providing support for subsequent coal point cloud data fitting and coal ash prediction.

[0026] S200: performing iterative fitting of coal point clouds in the multiple point cloud data according to the multiple coal contours respectively, to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein the multiple coal contours are adjusted to perform iterative fitting of point clouds respectively according to the multiple image quality coefficients.

[0027] Furthermore, step S200 of the present invention further includes: S210: performing iterative fitting of a coal point cloud in a first point cloud data in the plurality of point cloud data according to a first coal contour in the plurality of coal contours, to obtain a first initial fitting coal point cloud and a first initial fitting accuracy coefficient.

[0028] Furthermore, step S210 of the present invention further includes: S211: According to the first coal contour, perform random frame selection in the first point cloud data to obtain a first framed coal point cloud; S212: Calculate the ratio of the point cloud area ratio in the first framed coal point cloud to the area of ​​the first coal contour as the first frame selection accuracy coefficient; S213: Continue to use the first coal contour, perform random frame selection in the first point cloud data, and perform iterative frame selection fitting of the coal point cloud until the convergent frame selection fitting times are reached, and output the maximum frame selection accuracy coefficient and the corresponding framed coal point cloud as the first initial fitting accuracy coefficient and the first initial fitting coal point cloud.

[0029] Specifically, first, any one coal contour is randomly selected from the multiple coal contours as the first coal contour, and according to the first coal contour, a random frame selection is performed in the first point cloud data to obtain a first framed coal point cloud, wherein the first framed coal point cloud is a point cloud falling within the first coal contour; then, the point cloud area ratio within the first framed coal point cloud is calculated, such as the area occupied by a single point in the first framed coal point cloud multiplied by the number of all points to obtain the point cloud area. The point cloud area can also be estimated by calculating the spatial distribution of each sampling point in the point cloud, and then the ratio of the point cloud area ratio within the first framed coal point cloud to the area of ​​the first coal contour is used as the first frame selection accuracy coefficient, wherein the frame selection accuracy coefficient reflects the degree of matching between the point cloud and the coal contour in the current framed area. The closer the accuracy coefficient is to 1, the more the frame selection result matches the contour, and the better the fitting effect.

[0030] Then continue to use the first coal contour, perform random selection in the first point cloud data, and perform iterative selection and fitting of the coal point cloud, that is, perform multiple iterative random selections until the converged selection and fitting times are reached. The converged selection and fitting times can be set according to the point cloud fitting accuracy requirements, such as 100 times; obtain multiple selected coal point clouds and multiple selection accuracy coefficients; finally output the largest selection accuracy coefficient as the first initial fitting accuracy coefficient, and output the largest selection accuracy coefficient and the corresponding selected coal point cloud as the first initial fitting coal point cloud. Through repeated iterative selection and fitting, the matching degree between the coal point cloud and the coal contour can be maximized, and the precision and accuracy of the point cloud fitting results can be improved.

[0031] S220: Subtract the first image quality coefficient among the multiple image quality coefficients from 1 to obtain a first contour adjustment coefficient, use the first contour adjustment coefficient as the first contour adjustment ratio, randomly adjust the contour of the first contour adjustment ratio within the first coal contour, and obtain a first adjusted coal contour; S230: Use the first adjusted coal contour to iteratively fit the coal point cloud in the first point cloud data to obtain a first adjusted fitted coal point cloud and a first adjusted fitting accuracy coefficient; S240: Continue to adjust the first coal contour and iteratively fit the coal point cloud until convergence, output the maximum fitting accuracy coefficient and the fitted coal point cloud, and obtain the first coal point cloud and the first fitting accuracy coefficient; S250: Continue to use the second coal contour, adjust the contour according to the second image quality coefficient, and iteratively fit the coal point cloud in the second point cloud data until convergence to obtain a second coal point cloud and a second fitting accuracy coefficient.

[0032] Specifically, first, any one image quality coefficient among the multiple image quality coefficients is randomly selected as the first image quality coefficient, and the first image quality coefficient has a corresponding relationship with the first coal contour; then, 1 is subtracted from the first image quality coefficient, and the difference between the two is set as the first contour adjustment coefficient, and the contour adjustment coefficient is used to determine the adjustment range of the coal contour. The lower the image quality coefficient, the larger the contour adjustment coefficient, which means that the contour needs more adjustments to improve the fitting accuracy; for example, assuming that the first image quality coefficient is 0.95, the first contour adjustment coefficient is 0.05, that is, the first coal contour will be adjusted according to this ratio (0.05), and each time the adjustment is made, the length and shape of the contour will change slightly.

[0033] Next, the first contour adjustment coefficient is used as the first contour adjustment ratio, and the contour of the first contour adjustment ratio within the first coal contour is randomly adjusted. For example, assuming that the first image quality coefficient is 0.95, the first contour adjustment coefficient is 0.05, that is, the first contour adjustment ratio is 0.05; the first coal contour will be adjusted according to this ratio (0.05), and each time the adjustment is made, the length and shape of the contour will change slightly. The adjustment ratio is 5% of the length or shape of the coal contour. For example, a 5% length portion of the first coal contour is selected to randomly adjust the shape or length. For example, the length of the coal contour of the 5% length portion is 10 cm, which is 12 cm after random adjustment; the first adjusted coal contour is obtained.

[0034] Then, the first adjusted coal contour is adopted to perform iterative fitting of the coal point cloud in the first point cloud data until the predetermined fitting times are reached, the maximum frame selection accuracy coefficient is output as the first adjusted fitting accuracy coefficient, and the fitted coal point cloud corresponding to the first adjusted fitting accuracy coefficient is set as the first adjusted fitting coal point cloud, and the first adjusted fitting coal point cloud and the first adjusted fitting accuracy coefficient are output. Then, according to the first contour adjustment ratio, the first coal contour is adjusted continuously, and according to the adjusted second adjusted coal contour, the coal point cloud is iteratively fitted to obtain the second adjusted fitting coal point cloud and the second adjusted fitting accuracy coefficient; the first coal contour adjustment and coal point cloud iterative fitting are further performed using the same method until the predetermined convergence condition is met (i.e., the predetermined contour adjustment times and point cloud fitting times are reached), multiple fitted coal point clouds and multiple fitting accuracy coefficients are obtained, and the maximum fitting accuracy coefficient is output as the first fitting accuracy coefficient, and the fitted coal point cloud corresponding to the maximum fitting accuracy coefficient is set as the first coal point cloud.

[0035] Then, a second coal contour is randomly selected from the multiple coal contours, wherein the second coal contour is different from the first coal contour; then, the second coal contour is adjusted by the second image quality coefficient, and the coal point cloud is iteratively fitted in the second point cloud data until convergence, to obtain the second coal point cloud and the second fitting accuracy coefficient; and multiple coal point clouds and multiple fitting accuracy coefficients are obtained in sequence using the same method. By dynamically adjusting the coal contour and iteratively fitting multiple times, a coal point cloud that best matches the point cloud data is finally obtained, which can significantly improve the adaptability and accuracy of the point cloud fitting, thereby further improving the accuracy of the construction of the coal three-dimensional model.

[0036] S300: constructing a three-dimensional coal model according to the plurality of coal point clouds, processing to obtain coal volume information, and calculating to obtain coal density according to the coal volume information and coal mass.

[0037] Furthermore, step S300 of the present invention further includes: S310: constructing a three-dimensional coal model based on the first coal point cloud and the second coal point cloud in the multiple coal point clouds; S320: calculating and obtaining coal volume information based on the three-dimensional coal model; S330: calculating the ratio of the coal mass and the coal volume information to obtain coal density.

[0038] Specifically, the first coal point cloud and the second coal point cloud in the multiple coal point clouds are selected, that is, the coal point clouds at two different angles (two angles on the left and right of the conveyor belt); then, the first coal point cloud and the second coal point cloud collected at different angles are aligned, and point cloud registration technology (such as ICP algorithm) is usually used to achieve spatial alignment of point cloud data at different angles to ensure that different perspective data of the same object can be accurately connected; then, based on the alignment, the two sets of point cloud data are fused. During the fusion process, redundant point cloud information can be reduced by deduplication or weighted averaging, while ensuring the integrity and accuracy of the point cloud. The purpose of point cloud fusion is to generate a more complete three-dimensional model based on multiple perspectives; then, based on the fused point cloud data, surface reconstruction technology (such as Poisson reconstruction, etc.) is used to convert the point cloud into a three-dimensional mesh model. These technologies generate polygonal meshes by connecting adjacent points in the point cloud, and then construct the three-dimensional surface of the coal to obtain a three-dimensional model of the coal. By collecting and processing point cloud data from multiple angles, combined with point cloud registration, fusion and 3D mesh reconstruction technology, a 3D model of coal can be accurately constructed, which not only improves the accuracy and details of the model, but also provides strong support for subsequent coal ash analysis and quality inspection.

[0039] Then, the volume information of the coal is calculated based on the three-dimensional model of the coal, such as calculating the volume of the coal through the grid surface. The three-dimensional grid model is composed of several polyhedrons (usually triangular facets). For each triangular facet, the area of ​​the triangular facet and the corresponding height information (i.e., the distance from the triangular facet to the coal reference plane) are used to estimate the volume. Through the geometric relationship of these facets, the overall volume of the coal can be calculated. Then, the ratio of the coal mass (coal weight) to the coal volume information is calculated to obtain the coal density. By using the three-dimensional model and mass data of the coal, combined with mathematical models and algorithms, the volume and density of the coal can be efficiently calculated, providing effective data support for coal ash detection.

[0040] S400: Calculate and configure ash analysis accuracy based on the multiple fitting accuracy coefficients and the multiple image quality coefficients, perform integrated ash analysis on the coal density, and obtain ash information of the coal.

[0041] Furthermore, step S400 of the present invention further includes: S410: Calculate the mean of the multiple fitting accuracy coefficients to obtain an average fitting accuracy coefficient, and calculate the mean of the multiple image quality coefficients to obtain an average image quality coefficient; S420: Subtract the average fitting accuracy coefficient and the average image quality coefficient from 1 respectively to obtain a fitting error coefficient and an image error coefficient, and add them together to obtain an error coefficient.

[0042] Specifically, the average of the multiple fitting accuracy coefficients is calculated to obtain an average fitting accuracy coefficient; the average of the multiple image quality coefficients is calculated to obtain an average image quality coefficient; then the average fitting accuracy coefficient is subtracted from 1 to obtain a fitting error coefficient; the average image quality coefficient is subtracted from 1 to obtain an image error coefficient. Finally, the fitting error coefficient and the image error coefficient are added and summed to obtain an error coefficient, that is, an overall error.

[0043] S430: training an ash analysis channel including P ash analysis branches, where P is an integer greater than or equal to 1.

[0044] Further, step S430 of the present invention further includes: S431: According to the historical test data of coal ash, a sample coal density set and a sample coal ash set are collected as ash analysis supervision training data; S432: P data are randomly selected from the ash analysis supervision training data with replacement to obtain P branch supervision training data; S433: The P branch supervision training data are respectively used to supervise the training of P ash analysis branches, wherein the loss function of each ash analysis branch supervision training is as follows: ; Where LOSS is the loss, M is the number of supervised training data in each branch supervised training data, is the sample coal ash content in the i-th group of supervised training data, The coal ash is output by the ash analysis branch; S434: P ash analysis branches that have completed training are combined to obtain an ash analysis channel.

[0045] Specifically, first, based on the historical test data of coal ash, sample coal density and ash content at different sample coal densities are collected to obtain a sample coal density set and a sample coal ash (ash content) set, and the sample coal density set and the sample coal ash set are used as ash analysis supervision training data. Then the ash analysis supervision training data is divided into P parts to obtain P data sets, where P is an integer greater than or equal to 1, and the value of P can be set according to actual needs, such as 30; and P times are selected with replacement from the P data sets to construct the first branch supervision training data, and the same method is used to iteratively select P times to obtain P branch supervision training data.

[0046] Next, P ash analysis branches are constructed based on the BP neural network, wherein the ash analysis branch is a BP neural network model that can be iteratively optimized in machine learning, including an input layer, multiple hidden layers and an output layer, wherein the input data of the input layer is coal density, and the output data is coal ash (ash content); then, the sample coal density is used as input, and the sample coal ash is used as supervision, and the P branch supervision training data are used to perform supervised training on the P ash analysis branches respectively. First, the coal density is used as input, and is passed to the hidden layer through the input layer, and is continued to be calculated through each hidden layer, and finally the prediction result (ash content of coal) is output; then, the difference (error) between the prediction result and the actual coal ash content is calculated by the loss function, wherein the expression of the loss function is: ; Where LOSS is the loss, M is the number of supervised training data in each branch supervised training data, is the sample coal ash content in the i-th group of supervised training data, The coal ash output by the ash analysis branch; then calculate the gradient of each layer according to the loss function, use the chain rule to back-propagate the error, adjust the weight and bias of each layer from the output layer, and use the gradient descent algorithm to adjust the weight and bias in the network according to the calculated gradient, with the goal of minimizing the loss function; perform iterative training, optimize the weight and bias through multiple iterations until the loss function converges (less than the expected index), and obtain the P ash analysis branches that have been trained. Finally, combine the P ash analysis branches that have been trained to obtain the ash analysis channel.

[0047] S440: multiply the error coefficient by P and round it to obtain Q, where Q is an integer greater than or equal to 1 and less than P; S450: input the coal density into Q randomly selected ash analysis branches, output the Q branch coal ash, and calculate the mean to obtain the ash information of the coal.

[0048] Specifically, the error coefficient is multiplied by P and rounded to obtain Q, where Q is an integer greater than or equal to 1 and less than P. For example, assuming that the error coefficient is 0.12 and P is 30, Q is 0.12*30, which is rounded to 4. Then, Q ash analysis branches are randomly called from the P ash analysis branches of the ash analysis channel, and the coal density is input into the randomly selected Q ash analysis branches, and the Q branch coal ash (ash content) is output; finally, the Q branch coal ash is averaged to obtain the coal ash information.

[0049] By matching and calling the appropriate number of ash analysis branches according to the error coefficient, that is, when the error coefficient is large, it means that the prediction accuracy of the ash analysis is low, or the original data (such as coal image quality, point cloud fitting accuracy, etc.) is poor. At this time, by configuring more ash analysis branches to predict the coal ash content, the robustness and accuracy of the model can be increased. By integrating the prediction results of multiple branches, the deviation of a single branch can be effectively reduced, and the accuracy of the overall prediction can be improved. When the error coefficient is small, it means that the model prediction accuracy is high, or the quality of coal images, point clouds and other data is good, the number of branches that need to participate in the prediction will be reduced, which not only improves the calculation efficiency, but also reduces redundant calculations and reduces the burden on the system, thereby speeding up the ash detection process. At the same time, a small number of branches can still provide sufficient prediction accuracy, especially when the data quality is good, the processing speed can be improved without losing accuracy. By controlling the selection of the number of branches by the error coefficient, computing resources can be dynamically allocated to achieve efficient and accurate ash detection.

[0050] The embodiment of the present invention provides a method for nondestructive detection of coal ash by multimodal data fusion, which has at least the following technical effects: By performing multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition on coal, multiple coal images, multiple point cloud data and coal quality are obtained, wherein the multiple angles include the angles on both sides of the coal; then, coal contour recognition and image quality recognition are performed on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients; then, coal point clouds are iteratively fitted in the multiple point cloud data according to the multiple coal contours to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein the multiple coal contours are iteratively fitted according to the multiple image quality coefficients; further, a three-dimensional coal model is constructed according to the multiple coal point clouds, and the coal volume information is obtained by processing, and the coal density is calculated according to the coal volume information and the coal quality; finally, the ash analysis accuracy is calculated according to the multiple fitting accuracy coefficients and the multiple image quality coefficients, and the coal density is integrated ash analysis to obtain the ash information of the coal; that is, through multimodal data fusion and integrated learning analysis, more efficient and accurate coal ash detection can be achieved, and the applicability, efficiency and accuracy of ash content detection can be significantly improved, thereby meeting the real-time monitoring needs of ash in the coal production process.

[0051] For example 2, please refer to Figure 2 , Figure 2 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 2As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, the following steps are implemented: multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition are performed on coal to obtain multiple coal images, multiple point cloud data and coal quality; coal contour recognition and image quality recognition are performed on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients, wherein the multiple angles include angles on both sides of the coal; coal point cloud iterative fitting is performed in the multiple point cloud data according to the multiple coal contours to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein the multiple coal contours are adjusted according to the multiple image quality coefficients. Iterative fitting of point clouds; constructing a three-dimensional coal model based on the multiple coal point clouds, processing to obtain coal volume information, and calculating the coal density based on the coal volume information and the coal quality; calculating the configuration ash analysis accuracy based on the multiple fitting accuracy coefficients and the multiple image quality coefficients, performing integrated ash analysis on the coal density, and obtaining the ash information of the coal.

[0052] For example 3, please refer to Figure 3 , Figure 3 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 3 As shown, this embodiment provides a computer-readable storage medium 600, on which a second computer program 611 is stored. When the second computer program 611 is executed by the processor, the following steps are implemented: multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition are performed on the coal to obtain multiple coal images, multiple point cloud data and coal quality; coal contour recognition and image quality recognition are performed on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients, wherein the multiple angles include angles on both sides of the coal; coal point cloud iterative fitting is performed in the multiple point cloud data according to the multiple coal contours respectively to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein the multiple coal contours are adjusted according to the multiple image quality coefficients. Iterative fitting of point clouds; constructing a three-dimensional coal model based on the multiple coal point clouds, processing to obtain coal volume information, and calculating the coal density based on the coal volume information and the coal quality; calculating the configuration ash analysis accuracy based on the multiple fitting accuracy coefficients and the multiple image quality coefficients, performing integrated ash analysis on the coal density, and obtaining the ash information of the coal.

[0053] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0054] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0058] Although preferred embodiments of the present invention have been described, additional changes and modifications may occur to these embodiments once those skilled in the art understand the basic inventive concepts.

[0059] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A nondestructive detection method for coal ash based on multimodal data fusion, characterized in that: Methods include: Perform multi-angle image acquisition, multi-angle point cloud data acquisition and quality acquisition on the coal to obtain multiple coal images, multiple point cloud data and coal quality; perform coal contour recognition and image quality recognition on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients, wherein the multiple angles include angles on both sides of the coal; Iteratively fitting coal point clouds in the multiple point cloud data according to the multiple coal contours respectively to obtain multiple coal point clouds and multiple fitting accuracy coefficients, wherein iteratively fitting point clouds are adjusted for the multiple coal contours respectively according to the multiple image quality coefficients; constructing a three-dimensional coal model according to the plurality of coal point clouds, processing to obtain coal volume information, and calculating to obtain coal density according to the coal volume information and coal mass; According to the multiple fitting accuracy coefficients and the multiple image quality coefficients, the ash analysis accuracy is calculated and configured, and an integrated ash analysis is performed on the coal density to obtain the ash information of the coal.

2. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 1 is characterized in that: Performing coal contour recognition and image quality recognition on the multiple coal images to obtain multiple coal contours and multiple image quality coefficients includes: Pre-training a coal image recognition channel, wherein the coal image recognition channel includes a coal contour recognition path and a noise point recognition branch; Inputting the plurality of coal images into the coal image recognition channel respectively, and identifying and obtaining a plurality of coal contours and a plurality of noise point ratios; Multiple image quality coefficients are obtained by subtracting the multiple noise point ratios from 1.

3. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 2 is characterized in that: Pre-trained coal image recognition pipeline, including: According to the coal detection data in the historical time, a set of sample coal images is collected, and the proportion of coal contours and noise pixels in each sample coal image is marked to obtain a set of sample coal contours and a set of sample noise ratios; The sample coal image set is used as input data, the sample coal contour set and the sample noise point ratio set are used as supervision data, and based on the convolutional neural network, the coal contour recognition path and the noise point recognition branch are supervised and trained respectively; The coal contour recognition path and noise point recognition branch that have been trained are combined to obtain a coal image recognition channel.

4. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 1, characterized in that: Performing iterative fitting of coal point clouds in the plurality of point cloud data according to the plurality of coal contours respectively to obtain a plurality of coal point clouds and a plurality of fitting accuracy coefficients, including: According to a first coal contour among the multiple coal contours, performing iterative fitting of a coal point cloud in a first point cloud data among the multiple point cloud data, to obtain a first initial fitting coal point cloud and a first initial fitting accuracy coefficient; Subtracting a first image quality coefficient from the multiple image quality coefficients from 1 to obtain a first contour adjustment coefficient, using the first contour adjustment coefficient as a first contour adjustment ratio, and randomly adjusting a contour of the first contour adjustment ratio in the first coal contour to obtain a first adjusted coal contour; Using the first adjusted coal contour, performing iterative fitting of the coal point cloud in the first point cloud data, and obtaining a first adjusted fitting coal point cloud and a first adjusted fitting accuracy coefficient; Continue adjusting the first coal contour and iteratively fitting the coal point cloud until convergence, output a maximum fitting accuracy coefficient and a fitted coal point cloud, and obtain a first coal point cloud and a first fitting accuracy coefficient; Continue to use the second coal contour, adjust the contour according to the second image quality coefficient, and perform iterative fitting of the coal point cloud in the second point cloud data until convergence, so as to obtain the second coal point cloud and the second fitting accuracy coefficient.

5. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 4 is characterized in that: According to a first coal contour in the multiple coal contours, performing iterative fitting of a coal point cloud in a first point cloud data in the multiple point cloud data to obtain a first initial fitting coal point cloud and a first initial fitting accuracy coefficient, comprising: According to the first coal contour, a random frame selection is performed in the first point cloud data to obtain a first frame-selected coal point cloud; Calculate the ratio of the point cloud area ratio in the first frame-selected coal point cloud to the first coal contour area as the first frame-selection accuracy coefficient; Continue to use the first coal contour, perform random box selection in the first point cloud data, and perform iterative box selection fitting of the coal point cloud until the convergent box selection fitting times are reached, and output the maximum box selection accuracy coefficient and the corresponding box selection coal point cloud as the first initial fitting accuracy coefficient and the first initial fitting coal point cloud.

6. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 1, characterized in that: According to the plurality of coal point clouds, a three-dimensional coal model is constructed, and the coal volume information is obtained by processing, and according to the coal volume information and the coal mass, the coal density is calculated and obtained, including: constructing a three-dimensional coal model according to a first coal point cloud and a second coal point cloud in the plurality of coal point clouds; Calculating and obtaining coal volume information according to the three-dimensional coal model; The ratio of the coal mass to the coal volume information is calculated to obtain the coal density.

7. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 1, characterized in that: According to the multiple fitting accuracy coefficients and the multiple image quality coefficients, the ash analysis accuracy is calculated and configured, and the integrated ash analysis is performed on the coal density to obtain the ash information of the coal, including: Calculating the mean of the plurality of fitting accuracy coefficients to obtain an average fitting accuracy coefficient, and calculating the mean of the plurality of image quality coefficients to obtain an average image quality coefficient; Subtract the average fitting accuracy coefficient and the average image quality coefficient from 1 to obtain a fitting error coefficient and an image error coefficient, and add them together to obtain an error coefficient; Training an ash analysis channel including P ash analysis branches, where P is an integer greater than or equal to 1; The error coefficient is multiplied by P and rounded to obtain Q, where Q is an integer greater than or equal to 1 and less than P; The coal density is input into Q randomly selected ash analysis branches, and the Q branch coal ashes are obtained as output, and the mean is calculated to obtain the ash information of the coal.

8. The method for nondestructive detection of coal ash by multimodal data fusion according to claim 7, characterized in that: The training includes an ash analysis channel of P ash analysis branches, including: According to the historical test data of coal ash, a sample coal density set and a sample coal ash set are collected as supervised training data for ash analysis; Randomly select P pieces of data from the ash analysis supervised training data with replacement to obtain P pieces of branch supervised training data; The P branches are supervised and trained with the P ash analysis branches respectively, wherein the loss function of the supervised training of each ash analysis branch is as follows: ； Among them, LOSS is the loss, M is the number of supervised training data in each branch supervised training data, is the sample coal ash content in the i-th group of supervised training data, The coal ash output for the ash analysis branch; The P ash analysis branches that have been trained are combined to obtain an ash analysis channel.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the steps of the coal ash non-destructive detection method using multimodal data fusion as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, which, when executed by a processor, implements the steps of the method for nondestructive detection of coal ash using multimodal data fusion as described in any one of claims 1 to 8.

Images