Rapid detection method and system of tailing coal ash based on state space cross fusion

Abstract

The present invention discloses a method and system for rapid detection of tail coal ash based on state space cross fusion, the method comprising: obtaining spectral data of a coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture is a plurality of groups of coal slime mixtures with different ash values ​​and different concentrations; constructing a time series image multimodal cross ash detection model based on dual-branch multimodal state space feature extraction; performing tail coal ash detection processing on the spectral data of the coal slime mixture and the image data of the coal slime mixture based on the time series image multimodal cross ash detection model, and obtaining a coal slime mixture ash detection result. By using the present invention, adjacent feature extraction modeling is performed on spectral data and image data through a state space model, which can improve the efficiency and accuracy of tail coal ash detection. As a method and system for rapid detection of tail coal ash based on state space cross fusion, the present invention can be widely used in the technical field of tail coal ash detection.

Description

Technical Field

[0001] The present invention relates to the technical field of tail coal ash detection, and in particular to a method and system for rapid tail coal ash detection based on state space cross fusion. Background Art

[0002] Coal slime flotation is an important part of the coal production process. In the traditional flotation process, ash detection mainly includes manual sampling and testing methods and rapid ash methods. After collecting samples from the production line, the ash value is calculated by measuring the mass ratio before and after combustion. These methods rely on manual experience and have poor timeliness, and cannot guide flotation production in a timely manner. The existing online measurement methods for ash detection mainly include radioactive ash measurement method, photoelectric ash measurement method and image processing ash measurement method. Among them, the equipment required for radioactive ash measurement method is expensive, and there are radioactive safety issues during production and operation, which makes it difficult to promote and use; the photoelectric ash measurement method is cheap, but it cannot adapt to complex production environments, and the measurement accuracy is low, which will cause considerable losses to coal enterprises; the image processing ash measurement method only detects ash through the surface image or bubble image of the coal slime mixture, and cannot fully extract the internal image of the mixture and the intrinsic relationship between the tail coal ash content. There are certain disadvantages, and it requires artificial design features. The features selected for different working conditions may be different, and the versatility of the method is poor. Different working conditions may also affect the extraction of preset features, and have high requirements on the image acquisition environment, such as foam state, background impurities, shooting angle, amount of foaming agent, etc. The adaptability and robustness are poor, and there is randomness in the sedimentation process of the suspension. The time required and results of each sedimentation are somewhat different, which leads to low model detection accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a method and system for rapid detection of tail coal ash based on state space cross fusion, which can improve the efficiency and accuracy of tail coal ash detection by performing adjacent feature extraction modeling on spectral data and image data through the state space model.

[0004] The first technical solution adopted by the present invention is: a rapid detection method for tail coal ash based on state space cross fusion, comprising the following steps:

[0005] Acquiring spectral data of a coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture comprises several groups of coal slime mixtures having different ash values ​​and different concentrations;

[0006] Based on dual-branch multimodal state space feature extraction, a multimodal cross-gray detection model for time series images is constructed;

[0007] Based on the time series image multimodal cross ash detection model, the spectral data of the coal slime mixture and the image data of the coal slime mixture are processed to detect the tail coal ash content and obtain the coal slime mixture ash detection result.

[0008] Furthermore, the time series image multimodal cross-gray detection model includes a time series feature extraction branch, an image feature extraction branch, a feature fusion module and an ash detection module, wherein the time series feature extraction branch and the image feature extraction branch constitute the dual-branch multimodal state space feature extraction, the output end of the time series feature extraction branch is connected to the first input end of the feature fusion module, the output end of the image feature extraction branch is connected to the second input end of the feature fusion module, and the output end of the feature fusion module is connected to the input end of the ash detection module.

[0009] Further, the step of performing tail coal ash detection processing on the spectral data of the coal slime mixture and the image data of the coal slime mixture based on the time series image multimodal cross ash detection model to obtain the ash detection result of the coal slime mixture specifically includes:

[0010] The spectral data of the coal slime mixture and the image data of the coal slime mixture are respectively input into the time series image multimodal cross ash detection model;

[0011] Based on the time series feature extraction branch of the time series image multimodal cross ash detection model, the spectral data of the coal slime mixture is subjected to time series feature extraction processing to obtain the time series features of the coal slime mixture;

[0012] Based on the image feature extraction branch of the time series image multimodal cross ash detection model, the image feature extraction process is performed on the image data of the coal slime mixture to obtain the image features of the coal slime mixture;

[0013] Based on the feature fusion module of the time series image multimodal cross ash detection model, the time series features of the coal slime mixture and the image features of the coal slime mixture are fused to obtain the fusion features of the coal slime mixture;

[0014] The ash detection module based on the time series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain the ash detection result of the coal slime mixture.

[0015] Furthermore, the time series feature extraction branch based on the time series image multimodal cross ash detection model performs time series feature extraction processing on the spectral data of the coal slime mixture to obtain the time series features of the coal slime mixture, which specifically includes:

[0016] The spectral data of the coal slime mixture is input into the time series feature extraction branch of the time series image multimodal cross ash detection model;

[0017] Based on the position embedding module of the temporal feature extraction branch, the spectral data of the coal slime mixture is subjected to flattening operation, classification mark adding operation and position encoding operation in sequence to obtain the encoding matrix of the coal slime mixture;

[0018] The state space long sequence feature extraction module based on the time series feature extraction branch performs long sequence feature extraction processing on the coding matrix of the coal slime mixture to obtain the long sequence features of the coal slime mixture;

[0019] Based on the root mean square normalization module of the time series feature extraction branch, the long sequence features of the coal slime mixture are processed by mean square deviation normalization operation to obtain the time series features of the coal slime mixture.

[0020] Furthermore, the state space long sequence feature extraction module based on the time series feature extraction branch performs long sequence feature extraction processing on the coding matrix of the coal slime mixture to obtain the long sequence feature of the coal slime mixture, which specifically includes:

[0021] Inputting the coding matrix of the coal slime mixture into the state space long sequence feature extraction module of the time series feature extraction branch;

[0022] Based on the normalization layer and the first linear mapping layer of the state space long sequence feature extraction module, the coding matrix of the coal slime mixture is normalized and mapped to obtain the normalized coding matrix of the coal slime mixture;

[0023] The local feature extraction module based on the state space long sequence feature extraction module sequentially performs local feature extraction processing, nonlinear operation processing and hidden state calculation processing on the normalized coding matrix of the coal slime mixture to obtain the local features of the coal slime mixture;

[0024] The global feature extraction module based on the state space long sequence feature extraction module performs Silu function nonlinear processing on the normalized coding matrix of the coal slime mixture to obtain the global features of the coal slime mixture;

[0025] The local features of the coal slime mixture are fused with the global features of the coal slime mixture to obtain fused feature information of the coal slime mixture;

[0026] Based on the second linear mapping layer of the state space long sequence feature extraction module, the fused coal slime mixture feature information is linearly mapped to obtain the long sequence features of the coal slime mixture.

[0027] Furthermore, the image feature extraction branch based on the time series image multimodal cross ash detection model performs image feature extraction processing on the image data of the coal slime mixture to obtain the image features of the coal slime mixture, which specifically includes:

[0028] The image data of the coal-slime mixture is input into the image feature extraction branch of the time-series image multimodal cross ash detection model;

[0029] Based on the position embedding module of the image feature extraction branch, the image data of the coal slime mixture is subjected to segmentation operation processing, flattening operation processing and position encoding processing in sequence to obtain the image encoding matrix of the coal slime mixture;

[0030] The state space bidirectional sequence feature extraction module based on the image feature extraction branch performs bidirectional sequence feature extraction processing on the image coding matrix of the coal slime mixture to obtain the bidirectional sequence features of the coal slime mixture;

[0031] Based on the root mean square normalization module of the image feature extraction branch, the bidirectional sequence features of the coal slime mixture are processed by the mean square deviation normalization operation to obtain the image features of the coal slime mixture.

[0032] Furthermore, the state space bidirectional sequence feature extraction module based on the image feature extraction branch performs bidirectional sequence feature extraction processing on the image coding matrix of the coal slime mixture to obtain the bidirectional sequence features of the coal slime mixture, which specifically includes:

[0033] The image coding matrix of the coal slime mixture is input into the state space bidirectional sequence feature extraction module of the image feature extraction branch;

[0034] Based on the normalization layer and the first linear mapping layer of the state space bidirectional sequence feature extraction module, the image coding matrix of the coal slime mixture is subjected to normalization mapping processing to obtain a normalized image coding matrix of the coal slime mixture;

[0035] Based on the first branch module of the state space bidirectional sequence feature extraction module, the normalized image encoding matrix of the coal slime mixture is sequentially subjected to one-dimensional convolution to extract positive local features and hidden state calculation to obtain positive local features of the coal slime mixture;

[0036] The second branch module based on the state space bidirectional sequence feature extraction module sequentially performs one-dimensional convolution extraction and hidden state calculation on the normalized image encoding matrix of the coal slime mixture to obtain the reverse local features of the coal slime mixture;

[0037] The third branch module based on the state space bidirectional sequence feature extraction module performs Silu function nonlinear processing on the normalized image encoding matrix of the coal slime mixture to obtain the global image features of the coal slime mixture;

[0038] The forward local features of the coal slime mixture, the reverse local features of the coal slime mixture and the global features of the image of the coal slime mixture are subjected to feature fusion processing to obtain image feature information of the coal slime mixture;

[0039] Based on the second linear mapping layer and root mean square normalization layer of the state space bidirectional sequence feature extraction module, the image feature information of the coal slime mixture is processed by residual addition operation, root mean square normalization operation and regularization operation in sequence to obtain the bidirectional sequence features of the coal slime mixture.

[0040] Furthermore, the feature fusion module based on the time series image multimodal cross ash detection model fuses the time series features of the coal slime mixture with the image features of the coal slime mixture to obtain the fusion features of the coal slime mixture, which specifically includes:

[0041] Inputting the time series features of the coal slime mixture and the image features of the coal slime mixture into a feature fusion module of a time series image multimodal cross ash detection model;

[0042] Based on the attention mechanism module of the feature fusion module, QKV matrix calculation is performed on the time series features of the coal slime mixture and the image features of the coal slime mixture respectively, and the calculation results are cross-calculated to obtain the cross-calculated time series feature QKV matrix and the cross-calculated image feature QKV matrix;

[0043] Based on the feedforward neural network module of the feature fusion module, nonlinear processing is performed on the crossed time series feature QKV matrix and the crossed image feature QKV matrix respectively, so as to obtain the nonlinearly processed time series feature QKV matrix and the nonlinearly processed image feature QKV matrix;

[0044] Based on the self-attention module of the feature fusion module, the time series feature QKV matrix after nonlinear processing and the image feature QKV matrix after nonlinear processing are weighted and fused to obtain the fusion features of the coal slime mixture.

[0045] Furthermore, the ash detection module based on the time series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain the ash detection result of the coal slime mixture, which specifically includes:

[0046] The fusion features of the coal slime mixture are input into the ash detection module of the time series image multimodal cross ash detection model;

[0047] Based on the multi-layer perceptron layer of the ash detection module, nonlinear mapping is performed on the fusion features of the coal slime mixture to obtain classification features of the coal slime mixture with classification categories;

[0048] Based on the Softmax function of the ash detection module, the classification features of the coal slime mixture with classification categories are vectorized and processed to obtain the ash detection results of the coal slime mixture.

[0049] The second technical solution adopted by the present invention is: a rapid detection system for tail coal ash based on state space cross fusion, comprising:

[0050] The first module is used to obtain spectral data of the coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture is a plurality of groups of coal slime mixtures with different ash values ​​and different concentrations;

[0051] The second module is used to construct a multimodal cross-gray detection model for time series images based on dual-branch multimodal state space feature extraction;

[0052] The third module is used to perform tail coal ash detection processing on the spectral data of the coal slime mixture and the image data of the coal slime mixture based on the time series image multimodal cross ash detection model to obtain the ash detection result of the coal slime mixture.

[0053] The beneficial effects of the method and system of the present invention are as follows: the present invention can be deployed to a working site to collect data in real time by acquiring spectral data and image data of a coal slime mixture, without the need for human intervention in the detection project, thereby weakening dependence on experience, and then based on dual-branch multimodal state space feature extraction, a time series image multimodal cross-ash detection model is constructed, and the spectral data and image data are fused, so that the model can effectively learn the absorption spectra and image characteristics and laws of coal slime flotation tailings with different ash contents, and can correlate and model data at adjacent moments through a state space model, thereby realizing rapid detection of coal slime flotation tailings ash values ​​and improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of the steps of the method for rapid detection of tail coal ash content based on state space cross fusion of the present invention;

[0055] Figure 2 It is a structural block diagram of the tail coal ash rapid detection system based on state space cross fusion of the present invention;

[0056] Figure 3 It is a schematic diagram of a coal slime mixed liquid collection device constructed according to a specific embodiment of the present invention;

[0057] Figure 4 is a schematic diagram of comparison of mixed liquid images with different ash contents extracted in a specific embodiment of the present invention;

[0058] Figure 5 is a schematic diagram of a multimodal cross-gray detection model for time series images constructed according to a specific embodiment of the present invention;

[0059] Figure 6 is a schematic diagram of a time series feature extraction branch constructed according to a specific embodiment of the present invention;

[0060] Figure 7 is a schematic diagram of an image feature extraction branch constructed according to a specific embodiment of the present invention;

[0061] Figure 8 is a schematic diagram of a feature fusion module constructed according to a specific embodiment of the present invention;

[0062] Fig. 9 It is a schematic diagram of an ash detection module constructed according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] There are some deficiencies in the related technology, such as:

[0065] 1) The internal image of the coal slime mixture is collected through a quartz observation window of the flowing coal slime mixture, and the backlight source at the same angle is used for lighting. There are no problems such as reflection and light source angle deviation. Compared with the surface image and bubble image, its random contingency is greatly reduced, and the possibility of being affected by the working conditions is also reduced.

[0066] 2) Existing ash measurement methods rely on a single input to determine the ash content. Once the input fluctuates or has errors, the results will also be wrong.

[0067] 3) The images collected by the existing ash content measurement methods only represent the static features of the mixed liquid at a certain moment. The two images collected continuously have a certain correlation, but they are independent of each other in the feature extraction process of CNN, that is, they do not consider that there may be a certain relationship between images at similar moments, which has a certain impact on the accuracy of ash content detection.

[0068] 4) The spectral data collected by the existing ash measurement methods have a long time span, there are many uncontrollable factors in the collection process, and the detection process is offline, which cannot guide the flotation work in real time.

[0069] Based on this, the embodiment of the present invention adopts a micro-spectrometer and an industrial camera to directly collect the absorbance spectrum data and internal image data of the coal slime mixture on the production line. The absorption spectrum data taken is the information at a certain moment, and the longest collection duration does not exceed 2s, which avoids some unstable factors in the collection process, and is real-time online collection, with good timeliness. It is mainly based on spectrum data and supplemented by image data. The source of the input image is more stable and reliable. After training a good ash detection model, ash detection can be performed on coal slime mixtures with unknown ash content. The absorption spectrum and the internal image of the coal slime mixture are used for joint decision-making. Once one of the inputs fluctuates, the other input can also correct the output result in time. Compared with the single-input detection method, the robustness of ash detection is improved. Since the coal slime mixture is in a flowing state during the actual flotation process, the images at the previous and next moments have a certain continuity. Therefore, the embodiment of the present invention uses a linear state space model to extract image features, which not only takes into account the characteristics of each image itself, but also considers the feature correlation between images at adjacent moments, thereby improving the accuracy of ash detection.

[0070] Reference Figure 1 The present invention provides a method for rapid detection of tail coal ash based on state space cross fusion, the method comprising the following steps:

[0071] S100, acquiring spectral data of a coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture comprises a plurality of groups of coal slime mixtures having different ash values ​​and different concentrations;

[0072] S110, preparing coal slime mixtures with different ash values ​​and different concentrations;

[0073] Specifically, in this embodiment, the industrial flotation process is simulated to prepare coal slime mixtures with different ash values ​​and different concentrations. First, a number of coal slime mixtures with a known ash value of 25.6% are prepared, the weighing instrument is calibrated for the current horizontal plane, the weighing paper is placed in the weighing instrument for peeling, a certain mass of the coal slime mixture is taken with a medicine spoon and placed on the weighing paper for weighing, and the operation is repeated to weigh 0.25g, 0.5g, 0.75g, 1g, and 1.25g of samples for a total of 5 portions, and the 5 samples are respectively placed in 5 acrylic containers filled with 1000mL of pure water, and a stirring rotor is added to the solution, and a magnetic stirrer is used for pre-stirring, and the above steps are repeated. Figure 4 As shown, 5 portions of coal slime mixtures with different concentrations of ash values ​​of 34.4%, 38.0%, 44.0% and 49.4% were prepared respectively. The specific embodiment of the present invention needs to prepare 20 portions of coal slime mixtures with different ash contents and concentrations.

[0074] S120, constructing a coal slime mixed liquid collection device;

[0075] Specifically, in this embodiment, Figure 3 As shown, two 60cm*60cm anti-vibration optical platforms are spliced ​​and placed, and the optical bracket is fixed to the optical platform; two quartz flow dishes (quartz observation windows) and peristaltic pumps are connected through silicone hoses, and the two flow dishes are fixed through optical brackets and light collimator sleeves; both ends of the silicone hose are placed in an acrylic square container filled with a prepared coal sludge mixture, the silicone hose is fixed with a crab clamp, and the container is placed on a magnetic stirrer; the industrial camera is connected to the extension ring and the lens and fixed on the dovetail groove slide through the optical bracket, and a visual backlight is placed on the other side of the quartz flow dish; the visual backlight is started, and the industrial camera is connected to the computer; the xenon light source, the spectrometer and the collimator are connected with an optical fiber cable, and the spectrometer and the xenon light source are turned on.

[0076] S130, based on the coal slime mixture collection device, obtaining spectral data of the coal slime mixture and image data of the coal slime mixture.

[0077] Specifically, in this embodiment, after the acquisition device is built, modify the parameters on the acquisition program interface, set the industrial camera and spectrum acquisition time and acquisition interval, storage file location and file naming format; take out the optical cable on one side of the collimator connected to the xenon lamp light source and cover it with a black card, click the "Set reference light" button on the software interface, then the spectrometer emits a beeping buzzer sound, and the reference light setting is completed; connect the optical cable to the collimator, and when there is no sample solution in the circulation cuvette, click the "Set background light" button on the software interface, then the spectrometer emits a beeping buzzer sound, the reference light setting is completed, and the spectrum calibration is completed.

[0078] The calculation formula of the absorption spectrum is as follows:

[0079] ;

[0080] In the above formula, represents absorbance, represents the transmittance, represents the sample light, Represents background light, Indicates reference light.

[0081] Adjust the camera position by moving the slide knob, and rotate the aperture ring and focus ring on the lens to adjust the focal length, so that the image on the acquisition program interface is clear and the field of view is focused on the middle section of the circulation dish; put the hose into the coal slime mixture, and start the agitator to stir it evenly at a speed of 350r / min; when it is observed that there are no large-diameter coal slime particles in the container, start the peristaltic pump and run it at a speed of 300ml / min; when the circulation cuvette and the hose are filled with the coal slime mixture and there is no large amount of bubble accumulation in the circulation cuvette, click the "linked acquisition button" on the software interface to start data acquisition. Each time a batch of data is collected, the xenon lamp light source makes a dripping sound; when the acquisition reaches the set time The acquisition program will store the acquired images and spectral data in the set folder; after acquiring data of different ash content or different concentration, put the silicone hose into the container filled with pure water, start the peristaltic pump, and clean the pipe and the inside of the circulation dish; after cleaning, turn off the peristaltic pump, remove the pure water container, and complete one data acquisition; repeat the above steps to acquire coal slime mixtures of different ash content and different concentrations. In the specific embodiment of the present invention, 20 groups of data need to be acquired, each group of data takes about 1 hour, and each group of data contains more than 700 spectral data and more than 2800 image data, a total of 14000 spectral data and 56000 coal slime mixture images.

[0082] S200, based on dual-branch multimodal state space feature extraction, construct a time series image multimodal cross gray detection model;

[0083] Specifically, Figure 5 As shown, the time series image multimodal cross-gray detection model includes a time series feature extraction branch, an image feature extraction branch, a feature fusion module and an ash detection module, wherein the time series feature extraction branch and the image feature extraction branch constitute the dual-branch multimodal state space feature extraction, the output end of the time series feature extraction branch is connected to the first input end of the feature fusion module, the output end of the image feature extraction branch is connected to the second input end of the feature fusion module, and the output end of the feature fusion module is connected to the input end of the ash detection module.

[0084] S300, performing tail coal ash detection processing on the spectral data of the coal slime mixture and the image data of the coal slime mixture based on the time series image multimodal cross ash detection model to obtain the ash detection result of the coal slime mixture.

[0085] S310, inputting the spectral data of the coal slime mixture and the image data of the coal slime mixture into a time series image multimodal cross ash detection model respectively;

[0086] Specifically, the embodiment of the present invention proposes a time series-image multimodal cross fusion model. Through two feature extraction branches based on the state space model, image and spectral features are extracted, and then the two modal features are fused through a cross attention mechanism, and finally input into a multi-layer perceptron for gray detection tasks. Compared with the existing detection models, the accuracy and stability are greatly improved.

[0087] S320, based on the time series feature extraction branch of the time series image multimodal cross ash detection model, performing time series feature extraction processing on the spectral data of the coal slime mixture to obtain the time series features of the coal slime mixture;

[0088] Specifically, the spectral data of the coal slime mixture is input into the time series feature extraction branch of the time series image multimodal cross ash detection model; based on the position embedding module of the time series feature extraction branch, the spectral data of the coal slime mixture is flattened, classified mark added and position encoded in sequence to obtain the encoding matrix of the coal slime mixture; based on the state space long sequence feature extraction module of the time series feature extraction branch, the encoding matrix of the coal slime mixture is subjected to long sequence feature extraction to obtain the long sequence features of the coal slime mixture; based on the root mean square normalization module of the time series feature extraction branch, the long sequence features of the coal slime mixture are subjected to mean square deviation normalization to obtain the time series features of the coal slime mixture.

[0089] Among them, the coding matrix of the coal slime mixture is input into the state space long sequence feature extraction module of the time series feature extraction branch; based on the normalization layer and the first linear mapping layer of the state space long sequence feature extraction module, the coding matrix of the coal slime mixture is normalized and mapped to obtain the normalized coding matrix of the coal slime mixture; based on the local feature extraction module of the state space long sequence feature extraction module, the coding matrix of the normalized coal slime mixture is sequentially subjected to local feature extraction processing, nonlinear operation processing and hidden state calculation processing to obtain the local features of the coal slime mixture; based on the global feature extraction module of the state space long sequence feature extraction module, the coding matrix of the normalized coal slime mixture is subjected to Silu function nonlinear processing to obtain the global features of the coal slime mixture; the local features of the coal slime mixture and the global features of the coal slime mixture are subjected to feature fusion processing to obtain the fused feature information of the coal slime mixture; based on the second linear mapping layer of the state space long sequence feature extraction module, the fused feature information of the coal slime mixture is subjected to linear mapping processing to obtain the long sequence features of the coal slime mixture.

[0090] In this embodiment, if Figure 6As shown, a set of spectral data collected contains 2088 floating point numbers from 0 to 5, and 16 sets of data are batched as input into the time series feature extraction model. The embodiment of the present invention proposes a long sequence feature extraction model based on state space, which performs well in long sequence context modeling, can effectively extract local features and global features in the spectral sequence for multi-scale fusion, and improves the feature extraction efficiency.

[0091] The input is embedded in position. First, each set of data is flattened sequentially to transform the input data into Then, after exchanging the first and second dimensions, a 16-dimensional matrix is ​​concatenated based on the original vector. The classification token (tag) is used for subsequent classification and detection operations. At this time, the matrix becomes Finally, the model generates The encoding matrix is ​​used by the model to identify the order of each group of data input, and the input matrix and the position encoding matrix are added to complete the position encoding operation. The output matrix size at this time is .

[0092] Will The matrix is ​​input into the long sequence feature extraction module based on state space. First, 32 groups of size The one-dimensional sequence normalization operation does not change the matrix size. Then the normalized matrix is ​​mapped to two different , subspace, where The matrix first extracts the local feature information in each set of time series data through one-dimensional convolution, and gives the model nonlinear characteristics through the Silu function, which is expressed as:

[0093] ;

[0094] In the above formula, represents Silu function, The local feature extraction module represents the long sequence feature extraction module based on the state space

[0095] Then the convolution result is input into the SSM state space model for hidden state The calculation retains the important information of each input and models the correlation between the two previous and subsequent inputs. The matrix is ​​a trainable parameter, and its expression is:

[0096] ;

[0097] In the above formula, Represents the hidden state of the SSM state-space model.

[0098] for The matrix is ​​processed nonlinearly through Silu function to retain the global feature information of the absorption spectrum data, and then the global features and local features are fused at multiple scales. Finally, linear mapping is performed to output In the time series feature extraction model, the above process is performed 12 times in total. The size of the output matrix after multiple state space-based long sequence feature extraction modules is Finally, perform a mean square error normalization operation to separate the 16 processed classification tokens as the features of this round of data. The feature matrix size is , where the expression of the mean square error normalization operation is:

[0099] ;

[0100] In the above formula, Represents the mean square error normalization operation.

[0101] S330, performing image feature extraction processing on the image data of the coal slime mixture based on the image feature extraction branch of the time series image multimodal cross ash detection model to obtain image features of the coal slime mixture;

[0102] Specifically, the image data of the coal slime mixture is input into the image feature extraction branch of the time series image multimodal cross ash detection model; based on the position embedding module of the image feature extraction branch, the image data of the coal slime mixture is sequentially subjected to segmentation operation, flattening operation and position encoding operation to obtain the image encoding matrix of the coal slime mixture; based on the state space bidirectional sequence feature extraction module of the image feature extraction branch, the image encoding matrix of the coal slime mixture is subjected to bidirectional sequence feature extraction to obtain the bidirectional sequence features of the coal slime mixture; based on the root mean square normalization module of the image feature extraction branch, the bidirectional sequence features of the coal slime mixture are subjected to mean square deviation normalization operation to obtain the image features of the coal slime mixture.

[0103] Among them, the image coding matrix of the coal slime mixture is input into the state space bidirectional sequence feature extraction module of the image feature extraction branch; based on the normalization layer and the first linear mapping layer of the state space bidirectional sequence feature extraction module, the image coding matrix of the coal slime mixture is normalized and mapped to obtain the normalized image coding matrix of the coal slime mixture; based on the first branch module of the state space bidirectional sequence feature extraction module, the image coding matrix of the normalized coal slime mixture is sequentially subjected to one-dimensional convolution to extract positive local features and hidden state calculation to obtain the positive local features of the coal slime mixture; based on the second branch module of the state space bidirectional sequence feature extraction module, the image coding matrix of the normalized coal slime mixture is sequentially subjected to one-dimensional convolution to extract negative local features and hidden state calculation Calculate and obtain the reverse local features of the coal slime mixture; based on the third branch module of the state-space bidirectional sequence feature extraction module, perform Silu function (Sigmoid linear unit function) nonlinear processing on the normalized image coding matrix of the coal slime mixture to obtain the global image features of the coal slime mixture; perform feature fusion processing on the forward local features of the coal slime mixture, the reverse local features of the coal slime mixture and the global image features of the coal slime mixture to obtain the image feature information of the coal slime mixture; based on the second linear mapping layer and the root mean square normalization layer of the state-space bidirectional sequence feature extraction module, perform residual addition operation processing, root mean square normalization operation processing and regularization operation processing on the image feature information of the coal slime mixture in turn to obtain the bidirectional sequence features of the coal slime mixture.

[0104] In this embodiment, if Figure 7 As shown, the pixel size of a collected image data is RGB images are taken and 32 groups of data are grouped as input into the image feature extraction model. The embodiment of the present invention proposes a state-space-based bidirectional sequence feature extraction model, which performs bidirectional feature extraction on local information of the image to enhance the representation ability of the model, and finally introduces residual fusion global features to improve the expression ability of the network, and has a better feature extraction effect on high-resolution images.

[0105] Perform a center cropping operation on the input image and cut the image into Size, and then perform PatchEmbedding (region embedding) operation. First, the convolution kernel is , stride (step length) is , a two-dimensional convolution operation with padding 0, The image is divided into 256 pieces of size , flatten these small images and map them to a space of size 1044, and swap the second and third dimensions. At this time, the input size is ; Then concatenate 32 vectors of size The classification token is used as the basis for subsequent classification. Finally, the position encoding matrix is ​​generated by the model and added to the original input matrix at the corresponding position. Finally, the entire Patch Embedding operation is completed and an output is The vector is then used for subsequent operations.

[0106] Will The vector input is used to extract features in the state-space based bidirectional sequence feature extraction module. First, the input is normalized and the data is scaled to between 0 and 1 to facilitate subsequent operations and prevent overfitting of the model. Secondly, the normalized vector is linearly mapped to three subspaces: x, y, and z. The x vector is subjected to one-dimensional convolution to extract the positive local features, and the local features are input into the forward SSM to calculate the hidden state, memorize the hidden state of each input, and infer the output expression of the next state; similarly, the y vector is first subjected to a flip operation and then subjected to one-dimensional convolution to extract the reverse local features; the z vector directly obtains nonlinear features through the silu function, and finally, the features of x, z and y, z at different scales are fused to improve the expression ability of the model. The fused features are added with residuals to form a vector feature with a length of 1044, and the vector is subjected to root mean square normalization. Finally, the DropPath (path discarding) operation is appended, and the DropPath (path discarding) rate is 0.1, which randomly discards some results on the calculation path to reduce the amount of calculation of the model. The vector after passing through the feature extraction module becomes 32*257*1044.

[0107] After 22 bidirectional sequence feature extraction modules, the output of the model is , the vector is input into the RMS normalization layer, the output dimension remains unchanged, and finally the classification tag token embedded at the beginning is extracted separately as the feature vector extracted from the 32 pictures in this round, with a size of .

[0108] S340, a feature fusion module based on a time series image multimodal cross ash detection model, fusing the time series features of the coal slime mixture with the image features of the coal slime mixture to obtain fused features of the coal slime mixture;

[0109] Specifically, the time series features of the coal slime mixture and the image features of the coal slime mixture are input into the feature fusion module of the time series image multimodal cross ash detection model; based on the attention mechanism module of the feature fusion module, the QKV matrix calculation is performed on the time series features of the coal slime mixture and the image features of the coal slime mixture respectively, and the calculation results are cross-operated to obtain the cross-time series feature QKV matrix and the cross-image feature QKV matrix; based on the feedforward neural network module of the feature fusion module, the cross-time series feature QKV matrix and the cross-image feature QKV matrix are nonlinearly processed to obtain the nonlinearly processed time series feature QKV matrix and the nonlinearly processed image feature QKV matrix; based on the self-attention module of the feature fusion module, the nonlinearly processed time series feature QKV matrix and the nonlinearly processed image feature QKV matrix are weighted and fused to obtain the fused features of the coal slime mixture.

[0110] In this embodiment, if Figure 8 As shown in the figure, this patent proposes a feature fusion model based on cross attention. The QKV matrix of two feature inputs is calculated through the attention mechanism, and the QKV matrices of the two features are cross-calculated to achieve the purpose of cross fusion. It pays more attention to the most important parts of the two features and fuses them, solving the problem of feature redundancy caused by direct splicing and the problem of too long feature vector after fusion. The image features of are spliced ​​in pairs in the zero dimension (the interval between the acquisition of two adjacent pictures is only 1s, so it can be considered that the features in the two images are closely related), and become The characteristics of the time series extraction branch are obtained After the attention calculation of the features, the cross attention operation is performed, and its expression is:

[0111] ;

[0112] Then the time series part generates the query matrix Q1, and the image part generates the key matrix K2 and the value matrix V2 for attention operation; the image part generates the query matrix Q2, and the time series part generates the key matrix K1 and the value matrix V1 for attention operation. Then they are input into the feedforward neural network respectively to give nonlinear operation, and the expression is:

[0113] ;

[0114] The vector after nonlinear processing is then subjected to self-attention operation to optimize the weights of feature fusion in the forward propagation process. Finally, after passing through a layer of feedforward neural network, the two features are completely fused into characteristics.

[0115] S350, an ash detection module based on a time series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain an ash detection result of the coal slime mixture.

[0116] Specifically, the fusion features of the coal slime mixture are input into the ash detection module of the time series image multimodal cross ash detection model; based on the multi-layer perceptron layer of the ash detection module, the fusion features of the coal slime mixture are nonlinearly mapped to obtain the classification features of the coal slime mixture with classification categories; based on the Softmax function of the ash detection module, the classification features of the coal slime mixture with classification categories are vector converted to obtain the ash detection results of the coal slime mixture.

[0117] In this embodiment, if Fig. 9 As shown, the size after fusion is The features are input into the multi-layer perceptron, and the input space is mapped to the output category space through a nonlinear mapping relationship, and the features of the classification category are extracted. Finally, the 5-dimensional vector is converted into a 5-dimensional probability vector through the Softmax function. The value of each element in the vector is between 0 and 1, and the sum of all elements is 1. The expression is:

[0118] ;

[0119] In the above formula, Represents the Softmax function.

[0120] Reference Figure 2 , a rapid detection system for tail coal ash based on state space cross fusion, including:

[0121] The first module 201 is used to obtain spectral data of a coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture is a plurality of groups of coal slime mixtures having different ash values ​​and different concentrations;

[0122] The second module 202 is used to construct a time series image multimodal cross gray detection model based on dual-branch multimodal state space feature extraction;

[0123] The third module 203 is used to perform tail coal ash detection processing on the spectral data of the coal slime mixture and the image data of the coal slime mixture based on the time series image multimodal cross ash detection model to obtain the ash detection result of the coal slime mixture.

[0124] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A rapid detection method for tail coal ash based on state space cross fusion, characterized in that: The following steps are involved: Acquiring spectral data of a coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture comprises several groups of coal slime mixtures having different ash values ​​and different concentrations; Based on dual-branch multimodal state space feature extraction, a multimodal cross-gray detection model for time series images is constructed; The time series image multimodal cross-gray detection model includes a time series feature extraction branch, an image feature extraction branch, a feature fusion module and an ash detection module, wherein the time series feature extraction branch and the image feature extraction branch constitute the dual-branch multimodal state space feature extraction, the output end of the time series feature extraction branch is connected to the first input end of the feature fusion module, the output end of the image feature extraction branch is connected to the second input end of the feature fusion module, and the output end of the feature fusion module is connected to the input end of the ash detection module; The spectral data of the coal slime mixture and the image data of the coal slime mixture are respectively input into the time series image multimodal cross ash detection model; The spectral data of the coal slime mixture is input into the time series feature extraction branch of the time series image multimodal cross ash detection model; Based on the position embedding module of the temporal feature extraction branch, the spectral data of the coal slime mixture is subjected to flattening operation, classification mark adding operation and position encoding operation in sequence to obtain the encoding matrix of the coal slime mixture; The state space long sequence feature extraction module based on the time series feature extraction branch performs long sequence feature extraction processing on the coding matrix of the coal slime mixture to obtain the long sequence features of the coal slime mixture; Based on the root mean square normalization module of the time series feature extraction branch, the long sequence features of the coal slime mixture are processed by the mean square deviation normalization operation to obtain the time series features of the coal slime mixture; The image data of the coal-slime mixture is input into the image feature extraction branch of the time-series image multimodal cross ash detection model; Based on the position embedding module of the image feature extraction branch, the image data of the coal slime mixture is subjected to segmentation operation processing, flattening operation processing and position encoding processing in sequence to obtain the image encoding matrix of the coal slime mixture; The state space bidirectional sequence feature extraction module based on the image feature extraction branch performs bidirectional sequence feature extraction processing on the image coding matrix of the coal slime mixture to obtain the bidirectional sequence features of the coal slime mixture; Based on the root mean square normalization module of the image feature extraction branch, the bidirectional sequence features of the coal slime mixture are processed by the root mean square normalization operation to obtain the image features of the coal slime mixture; Inputting the time series features of the coal slime mixture and the image features of the coal slime mixture into a feature fusion module of a time series image multimodal cross ash detection model; Based on the attention mechanism module of the feature fusion module, QKV matrix calculation is performed on the time series features of the coal slime mixture and the image features of the coal slime mixture respectively, and the calculation results are cross-calculated to obtain the cross-calculated time series feature QKV matrix and the cross-calculated image feature QKV matrix; Among them, the data processing process of the attention mechanism module is: Obtained by the image feature extraction branch The image features of are concatenated two by two in the zero dimension, becoming The features obtained by extracting the temporal features After the attention calculation is performed on the features, a cross attention operation is performed. Then the time series part generates the query matrix Q1, and the image part generates the key matrix K2 and the value matrix V2 for attention operation; the image part generates the query matrix Q2, and the time series part generates the key matrix K1 and the value matrix V1 for attention operation; Based on the feedforward neural network module of the feature fusion module, nonlinear processing is performed on the crossed time series feature QKV matrix and the crossed image feature QKV matrix respectively, so as to obtain the nonlinearly processed time series feature QKV matrix and the nonlinearly processed image feature QKV matrix; Based on the self-attention module of the feature fusion module, the time series feature QKV matrix after nonlinear processing and the image feature QKV matrix after nonlinear processing are weighted and fused to obtain the fusion features of the coal slime mixture; The ash detection module based on the time series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain the ash detection result of the coal slime mixture.

2. The rapid detection method for tail coal ash based on state space cross fusion according to claim 1 is characterized in that: The state space long sequence feature extraction module based on the time series feature extraction branch performs long sequence feature extraction processing on the coding matrix of the coal slime mixture to obtain the long sequence feature of the coal slime mixture. The step specifically includes: Inputting the coding matrix of the coal slime mixture into the state space long sequence feature extraction module of the time series feature extraction branch; Based on the normalization layer and the first linear mapping layer of the state space long sequence feature extraction module, the coding matrix of the coal slime mixture is normalized and mapped to obtain the normalized coding matrix of the coal slime mixture; The local feature extraction module based on the state space long sequence feature extraction module sequentially performs local feature extraction processing, nonlinear operation processing and hidden state calculation processing on the normalized coding matrix of the coal slime mixture to obtain the local features of the coal slime mixture; The global feature extraction module based on the state space long sequence feature extraction module performs Silu function nonlinear processing on the normalized coding matrix of the coal slime mixture to obtain the global features of the coal slime mixture; The local features of the coal slime mixture are fused with the global features of the coal slime mixture to obtain fused feature information of the coal slime mixture; Based on the second linear mapping layer of the state space long sequence feature extraction module, the fused coal slime mixture feature information is linearly mapped to obtain the long sequence features of the coal slime mixture.

3. The rapid detection method of tail coal ash based on state space cross fusion according to claim 2 is characterized in that: The state space bidirectional sequence feature extraction module based on the image feature extraction branch performs bidirectional sequence feature extraction processing on the image coding matrix of the coal slime mixture to obtain the bidirectional sequence features of the coal slime mixture. The step specifically includes: The image coding matrix of the coal slime mixture is input into the state space bidirectional sequence feature extraction module of the image feature extraction branch; Based on the normalization layer and the first linear mapping layer of the state space bidirectional sequence feature extraction module, the image coding matrix of the coal slime mixture is subjected to normalization mapping processing to obtain a normalized image coding matrix of the coal slime mixture; Based on the first branch module of the state space bidirectional sequence feature extraction module, the normalized image encoding matrix of the coal slime mixture is sequentially subjected to one-dimensional convolution to extract positive local features and hidden state calculation to obtain positive local features of the coal slime mixture; The second branch module based on the state space bidirectional sequence feature extraction module sequentially performs one-dimensional convolution extraction and hidden state calculation on the normalized image encoding matrix of the coal slime mixture to obtain the reverse local features of the coal slime mixture; The third branch module based on the state space bidirectional sequence feature extraction module performs Silu function nonlinear processing on the normalized image encoding matrix of the coal slime mixture to obtain the global image features of the coal slime mixture; The forward local features of the coal slime mixture, the reverse local features of the coal slime mixture and the global features of the image of the coal slime mixture are subjected to feature fusion processing to obtain image feature information of the coal slime mixture; Based on the second linear mapping layer and root mean square normalization layer of the state space bidirectional sequence feature extraction module, the image feature information of the coal slime mixture is processed by residual addition operation, root mean square normalization operation and regularization operation in sequence to obtain the bidirectional sequence features of the coal slime mixture.

4. The rapid detection method of tail coal ash based on state space cross fusion according to claim 3 is characterized in that: The ash detection module based on the time-series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain the ash detection result of the coal slime mixture. This step specifically includes: The fusion features of the coal slime mixture are input into the ash detection module of the time series image multimodal cross ash detection model; Based on the multi-layer perceptron layer of the ash detection module, nonlinear mapping is performed on the fusion features of the coal slime mixture to obtain classification features of the coal slime mixture with classification categories; Based on the Softmax function of the ash detection module, the classification features of the coal slime mixture with classification categories are vectorized and processed to obtain the ash detection results of the coal slime mixture.

5. The rapid detection system of tail coal ash based on state space cross fusion is characterized by: Includes the following modules: The first module is used to obtain spectral data of the coal slime mixture and image data of the coal slime mixture, wherein the coal slime mixture is a plurality of groups of coal slime mixtures with different ash values ​​and different concentrations; The second module is used to construct a multimodal cross-gray detection model for time series images based on dual-branch multimodal state space feature extraction; The time series image multimodal cross-gray detection model includes a time series feature extraction branch, an image feature extraction branch, a feature fusion module and an ash detection module, wherein the time series feature extraction branch and the image feature extraction branch constitute the dual-branch multimodal state space feature extraction, the output end of the time series feature extraction branch is connected to the first input end of the feature fusion module, the output end of the image feature extraction branch is connected to the second input end of the feature fusion module, and the output end of the feature fusion module is connected to the input end of the ash detection module; The third module is used to input the spectral data of the coal slime mixture and the image data of the coal slime mixture into the time series image multimodal cross ash detection model respectively; The spectral data of the coal slime mixture is input into the time series feature extraction branch of the time series image multimodal cross ash detection model; Based on the position embedding module of the temporal feature extraction branch, the spectral data of the coal slime mixture is subjected to flattening operation, classification mark adding operation and position encoding operation in sequence to obtain the encoding matrix of the coal slime mixture; The state space long sequence feature extraction module based on the time series feature extraction branch performs long sequence feature extraction processing on the coding matrix of the coal slime mixture to obtain the long sequence features of the coal slime mixture; Based on the root mean square normalization module of the time series feature extraction branch, the long sequence features of the coal slime mixture are processed by the mean square deviation normalization operation to obtain the time series features of the coal slime mixture; The image data of the coal-slime mixture is input into the image feature extraction branch of the time-series image multimodal cross ash detection model; Based on the position embedding module of the image feature extraction branch, the image data of the coal slime mixture is subjected to segmentation operation processing, flattening operation processing and position encoding processing in sequence to obtain the image encoding matrix of the coal slime mixture; The state space bidirectional sequence feature extraction module based on the image feature extraction branch performs bidirectional sequence feature extraction processing on the image coding matrix of the coal slime mixture to obtain the bidirectional sequence features of the coal slime mixture; Based on the root mean square normalization module of the image feature extraction branch, the bidirectional sequence features of the coal slime mixture are processed by the root mean square normalization operation to obtain the image features of the coal slime mixture; Inputting the time series features of the coal slime mixture and the image features of the coal slime mixture into a feature fusion module of a time series image multimodal cross ash detection model; Based on the attention mechanism module of the feature fusion module, QKV matrix calculation is performed on the time series features of the coal slime mixture and the image features of the coal slime mixture respectively, and the calculation results are cross-calculated to obtain the cross-calculated time series feature QKV matrix and the cross-calculated image feature QKV matrix; Among them, the data processing process of the attention mechanism module is: Obtained by the image feature extraction branch The image features of are concatenated two by two in the zero dimension, becoming The features obtained by extracting the temporal features After the attention calculation is performed on the features, a cross attention operation is performed. Then the time series part generates the query matrix Q1, and the image part generates the key matrix K2 and the value matrix V2 for attention operation; the image part generates the query matrix Q2, and the time series part generates the key matrix K1 and the value matrix V1 for attention operation; Based on the feedforward neural network module of the feature fusion module, nonlinear processing is performed on the crossed time series feature QKV matrix and the crossed image feature QKV matrix respectively, so as to obtain the nonlinearly processed time series feature QKV matrix and the nonlinearly processed image feature QKV matrix; Based on the self-attention module of the feature fusion module, the time series feature QKV matrix after nonlinear processing and the image feature QKV matrix after nonlinear processing are weighted and fused to obtain the fusion features of the coal slime mixture; The ash detection module based on the time series image multimodal cross ash detection model performs tail coal ash detection processing on the fusion features of the coal slime mixture to obtain the ash detection result of the coal slime mixture.

Images