Intelligent coal caving control method and system based on multimodal data

Abstract

The present disclosure proposes an intelligent coal caving control method and system based on multimodal data, the method comprising: acquiring multimodal data; wherein the multimodal data comprises: video data, audio data, vibration data, and tail beam inclination data and plug board travel data of the top coal caving working face; predicting and generating coal caving core parameters according to the multimodal data; wherein the coal caving core parameters at least comprise: coal caving time, target tail beam swing angle, target tail beam swing frequency, target plug board telescopic distance and target plug board telescopic frequency; according to the coal caving core parameters, controlling the hydraulic support to swing the tail beam according to the target tail beam swing angle and target tail beam swing frequency, and the plug board to telescope according to the target plug board telescopic distance and target plug board telescopic frequency within the coal caving time, and performing the top coal caving operation. Thus, the coal gangue state can be automatically identified, and the top coal caving mining can be controlled to automatically caving coal, which is safe and reliable.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent control of coal mine production, and in particular to an intelligent coal caving control method and system based on multimodal data. Background Art

[0002] In related technologies, top coal caving is done manually. Caving technicians observe the coal flow on the rear scraper conveyor through the gaps between hydraulic supports to see if gangue is being released and the amount of gangue being released, and then decide whether to stop caving. This method is labor-intensive, inefficient, and prone to operator errors due to subjective reasons. Summary of the Invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] In a first aspect, an intelligent coal caving control method based on multimodal data is proposed, the method comprising: acquiring multimodal data; wherein the multimodal data comprises: video data, audio data, vibration data, and tail beam inclination data and insert plate stroke data of the top coal caving working face; predicting and generating coal caving core parameters based on the multimodal data; wherein the coal caving core parameters comprise at least: coal caving time, target tail beam swinging angle, target tail beam swinging frequency, target insert plate extension and extension distance, and target insert plate extension and extension frequency; and controlling the hydraulic support to swing the tail beam according to the target tail beam swinging angle and the target tail beam swinging frequency, and to extend and retract the insert plate according to the target insert plate extension and extension frequency within the coal caving time according to the coal caving core parameters, thereby executing the top coal caving operation.

[0005] In some embodiments, the prediction and generation of coal caving core parameters based on the multimodal data includes: performing feature extraction on the multimodal data to obtain a multimodal feature vector; performing multimodal feature fusion on the multimodal feature vector to respectively predict the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target plug plate extension distance and the target plug plate extension frequency.

[0006] In some embodiments, the multimodal data also includes: coal mining machine speed, scraper conveyor power, top coal caving working face pressure data, top coal caving working face inclination length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average interbedded gangue rate, initial coal seam mineability index and gas concentration.

[0007] In some embodiments, the step of extracting features from the multimodal data to obtain multimodal feature vectors includes:

[0008] Inputting the video data into a trained coal gangue recognition model, performing feature extraction on the video data, and obtaining a first coal gangue feature vector;

[0009] Inputting the audio data into a trained voiceprint coal gangue recognition model, performing feature extraction on the audio data, and obtaining a second coal gangue feature vector;

[0010] Inputting the vibration data into a trained vibration frequency coal gangue recognition model, performing feature extraction on the vibration data, and obtaining a third coal gangue feature vector;

[0011] The tail beam inclination angle data, the slide block stroke data, the coal mining machine speed, the scraper conveyor power, the top coal caving face pressure data, the top coal caving face dip length, the initial average coal seam thickness, the initial coal seam dip angle, the initial average coal seam intercalation rate, the initial coal seam mineability index, and the gas concentration are mapped to their respective representation spaces to generate tail beam inclination angle vector, slide block stroke vector, coal mining machine speed vector, scraper conveyor power vector, top coal caving face pressure vector, top coal caving face dip length vector, initial average coal seam thickness vector, initial coal seam dip angle vector, initial average coal seam intercalation rate vector, initial coal seam mineability index vector, and gas concentration vector.

[0012] In some embodiments, the multimodal feature fusion is performed on the multimodal feature vector to respectively predict the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target plugging plate extension distance, and the target plugging plate extension frequency, including:

[0013] In response to identifying the coal and gangue state as a state to be released as determined by at least one of the first coal and gangue feature vector, the second coal and gangue feature vector, and the third coal and gangue feature vector, multimodal feature fusion is performed based on the tail beam inclination angle vector, the insert plate stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving face pressure vector, the top coal caving face dip length vector, the initial coal seam average thickness vector, the initial coal seam dip angle vector, the initial coal seam average gangue ratio vector, the initial coal seam mineability index vector, and the gas concentration vector to predict the coal release time, the target tail beam swing angle, the target tail beam swing frequency, the target insert plate extension distance, and the target insert plate extension frequency, respectively.

[0014] In some embodiments, the method further includes: acquiring a video sample dataset, training a coal gangue recognition model, and generating a trained coal gangue recognition model; acquiring an audio sample dataset, training a voiceprint coal gangue recognition model, and generating a trained voiceprint coal gangue recognition model; and acquiring a vibration sample dataset, training a vibration frequency coal gangue recognition model, and generating a trained vibration frequency coal gangue recognition model.

[0015] A second aspect of this disclosure proposes an intelligent coal caving control system based on multimodal data. The system includes: a data acquisition unit for acquiring multimodal data, including video data, audio data, vibration data, tail beam inclination angle data, and insert plate stroke data of the top coal caving face; and a parameter prediction unit for predicting and generating core coal caving parameters based on the multimodal data. The core coal caving parameters include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target insert plate extension distance, and target insert plate extension frequency. Based on the core coal caving parameters, the system controls the hydraulic support to swing the tail beam according to the target tail beam swing angle and target tail beam swing frequency during the coal caving time, and the insert plate to extend and retract according to the target insert plate extension distance and target insert plate extension frequency, thereby performing the top coal caving operation.

[0016] In some embodiments, the parameter prediction unit includes: a vector acquisition module, used to extract features from the multimodal data to obtain multimodal feature vectors; and a processing module, used to perform multimodal feature fusion on the multimodal feature vectors to predict the coal release time, the target tail beam swing angle, the target tail beam swing frequency, the target insert plate extension distance, and the target insert plate extension frequency, respectively.

[0017] In some embodiments, the multimodal data also includes: coal mining machine speed, scraper conveyor power, top coal caving working face pressure data, top coal caving working face inclination length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average interbedded gangue rate, initial coal seam mineability index and gas concentration.

[0018] In some embodiments, the vector acquisition module is specifically used for:

[0019] Inputting the video data into a trained coal gangue recognition model, performing feature extraction on the video data, and obtaining a first coal gangue feature vector;

[0020] The audio data is input into the trained voiceprint coal gangue recognition model, and features are extracted from the audio data to obtain the second coal gangue feature vector.

[0021] Inputting the vibration data into a trained vibration frequency coal gangue recognition model, performing feature extraction on the vibration data, and obtaining a third coal gangue feature vector;

[0022] The tail beam inclination angle data, the slide block stroke data, the coal mining machine speed, the scraper conveyor power, the top coal caving face pressure data, the top coal caving face dip length, the initial average coal seam thickness, the initial coal seam dip angle, the initial average coal seam intercalation rate, the initial coal seam mineability index, and the gas concentration are mapped to their respective representation spaces to generate tail beam inclination angle vector, slide block stroke vector, coal mining machine speed vector, scraper conveyor power vector, top coal caving face pressure vector, top coal caving face dip length vector, initial average coal seam thickness vector, initial coal seam dip angle vector, initial average coal seam intercalation rate vector, initial coal seam mineability index vector, and gas concentration vector.

[0023] In some embodiments, the parameter prediction unit is specifically used to: identify the coal and gangue state as a state to be released in response to at least one of the first coal and gangue feature vector, the second coal and gangue feature vector, and the third coal and gangue feature vector; and perform multimodal feature fusion based on the tail beam inclination angle vector, the insert plate stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving face pressure vector, the top coal caving face dip length vector, the initial coal seam average thickness vector, the initial coal seam dip angle vector, the initial coal seam average interbedded gangue ratio vector, the initial coal seam mineability index vector, and the gas concentration vector to predict the coal release time, the target tail beam swing angle, the target tail beam swing frequency, the target insert plate extension distance, and the target insert plate extension frequency, respectively.

[0024] In some embodiments, the system further includes: a first model training unit, configured to acquire a video sample dataset, train a coal gangue recognition model, and generate a trained coal gangue recognition model; a second model training unit, configured to acquire an audio sample dataset, train a voiceprint coal gangue recognition model, and generate a trained voiceprint coal gangue recognition model; and a third model training unit, configured to acquire a vibration sample dataset, train a vibration frequency coal gangue recognition model, and generate a trained vibration frequency coal gangue recognition model.

[0025] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0026] By acquiring multimodal data, including video, audio, and vibration data of the top coal caving face, as well as tail beam inclination angle and insert plate stroke data, core parameters for coal caving are predicted and generated based on this data. These core parameters include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target insert plate extension distance, and target insert plate extension frequency. Based on these core parameters, the hydraulic supports are controlled to swing the tail beam according to the target tail beam swing angle and frequency, and extend and retract the insert plates according to the target insert plate extension distance and frequency, thus executing the top coal caving operation. This allows for automatic identification of the coal and gangue state and control of top coal caving mining for automatic coal caving, ensuring safety and reliability. Attached Figure Description

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the background technology, the drawings required for use in the embodiments of the present disclosure or the background technology will be described below.

[0028] Figure 1 A flowchart of an intelligent coal caving control method based on multimodal data provided by an embodiment of the present disclosure;

[0029] Figure 2 A structural diagram of a hydraulic support in an intelligent coal caving control method based on multimodal data provided in an embodiment of the present disclosure;

[0030] Figure 3 A flowchart of sub-step S2 of the intelligent coal discharge control method based on multimodal data provided in the embodiments of this disclosure;

[0031] Figure 4 A structural diagram of an intelligent coal feeding control system based on multimodal data provided in this disclosure embodiment;

[0032] Figure 5 A structural diagram of a parameter prediction unit in an intelligent coal caving control system based on multimodal data provided by an embodiment of the present disclosure;

[0033] Figure 6 This is a structural diagram of another intelligent coal caving control system based on multimodal data provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0035] In the process of top-coal caving mining, the freely falling top-coal generally consists of three layers: the coal seam, the coal-gangue mixture layer, and the gangue layer. Therefore, the coal caving process can be basically divided into three stages: the first stage is the coal caving stage, the second stage is the coal-gangue mixture stage, and the last stage is the gangue caving stage. Due to the complexity of the underground working face in coal mines and the current lack of a reliable method for identifying the thickness of the roof coal seam and the coal-rock interface, coal-rock identification has always been a technical challenge in coal seam mining.

[0036] Based on this, this disclosure provides an intelligent coal release control method and system based on multimodal data to automatically identify the state of coal and gangue and control top coal caving mining for automatic coal release, which is safe and reliable.

[0037] The following describes an intelligent coal caving control method and system based on multimodal data according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0038] Figure 1 This is a flow chart of an intelligent coal caving control method based on multimodal data provided by an embodiment of the present disclosure.

[0039] like Figure 1 As shown, the intelligent coal caving control method based on multimodal data provided by the embodiment of the present disclosure includes but is not limited to the following steps:

[0040] S1: Acquire multimodal data; wherein the multimodal data includes: video data, audio data, vibration data, tail beam inclination data and insert plate stroke data of the top coal caving working face.

[0041] In the embodiment of the present disclosure, multimodal data is acquired, and the multimodal data includes: video data, audio data, and vibration data of the top coal caving working face, as well as tail beam inclination data and inserter stroke data.

[0042] The video data can be acquired using an image acquisition device, such as a camera. In this embodiment, the image acquisition device can be installed at a specific location in the top coal caving face, for example, on the top beam of the hydraulic support or on the tail beam component. In this embodiment, the image acquisition device can collect information about the falling coal blocks during top coal caving, and the state of the coal and gangue during the top coal caving process can be obtained by analyzing the video data.

[0043] Audio data can be acquired through sound acquisition devices, such as sound sensors and microphones. These devices can be installed on the tail beam component of the hydraulic support to acquire audio data of coal chunks falling onto the tail beam during the top coal caving process.

[0044] It is understandable that coal chunks and gangue make different sounds when they fall onto the tail beam. Different proportions of coal chunks and gangue will also produce different sounds when they fall onto the tail beam. By acquiring and analyzing the audio data of coal chunks falling onto the tail beam during the top coal caving process, the coal and gangue state of the falling coal chunks can be determined.

[0045] Vibration data can be obtained through vibration sensors, which can be installed on the tail beam component of the hydraulic support. These sensors can acquire vibration data of the tail beam component as coal blocks fall onto it during the top coal caving process.

[0046] It is understandable that the weight density of coal blocks and gangue is different, and the frequencies of vibration of the tail beam components caused by coal blocks and gangue falling on the tail beam are different. The frequencies of vibration of the tail beam components caused by coal blocks falling on the tail beam during the top coal caving process can be obtained and analyzed to determine the coal and gangue status of the fallen coal blocks.

[0047] In some embodiments, the coal and gangue state includes three states: all-coal state, coal-gangue mixed state, and all-gangue state. In the all-coal state, coal accounts for 70%-100%; in the coal-gangue mixed state, coal accounts for 10%-70%; and in the all-gangue state, coal accounts for 0%-10%.

[0048] In this embodiment of the disclosure, multimodal data, including video data, audio data, and vibration data, can be acquired to identify the state of coal and gangue. In this case, tail beam tilt angle data and insert plate stroke data can be acquired to obtain the current posture data of the hydraulic support.

[0049] Among them, when the coal gangue state is a full coal state or a coal gangue mixed state, the posture of the hydraulic support is judged according to the current posture data of the hydraulic support. When it is judged that the current state is coal releasing, the method of controlling the tail beam and the plug plate is further determined, and the coal releasing action is executed; or, when it is judged that the current state is parked, the method of controlling the tail beam and the plug plate is further determined, and the coal releasing action is executed; or, when the coal gangue state is a full coal gangue state, the posture of the hydraulic support is judged according to the current posture data of the hydraulic support. When it is judged that the current state is coal releasing, the method of controlling the tail beam and the plug plate is further determined, and the coal releasing action is stopped; or, when the coal gangue state is a full coal gangue state, the posture of the hydraulic support is judged according to the current posture data of the hydraulic support. When it is judged that the current state is parked, the method of controlling the tail beam and the plug plate is further determined, and the coal releasing action is stopped, etc.

[0050] The coal discharging state may be when coal and / or coal gangue falls into the scraper; the parking state may be when no coal or coal gangue falls into the scraper.

[0051] In one possible implementation, Figure 2 As shown, during the top coal mining process, coal and / or gangue from the roof will fall onto the tail beam of the hydraulic support. The electro-hydraulic control system of the hydraulic support controls the tail beam components (such as...) via electromagnetic fluid exchange valves. Figure 2 The coal and / or coal gangue will slide into the scraper conveyor ( Figure 2 The five components are transported to a belt conveyor and then conveyed to the surface.

[0052] In this embodiment of the disclosure, multimodal data is acquired via a camera (including an illumination lamp) (e.g., Figure 2 The six components), sound and vibration sensors (such as Figure 2 8 components in), tilt sensor (such as Figure 2 The system, consisting of seven components and stroke sensors, collects video, audio, vibration, tail beam tilt angle, and insert plate stroke data. Tail beam tilt angle data indicates the swaying of the hydraulic support's tail beam component; insert plate stroke data indicates the extension and retraction of the hydraulic support's insert plate component; video, audio, and vibration data can identify coal discharge conditions, including the coal-gangue ratio and the operating status of the tail beam component.

[0053] S2: Based on the multimodal data, the core parameters of coal caving are predicted and generated; wherein the core parameters of coal caving include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target plug plate extension and extension distance, and target plug plate extension and extension frequency; based on the core parameters of coal caving, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and target tail beam swing frequency, and to extend and retract the plug plate according to the target plug plate extension and extension distance and target plug plate extension and extension frequency within the coal caving time, thereby executing the top coal caving operation.

[0054] In this embodiment of the disclosure, after acquiring multimodal data, including video data, audio data, vibration data, tail beam inclination data, and insert plate stroke data, the core parameters for coal discharge can be predicted and generated based on the multimodal data.

[0055] Among them, based on multimodal data, the coal feeding time, target tail beam swing angle, target tail beam swing frequency, target insert plate extension distance and target insert plate extension frequency are predicted and generated.

[0056] In this embodiment of the disclosure, multimodal data is acquired and transmitted to an edge computing device. Based on the received multimodal data and a pre-stored intelligent model, the edge computing device predicts and generates the coal discharge time, the target tail beam swing angle, the target tail beam swing frequency, the target insert plate extension distance, and the target insert plate extension frequency.

[0057] Among them, acquiring multimodal data and transmitting it to edge computing devices can be done through the 5G network layer. Because the 5G network layer has the characteristics of low latency, large connectivity, and ultra-high bandwidth, the collected multimodal data can be quickly sent to the edge computing devices through the 5G network layer.

[0058] In this embodiment of the disclosure, the coal release time, target tail beam swing angle, target tail beam swing frequency, target insert plate extension distance and target insert plate extension frequency can be set first based on human experience. Then, during the top coal caving mining process, multimodal data will be acquired in real time. Based on the multimodal data, the core parameters of coal release will be predicted and generated to update the intelligent model in real time.

[0059] In some embodiments, the intelligent model is trained on a cloud computing node. In this embodiment, after receiving multimodal data, the edge computing device transmits it to the cloud computing node, where the intelligent model is trained and updated. After training is complete, the cloud computing node sends the intelligent model back to the edge computing device, which then predicts and generates core parameters for coal release based on the multimodal data.

[0060] Among them, edge computing devices can transmit multimodal data to cloud computing nodes through the 5G network layer. Due to the characteristics of low latency, large connection and ultra-high bandwidth of the 5G network layer, the collected multimodal data can be quickly sent to the cloud computing nodes through the 5G network layer.

[0061] In this embodiment of the disclosure, when the core parameters for coal caving are predicted and generated based on multimodal data, these core parameters include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target insert plate extension distance, and target insert plate extension frequency. This allows the hydraulic support to control the tail beam to swing according to the target tail beam swing angle and frequency, and the insert plate to extend and retract according to the target insert plate extension distance and frequency, thus performing the top coal caving operation. Therefore, the coal and gangue state can be automatically identified, and the top coal caving mining can be controlled to automatically caving coal, ensuring safety and reliability.

[0062] By implementing the disclosed embodiments, multimodal data is acquired, wherein the multimodal data includes video data, audio data, vibration data, tail beam inclination data, and blade travel data of the top coal caving working face. Based on the multimodal data, core coal caving parameters are predicted and generated, wherein the core coal caving parameters include at least the coal caving time, target tail beam swing angle, target tail beam swing frequency, target blade extension distance, and target blade extension frequency. Based on the core coal caving parameters, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and target tail beam swing frequency, and to extend and retract the blade according to the target blade extension distance and target blade extension frequency within the coal caving time, thereby executing the top coal caving operation. Thus, the coal gangue state can be automatically identified, and top coal caving mining can be controlled to automatically caving coal, which is safe and reliable.

[0063] like Figure 3 As shown, in some embodiments, S2: Based on multimodal data, predict and generate core parameters for coal discharge, including:

[0064] S21: Extract features from multimodal data to obtain multimodal feature vectors.

[0065] In some embodiments, the multimodal data further includes: coal mining machine speed, scraper conveyor power, top coal caving face pressure data, top coal caving face dip length, initial average coal seam thickness, initial coal seam dip angle, initial average interbedded rock ratio, initial coal seam mineability index, and gas concentration.

[0066] Among them, the initial average coal seam thickness, initial coal seam dip angle, initial average coal seam interbedded rock ratio, and initial coal seam mineability index can be obtained during the initial geological mapping.

[0067] It is understood that during the top coal caving process, the thickness, position, and shape of the coal seam can be obtained in real time. When the initial average thickness, initial inclination, initial average gangue ratio, and initial mineable index of the coal seam are known, the current average thickness, inclination, average gangue ratio, and mineable index of the coal seam can be determined based on the real-time acquired thickness, position, and shape of the coal seam. Thus, it is possible to determine whether top coal caving should continue, as well as information such as the location of the top coal caving. Once it is determined that top coal caving should continue and the location of the caving should be determined, the hydraulic support can be controlled to perform the top coal caving operation.

[0068] In some embodiments, S21: extracting features from multimodal data to obtain a multimodal feature vector includes:

[0069] Input the video data into the trained gangue recognition model, perform feature extraction on the video data, and obtain the first gangue feature vector;

[0070] The audio data is input into the trained voiceprint coal gangue recognition model, and features are extracted from the audio data to obtain the second coal gangue feature vector.

[0071] The vibration data is input into the trained vibration frequency coal gangue identification model, and the vibration data is used to extract features to obtain the third coal gangue feature vector.

[0072] The tail beam inclination data, spigot stroke data, shearer speed, scraper conveyor power, top coal caving working face pressure data, top coal caving working face dip length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average gangue ratio, initial coal seam mineability index and gas concentration are mapped to their respective representation spaces to generate the tail beam inclination vector, spigot stroke vector, shearer speed vector, scraper conveyor power vector, top coal caving working face pressure vector, top coal caving working face dip length vector, initial coal seam average thickness vector, initial coal seam inclination vector, initial coal seam average gangue ratio vector, initial coal seam mineability index vector and gas concentration vector.

[0073] In this embodiment of the disclosure, video data is input into a trained coal gangue identification model, and features are extracted from the video data to obtain a first coal gangue feature vector. The coal gangue identification model can be a Faster R-CNN algorithm model.

[0074] In this embodiment of the disclosure, audio data is input into a trained voiceprint coal gangue recognition model, and features are extracted from the audio data to obtain a second coal gangue feature vector. The voiceprint coal gangue recognition model can be a GSV model, which is a combination of a Gaussian mixture model improved by GMM-UBM and an SVM model.

[0075] In this embodiment of the disclosure, vibration data is input into a trained vibration frequency coal gangue identification model, and feature extraction is performed on the vibration data to obtain a third coal gangue feature vector. The vibration frequency coal gangue identification model can be a neural network model.

[0076] In the embodiment of the present disclosure, the tail beam inclination data, the plug plate travel data, the coal mining machine speed, the scraper conveyor power, the top coal caving working face pressure data, the top coal caving working face inclination length, the initial coal seam average thickness, the initial coal seam inclination, the initial coal seam average interbedded gangue rate, the initial coal seam mineable index and the gas concentration are mapped to their respective representation spaces to generate the tail beam inclination vector, the plug plate travel vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average interbedded gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, which can be collaboratively used. The representation method, based on the similarity model, maps the tail beam inclination data, spigot stroke data, shearer speed, scraper conveyor power, top coal caving working face pressure data, top coal caving working face dip length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average gangue ratio, initial coal seam mineable index and gas concentration to their respective representation spaces, and generates the tail beam inclination vector, spigot stroke vector, shearer speed vector, scraper conveyor power vector, top coal caving working face pressure vector, top coal caving working face dip length vector, initial coal seam average thickness vector, initial coal seam inclination vector, initial coal seam average gangue ratio vector, initial coal seam mineable index vector and gas concentration vector.

[0077] In some embodiments, the intelligent coal discharge control method based on multimodal data provided by the embodiments of the present disclosure also includes: obtaining a video sample data set, training a coal gangue recognition model, and generating a trained coal gangue recognition model; obtaining an audio sample data set, training a voiceprint coal gangue recognition model, and generating a trained voiceprint coal gangue recognition model; obtaining a vibration sample data set, training a vibration frequency coal gangue recognition model, and generating a trained vibration frequency coal gangue recognition model.

[0078] In the disclosed embodiment, a video sample dataset is obtained and a gangue recognition model is trained to generate a trained gangue recognition model. The video sample dataset includes multiple sets of video sample data and corresponding gangue states. If the gangue recognition model uses the FasterRCNN algorithm model, the multiple sets of video sample data and corresponding gangue states are respectively input into the FasterRCNN algorithm model based on target supervision, and the FasterRCNN algorithm model is trained to generate a trained gangue recognition model.

[0079] In this embodiment of the disclosure, an audio sample dataset is obtained, and the voiceprint coal gangue recognition model is trained to generate a trained voiceprint coal gangue recognition model. The audio sample dataset includes multiple sets of audio sample data and corresponding coal gangue states. When the voiceprint coal gangue recognition model adopts the GSV model, based on the target supervision method, the multiple sets of audio sample data and corresponding coal gangue states are respectively input into the GSV model to train the GSV model and generate a trained voiceprint coal gangue recognition model.

[0080] In this embodiment of the disclosure, a vibration sample dataset is obtained, and a vibration frequency coal gangue identification model is trained to generate a trained vibration frequency coal gangue identification model. The vibration sample dataset includes multiple sets of vibration sample data and corresponding coal gangue states. When the vibration frequency coal gangue identification model adopts a neural network model, based on the target supervision method, multiple sets of vibration sample data and corresponding coal gangue states are input into the neural network model respectively, and the neural network model is trained to generate a trained vibration frequency coal gangue identification model.

[0081] S22: performing multimodal feature fusion on the multimodal feature vector to respectively predict the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target plugboard extension and retraction distance, and the target plugboard extension and retraction frequency.

[0082] In the embodiment of the present disclosure, multimodal feature fusion is performed on the multimodal feature vector, and a generative adversarial network model can be used for multimodal fusion. The information of multiple modes is combined to perform target prediction, and the coal release time, target tail beam swing angle, target tail beam swing frequency, target plug plate extension distance and target plug plate extension frequency are predicted respectively.

[0083] In some embodiments, in response to identifying the coal and gangue state as a state to be released as determined by at least one of the first coal and gangue feature vector, the second coal and gangue feature vector, and the third coal and gangue feature vector, multimodal feature fusion is performed based on the tail beam inclination angle vector, the slide plate stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving face pressure vector, the top coal caving face dip length vector, the initial average coal seam thickness vector, the initial coal seam inclination angle vector, the initial average coal seam interbedded gangue ratio vector, the initial coal seam mineability index vector, and the gas concentration vector to predict the coal release time, the target tail beam swing angle, the predicted target tail beam swing frequency, the target slide plate extension distance, and the target slide plate extension frequency.

[0084] In the embodiment of the present disclosure, the first coal gangue feature vector identifies that the coal gangue state is a state to be coal caving. According to the tail beam inclination vector, the inserter stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, multimodal feature fusion is performed to predict the coal caving time, the target tail beam swing angle, the predicted target tail beam swing frequency, the target inserter extension and extension distance and the target inserter extension and extension frequency respectively.

[0085] Among them, the first coal gangue feature vector identifies the coal gangue state as the state of coal to be discharged. The coal gangue state can be identified by a trained coal gangue identification model.

[0086] In the embodiment of the present disclosure, the second coal gangue feature vector identifies that the coal gangue state is a state to be coal caving. According to the tail beam inclination vector, the inserter stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, multimodal feature fusion is performed to predict the coal caving time, the target tail beam swing angle, the predicted target tail beam swing frequency, the target inserter extension distance and the target inserter extension frequency, respectively.

[0087] Among them, the second coal gangue feature vector identifies the coal gangue state as the state of coal to be discharged. The coal gangue state can be identified by a trained acoustic coal gangue recognition model.

[0088] In the embodiment of the present disclosure, the third coal gangue feature vector identifies that the coal gangue state is a state to be coal-laying. According to the tail beam inclination vector, the insert plate stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, multimodal feature fusion is performed to predict the coal-laying time, the target tail beam swing angle, the predicted target tail beam swing frequency, the target insert plate extension distance and the target insert plate extension frequency respectively.

[0089] Among them, the second coal gangue feature vector identifies the coal gangue state as the state of coal to be discharged. The coal gangue state can be identified by a trained vibration frequency coal gangue identification model.

[0090] To achieve the above embodiments, this disclosure also proposes an intelligent coal feeding control system based on multimodal data.

[0091] Figure 4 This is a structural diagram of an intelligent coal feeding control system 1 based on multimodal data, provided in an embodiment of this disclosure.

[0092] Specifically, such as Figure 4 As shown in the figure, the intelligent coal feeding control system 1 based on multimodal data provided in this embodiment includes: a data acquisition unit 11 and a parameter prediction unit 12.

[0093] The data acquisition unit 11 is used to acquire multimodal data; wherein the multimodal data includes: video data, audio data, and vibration data of the top coal caving working face, as well as tail beam inclination data and inserter stroke data.

[0094] The parameter prediction unit 12 is used to predict and generate core parameters for coal caving based on multimodal data; wherein the core parameters for coal caving include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target plug plate extension and extension distance, and target plug plate extension and extension frequency; based on the core parameters for coal caving, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and target tail beam swing frequency, and to extend and retract the plug plate according to the target plug plate extension and extension distance and target plug plate extension and extension frequency within the coal caving time, thereby executing the top coal caving operation.

[0095] By implementing the disclosed embodiments, the data acquisition unit 11 is configured to acquire multimodal data, wherein the multimodal data includes video data, audio data, vibration data, tail beam inclination data, and blade travel data of the top coal caving working face. The parameter prediction unit 12 is configured to predict and generate core coal caving parameters based on the multimodal data, wherein the core coal caving parameters include at least the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target blade extension distance, and the target blade extension frequency. Based on the core coal caving parameters, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and the target tail beam swing frequency, and to extend and retract the blade according to the target blade extension distance and the target blade extension frequency, thereby executing the top coal caving operation. Thus, the coal gangue state can be automatically identified, and top coal caving mining can be controlled to automatically caving coal, safely and reliably.

[0096] like Figure 5 As shown, in some embodiments, the parameter prediction unit 12 includes a vector acquisition module 121 and a processing module 122.

[0097] The vector acquisition module 121 is used to extract features from multimodal data and obtain multimodal feature vectors.

[0098] The processing module 122 is used to perform multimodal feature fusion on the multimodal feature vector to respectively predict the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target plugboard extension distance and the target plugboard extension frequency.

[0099] In some embodiments, the multimodal data further includes: coal mining machine speed, scraper conveyor power, top coal caving face pressure data, top coal caving face dip length, initial average coal seam thickness, initial coal seam dip angle, initial average interbedded rock ratio, initial coal seam mineability index, and gas concentration.

[0100] In some embodiments, the vector acquisition module 121 is specifically used for:

[0101] Input the video data into the trained gangue recognition model, perform feature extraction on the video data, and obtain the first gangue feature vector;

[0102] The audio data is input into the trained voiceprint coal gangue recognition model, and features are extracted from the audio data to obtain the second coal gangue feature vector.

[0103] The vibration data is input into the trained vibration frequency coal gangue identification model, and the vibration data is used to extract features to obtain the third coal gangue feature vector.

[0104] The tail beam inclination data, spigot stroke data, shearer speed, scraper conveyor power, top coal caving working face pressure data, top coal caving working face dip length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average gangue ratio, initial coal seam mineability index and gas concentration are mapped to their respective representation spaces to generate the tail beam inclination vector, spigot stroke vector, shearer speed vector, scraper conveyor power vector, top coal caving working face pressure vector, top coal caving working face dip length vector, initial coal seam average thickness vector, initial coal seam inclination vector, initial coal seam average gangue ratio vector, initial coal seam mineability index vector and gas concentration vector.

[0105] In some embodiments, the processing module is specifically configured to:

[0106] In response to at least one of the first coal and gangue feature vector, the second coal and gangue feature vector, and the third coal and gangue feature vector, the coal and gangue state is identified as the state to be released. Based on the tail beam inclination angle vector, the slide plate stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving face pressure vector, the top coal caving face dip length vector, the initial average coal seam thickness vector, the initial coal seam dip angle vector, the initial average coal seam interbedded gangue ratio vector, the initial coal seam mineability index vector, and the gas concentration vector, multimodal feature fusion is performed to predict the coal release time, the target tail beam swing angle, the predicted target tail beam swing frequency, the target slide plate extension distance, and the target slide plate extension frequency.

[0107] like Figure 6 As shown, in some embodiments, the intelligent coal feeding control system 1 based on multimodal data provided in this disclosure further includes: a first model training unit 13, a first model training unit 13, and a first model training unit 13.

[0108] The first model training unit 13 is used to acquire video sample datasets, train the coal gangue recognition model, and generate a trained coal gangue recognition model.

[0109] The second model training unit 14 is used to acquire an audio sample dataset, train the voiceprint coal gangue recognition model, and generate a trained voiceprint coal gangue recognition model.

[0110] The third model training unit 15 is used to acquire the vibration sample dataset, train the vibration frequency coal gangue identification model, and generate the trained vibration frequency coal gangue identification model.

[0111] Regarding the intelligent coal caving control system 1 based on multimodal data in the above embodiment, the specific methods of each device have been described in detail in the embodiment of the intelligent coal caving control method based on multimodal data, and will not be repeated here.

[0112] The intelligent coal feeding control system 1 based on multimodal data provided in this disclosure has the same beneficial effects as the intelligent coal feeding control method based on multimodal data, and will not be described again here.

[0113] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0114] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," and "exemplary embodiment" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0116] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0117] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0118] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0119] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0120] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0121] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0122] Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are illustrative and are not to be construed as limitations on the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. An intelligent coal caving control method based on multimodal data, characterized in that: include: Acquire multimodal data; wherein the multimodal data includes: video data, audio data, vibration data, tail beam inclination data, plug board travel data, coal mining machine speed, scraper conveyor power, top coal caving face pressure data, top coal caving face inclination length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average gangue ratio, initial coal seam mineability index and gas concentration; According to the multimodal data, the coal caving core parameters are predicted and generated; wherein the coal caving core parameters at least include: coal caving time, target tail beam swing angle, target tail beam swing frequency, target plug plate telescopic distance and target plug plate telescopic frequency; according to the coal caving core parameters, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and the target tail beam swing frequency, and the plug plate is telescopic according to the target plug plate telescopic distance and the target plug plate telescopic frequency within the coal caving time, so as to perform the top coal caving operation; Based on the multimodal data, the core parameters of coal caving are predicted to include: The video data is input into a trained coal gangue recognition model, and features are extracted from the video data to obtain a first coal gangue feature vector; the audio data is input into a trained voiceprint coal gangue recognition model, and features are extracted from the audio data to obtain a second coal gangue feature vector; the vibration data is input into a trained vibration frequency coal gangue recognition model, and features are extracted from the vibration data to obtain a third coal gangue feature vector; the tail beam inclination angle data, the plug board stroke data, the coal mining machine speed, the scraper conveyor power, and the top coal caving work data are ... The surface pressure data, the inclined length of the top coal caving working face, the average thickness of the initial coal seam, the inclination angle of the initial coal seam, the average gangue rate of the initial coal seam, the initial coal seam recoverable index and the gas concentration are respectively mapped to their respective representation spaces to generate the tail beam inclination vector, the plug plate travel vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the inclined length vector of the top coal caving working face, the average thickness vector of the initial coal seam, the inclination angle vector of the initial coal seam, the average gangue rate vector of the initial coal seam, the recoverable index vector of the initial coal seam and the gas concentration vector; In response to at least one of the first coal gangue feature vector, the second coal gangue feature vector and the third coal gangue feature vector identifying that the coal gangue state is a state to be coal caving, multimodal feature fusion is performed according to the tail beam inclination vector, the plugboard stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, to respectively predict the coal caving time, the target tail beam swing angle, the target tail beam swing frequency, the target plugboard extension and retraction distance and the target plugboard extension and retraction frequency.

2. The method according to claim 1, characterized in that The method further comprises: Obtain a video sample data set, train a coal gangue recognition model, and generate a trained coal gangue recognition model; Acquire an audio sample data set, train the voiceprint coal gangue recognition model, and generate a trained voiceprint coal gangue recognition model; A vibration sample data set is obtained, and a vibration frequency coal gangue recognition model is trained to generate a trained vibration frequency coal gangue recognition model.

3. An intelligent coal discharge control system based on multimodal data, characterized in that: The system comprises: A data acquisition unit, used to acquire multimodal data; wherein the multimodal data includes: video data, audio data, vibration data, tail beam inclination data, plug board travel data, coal mining machine speed, scraper conveyor power, top coal caving face pressure data, top coal caving face inclination length, initial coal seam average thickness, initial coal seam inclination, initial coal seam average gangue ratio, initial coal seam mineability index and gas concentration; A parameter prediction unit is used to predict and generate core parameters of coal caving according to the multimodal data; wherein the core parameters of coal caving include at least: coal caving time, target tail beam swing angle, target tail beam swing frequency, target plug plate telescopic distance and target plug plate telescopic frequency; according to the core parameters of coal caving, the hydraulic support is controlled to swing the tail beam according to the target tail beam swing angle and the target tail beam swing frequency, and the plug plate is telescopic according to the target plug plate telescopic distance and the target plug plate telescopic frequency within the coal caving time, so as to perform top coal caving operation; The parameter prediction unit includes: a vector acquisition module and a processing module; The vector acquisition module is used to input the video data into a trained coal gangue recognition model, perform feature extraction on the video data, and obtain a first coal gangue feature vector; input the audio data into a trained voiceprint coal gangue recognition model, perform feature extraction on the audio data, and obtain a second coal gangue feature vector; input the vibration data into a trained vibration frequency coal gangue recognition model, perform feature extraction on the vibration data, and obtain a third coal gangue feature vector; input the tail beam inclination data, the plug board stroke data, the coal mining machine speed, the scraper conveyor power, the The pressure data of the top coal caving working face, the inclined length of the top coal caving working face, the average thickness of the initial coal seam, the inclination angle of the initial coal seam, the average gangue rate of the initial coal seam, the recoverable index of the initial coal seam and the gas concentration are respectively mapped to their respective representation spaces to generate the tail beam inclination vector, the plug plate travel vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the inclined length vector of the top coal caving working face, the average thickness vector of the initial coal seam, the inclination angle vector of the initial coal seam, the average gangue rate vector of the initial coal seam, the recoverable index vector of the initial coal seam and the gas concentration vector; The processing module is used to identify the coal gangue state as a state to be coal-released in response to at least one of the first coal gangue feature vector, the second coal gangue feature vector and the third coal gangue feature vector, and perform multimodal feature fusion according to the tail beam inclination vector, the plugboard stroke vector, the coal mining machine speed vector, the scraper conveyor power vector, the top coal caving working face pressure vector, the top coal caving working face inclination length vector, the initial coal seam average thickness vector, the initial coal seam inclination vector, the initial coal seam average gangue rate vector, the initial coal seam mineable index vector and the gas concentration vector, to respectively predict the coal release time, the target tail beam swing angle, the target tail beam swing frequency, the target plugboard extension and extension distance and the target plugboard extension and extension frequency.

Images