Automatic control system and method for open pit coal mine excavator

Abstract

The invention discloses an automatic control system and method for an open pit coal mine excavator, and the method comprises the steps: obtaining a coal pile surface state image and a coal pile spectrum image set collected by a spectrum sensor and a camera; a deep learning-based neural network model is adopted to perform feature extraction on the coal pile and the coal pile to capture external morphological features and internal attribute features of the coal pile so as to obtain surface state features and spectral feature representations of the coal pile; and further performing multi-mode comprehensive analysis on the surface state characteristics and the spectral characteristics to mine essential correlation characteristic representation between the surface state characteristics and the spectral characteristics, and accurately judging the type of the coal pile based on the essential correlation characteristic representation. Through the mode, accurate recognition of the coal pile can be realized, so that the mining task instruction is intelligently generated, and the mining efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent control, and in particular to an automatic control system and method for an open-pit coal mine excavator. Background Art

[0002] In the process of open-pit coal mining, the automatic control system of the excavator is one of the key technologies to achieve efficient and precise operation. However, the existing automatic control system has significant defects in coal pile identification and excavation priority decision-making, resulting in low mining efficiency and waste of resources. Traditional methods mainly rely on data collected by a single sensor (such as a camera or spectral sensor), lacking multimodal fusion analysis of the surface state and spectral characteristics of the coal pile. The limitations of this single data source make it difficult for the system to comprehensively and accurately identify the type of coal pile, especially under complex geological conditions, where the identification error is large. In addition, when generating excavation priorities, existing systems often rely on manual experience or simple rules, making it difficult to achieve intelligent and adaptive decision-making, resulting in low mining efficiency and unreasonable resource allocation.

[0003] Therefore, in order to overcome the above problems and improve the operating efficiency and resource utilization of open-pit coal mine excavators, an optimized automatic control system for open-pit coal mine excavators is expected. Summary of the invention

[0004] The embodiments of the present application aim to solve at least one of the technical problems existing in the prior art, and provide an automatic control system and method for an open-pit coal mine excavator, by acquiring a coal pile surface state image and a coal pile spectrum image set collected by a spectral sensor and a camera, and using a neural network model based on deep learning to extract features from both to capture the external morphological features and intrinsic attribute features of the coal pile, so as to obtain the surface state features and spectral feature representation of the coal pile, and further perform multimodal comprehensive analysis on the surface state features and spectral features to dig out the essential correlation feature representation between the two, and accurately judge the type of the coal pile based on this. In this way, accurate identification of the coal pile can be achieved, thereby intelligently generating excavation task instructions, improving excavation efficiency and resource utilization.

[0005] On the one hand, the present application provides an automatic control system for an open-pit coal mine excavator, comprising:

[0006] An image acquisition module, used for acquiring the coal pile surface state image and the coal pile spectrum image of each coal pile acquired by the spectrum sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image};

[0007] A feature extraction module, used for extracting the coal pile surface state features and coal pile spectrum features of the set of {coal pile surface state image, coal pile spectrum image} to obtain a set of {coal pile surface state feature map, coal pile spectrum feature map};

[0008] A feature interaction analysis module, used for performing feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectrum feature map} to obtain a set of coal pile surface state-spectrum multimodal joint perception coding feature maps;

[0009] A coal pile type identification module, used for performing coal pile type identification on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type identification results;

[0010] The mining task instruction generation module is used to generate mining task instructions based on the set of coal pile type identification results.

[0011] Optionally, the feature extraction module includes:

[0012] A coal pile surface state feature extraction unit, used for inputting each coal pile surface state image in the set of coal pile surface state images into a coal pile surface state feature extractor based on a depthwise separable convolution model to obtain a set of coal pile surface state feature maps;

[0013] The coal pile spectral feature extraction unit is used to input each coal pile spectral image in the set of coal pile spectral images into a coal pile spectral feature extractor based on a deep separable convolution model to obtain a set of coal pile spectral feature maps.

[0014] Optionally, the coal pile surface state feature extraction unit is further used to:

[0015] Cascading each coal pile surface state image in the set of coal pile surface state images along the channel dimension to obtain a coal pile surface state feature tensor;

[0016] Using each layer of the deep separable convolution model, in the forward pass of the layer, convolution processing, pooling processing and nonlinear activation processing are performed on the feature tensor of the surface state feature of the coal pile based on the separable convolution kernel, so that the last layer of the deep separable convolution model outputs a set of surface state feature maps of the coal pile.

[0017] Optionally, the feature interaction analysis module includes:

[0018] A feature pair construction unit, used to perform causal association analysis on the coal pile surface state feature map and the coal pile spectrum feature map, and obtain a set of {coal pile surface state feature, coal pile spectrum feature} feature pairs constituting a causal chain unit based on the analysis result;

[0019] A coal pile surface state-spectral feature joint perception unit, used for performing feature joint perception encoding on the set of {coal pile surface state feature, coal pile spectral feature} features containing the causal chain unit to obtain a set of coal pile surface state-spectral multimodal joint perception encoding feature vectors;

[0020] A feature aggregation unit is used to perform feature aggregation on the set of the coal pile surface state-spectral multimodal joint perception coding feature vectors to obtain the coal pile surface state-spectral multimodal joint perception coding feature map.

[0021] Optionally, the feature pair construction unit is further used to:

[0022] Performing feature decoupling on the coal pile surface state feature map and the coal pile spectrum feature map along the channel dimension of the coal pile surface state feature map and the coal pile spectrum feature map to obtain a set of coal pile surface state local feature coding vectors and a set of coal pile spectrum local feature coding vectors;

[0023] The set of local feature coding vectors of the coal pile surface state and the set of local feature coding vectors of the coal pile spectrum are input into the causal chain unit screening network to obtain a set of feature pairs of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} constituting the causal chain unit as the set of feature pairs of {coal pile surface state feature, coal pile spectrum feature}.

[0024] Optionally, the coal pile surface state-spectral feature joint perception unit is also used to: input each feature pair of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} constituting the causal chain unit in the set of feature pairs constituting the causal chain unit into the causal chain unit semantic dynamic response network to obtain the set of coal pile surface state-spectrum multimodal joint perception coding feature vectors.

[0025] Optionally, the coal pile type identification module is further used to:

[0026] Each coal pile surface state-spectral multimodal joint perceptual coding feature map in the set of coal pile surface state-spectral multimodal joint perceptual coding feature maps is respectively input into a classifier-based coal pile type identifier to obtain a set of coal pile type identification results.

[0027] Optionally, the mining task instruction generating module is further used to:

[0028] Based on the set of coal pile type identification results, determining the excavation priority of the open-pit coal mine excavator to obtain a set of excavation priorities;

[0029] Based on the set of mining priorities, a mining task instruction is generated.

[0030] On the other hand, the present application provides an automatic control method for an open-pit coal mine excavator, comprising:

[0031] Acquire the coal pile surface state image and the coal pile spectrum image of each coal pile collected by the spectrum sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image};

[0032] Extracting the coal pile surface state features and coal pile spectrum features of the set of {coal pile surface state image, coal pile spectrum image} to obtain a set of {coal pile surface state feature map, coal pile spectrum feature map};

[0033] Performing feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectrum feature map} to obtain a set of coal pile surface state-spectrum multimodal joint perception coding feature maps;

[0034] Performing coal pile type recognition on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type recognition results;

[0035] Based on the set of coal pile type identification results, an excavation task instruction is generated.

[0036] Compared with the related art, the automatic control system and method of an open-pit coal mine excavator provided by the present application obtains the coal pile surface state image and coal pile spectrum image set collected by the spectral sensor and the camera, and uses the neural network model based on deep learning to extract features of the two respectively to capture the external morphological characteristics and intrinsic attribute characteristics of the coal pile, so as to obtain the surface state characteristics and spectral feature representation of the coal pile, and further conducts multimodal comprehensive analysis of the surface state characteristics and spectral characteristics to dig out the essential correlation feature representation between the two, and accurately judges the type of coal pile based on this. In this way, the coal pile can be accurately identified, thereby intelligently generating excavation task instructions, improving excavation efficiency and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a block diagram of an automatic control system for an open-pit coal mine excavator according to an embodiment of the present application;

[0038] Figure 2 A schematic diagram of data flow of an automatic control system of an open-pit coal mine excavator according to an embodiment of the present application;

[0039] Figure 3 is a block diagram of a feature interaction analysis module in an automatic control system of an open-pit coal mine excavator according to an embodiment of the present application;

[0040] Figure 4 The flowchart is a method for automatically controlling an open-pit coal mine excavator according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0042] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0043] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0044] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0045] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0046] In open-pit coal mining, the existing excavator automatic control system has significant defects in coal pile identification and excavation priority decision-making, including relying solely on single sensor data and lacking multimodal fusion analysis, making it difficult to fully and accurately identify the type of coal pile, and relying more on manual experience or simple rules when generating excavation priorities, resulting in low mining efficiency and waste of resources.

[0047] Based on this, in the technical solution of the present application, an automatic control system for an open-pit coal mine excavator is proposed. Figure 1 It is a block diagram of an automatic control system of an open-pit coal mine excavator according to an embodiment of the present application. Figure 2Schematic diagram of data flow of the automatic control system of an open-pit coal mine excavator according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the automatic control system 300 of the open-pit coal mine excavator according to the embodiment of the present application includes: an image acquisition module 310, which is used to obtain the coal pile surface state image and the coal pile spectrum image of each coal pile collected by the spectral sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image}. A feature extraction module 320, which is used to extract the coal pile surface state features and coal pile spectrum features of the set of {coal pile surface state image, coal pile spectrum image} to obtain a set of {coal pile surface state feature map, coal pile spectrum feature map}. A feature interaction analysis module 330, which is used to perform feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectrum feature map} to obtain a set of coal pile surface state-spectral multimodal joint perception coding feature maps. A coal pile type identification module 340, which is used to perform coal pile type identification on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type identification results. The excavation task instruction generation module 350 is used to generate an excavation task instruction based on the set of coal pile type identification results.

[0048] Exemplarily, the image acquisition module 310 is used to obtain the coal pile surface state image and coal pile spectrum image of each coal pile collected by the spectral sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image}. Among them, the coal pile surface state image collected by the camera can capture the external morphology and other features of the coal pile, while the coal pile spectrum image collected by the spectral sensor can reflect the intrinsic attribute characteristics of the coal pile. Through multimodal data acquisition, the limitations of a single data source can be overcome, providing a more comprehensive and rich data foundation for the subsequent accurate identification of coal pile types.

[0049] Exemplarily, the feature extraction module 320 is used to extract the coal pile surface state features and coal pile spectral features of the set of {coal pile surface state images, coal pile spectral images} to obtain a set of {coal pile surface state feature maps, coal pile spectral feature maps}. That is, in the technical solution of the present application, first, each coal pile surface state image in the set of coal pile surface state images is input into a coal pile surface state feature extractor based on a deep separable convolution model to obtain a set of coal pile surface state feature maps. It should be understood that the set of coal pile surface state images collected by the camera contains rich information, such as the overall shape of the coal pile, the texture features of the coal pile surface, such as the roughness of the surface, whether there are cracks, etc., and the color distribution of the coal pile surface. However, these original image data are relatively complex and large, and it is difficult to directly use them to mine valuable information for coal pile type identification. In addition, considering that in a scene such as an open-pit coal mine where a large amount of coal pile surface state image data needs to be processed, the traditional convolution operation will generate a huge amount of calculation, which may lead to low system operation efficiency or even inability to process data in a timely manner. Compared with traditional convolution, depthwise separable convolution decomposes the standard convolution into depthwise convolution and pointwise convolution, which greatly reduces the complexity of the operation and enables the system to quickly process a large amount of image data under limited computing resources. Therefore, in the technical solution of the application, each coal pile surface state image in the set of coal pile surface state images is input into a coal pile surface state feature extractor based on a depthwise separable convolution model to extract effective information that can reflect the external morphological characteristics of the coal pile, such as the shape of the coal pile, surface texture and other feature information, so as to obtain a set of coal pile surface state feature maps, which provide an important basis for accurately judging the type of coal pile. In a specific example, the set of coal pile surface state images is input into a coal pile surface state feature extractor based on a depthwise separable convolution model, and each layer of the depthwise separable convolution model processes the input image in turn. In the forward pass of the layer, the separable convolution kernel is used to perform convolution processing on the image to extract local features in the image, so as to accurately find key information that can reflect the morphological characteristics of the coal pile in the complex coal pile image. Then, pooling is performed to reduce the data dimension and retain the main features, so as to screen and simplify a large amount of coal pile image information, remove redundant information, and leave the most valuable part for identifying the type of coal pile. Then, through nonlinear activation processing, the nonlinear expression ability of the model is increased, so that the model can better adapt to the complex and changeable feature relationships in the surface state image of the coal pile. Finally, the last layer of the deep separable convolutional model outputs a set of coal pile surface state feature maps, which accurately reflect the external morphological characteristics of each coal pile, and provide effective data support for the subsequent coal pile type identification and mining task instruction generation.

[0050] Similarly, considering that the spectral image of the coal pile contains rich intrinsic attribute information of the coal pile, such as the composition and quality of the coal, but the original spectral image data of the coal pile is large and complex, it is difficult to directly obtain valuable information for identifying the type of coal pile from it. Therefore, each spectral image of the coal pile in the set of spectral images of the coal pile is input into the deep separable convolution model to extract effective information that can reflect the intrinsic attribute characteristics of the coal pile, such as the composition characteristics and quality characteristics of the coal, so as to obtain a set of spectral feature maps of the coal pile. By extracting the surface state characteristics and intrinsic attribute characteristics of the coal pile, the comprehensive information representation reflecting the essential characteristics of the coal pile can be mined, which provides an important basis for accurately judging the type of the coal pile, helps to realize the accurate identification of the coal pile, and thus intelligently generates mining task instructions.

[0051] Exemplarily, the feature interaction analysis module 330 is used to perform feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectral feature map} to obtain a set of coal pile surface state-spectral multimodal joint perception coding feature maps. It should be understood that although the set of coal pile surface state feature maps and the set of coal pile spectral feature maps obtained by deep separable convolution respectively reflect the external morphological characteristics and intrinsic attribute characteristics of the coal pile, these feature maps respectively reflect the characteristics of different aspects of the coal pile, and due to differences in sources and properties, the distribution, scale and semantic background of each feature are obviously different. It is difficult to directly analyze them to deeply explore the intrinsic connection between the surface state of the coal pile and the spectral characteristics, which affects the accuracy of coal pile type judgment. In order to overcome the above difficulties, the set of coal pile surface state feature maps and the coal pile spectral feature maps are fused and analyzed, and traditional feature fusion methods, such as simple splicing or weighted fusion, often lack the ability to model implicit high-order feature relationships in complex scenes, and easily ignore the causal dependency between coal pile surface state features and coal pile spectral features. Therefore, in the technical solution of the application, a feature interaction analysis based on strong causal interconnection is performed on the set of {coal pile surface state feature map, coal pile spectral feature map}, so as to deeply explore the essential relationship between the coal pile surface state features and the coal pile spectral features, and obtain a set of coal pile surface state-spectral multimodal joint perception coding feature maps. Furthermore, the feature interaction analysis module based on strong causal interconnection can explicitly or implicitly learn the causal dependency between the coal pile surface state and the spectral features by introducing causal chain units, so as to screen out the interaction patterns with substantial causal relationships between the coal pile surface state and the spectral features, retain the causal-driven high-coupling semantic features, and capture the implicit dependencies that are difficult to model in traditional weighted or splicing models, so as to provide a more comprehensive and accurate basis for the subsequent accurate judgment of the coal pile type, thereby realizing intelligent mining task planning and reasonable resource allocation. Specifically, in the technical solution of the present application, if Figure 3As shown, the feature interaction analysis module 330 includes: a feature pair construction unit 331, which is used to perform causal association analysis on the coal pile surface state feature map and the coal pile spectral feature map, and obtain a set of {coal pile surface state feature, coal pile spectral feature} feature pairs that constitute a causal chain unit based on the analysis results. A coal pile surface state-spectral feature joint perception unit 332, which is used to perform feature joint perception encoding on the set of {coal pile surface state feature, coal pile spectral feature} feature pairs containing the causal chain unit to obtain a set of coal pile surface state-spectral multimodal joint perception encoding feature vectors. A feature aggregation unit 333, which is used to perform feature aggregation on the set of coal pile surface state-spectral multimodal joint perception encoding feature vectors to obtain the coal pile surface state-spectral multimodal joint perception encoding feature map.

[0052] Specifically, the feature pair construction unit 331 is used to perform causal association analysis on the coal pile surface state feature map and the coal pile spectrum feature map, and obtain a set of {coal pile surface state feature, coal pile spectrum feature} feature pairs constituting a causal chain unit based on the analysis results. In the embodiment of the present application, first of all, it should be understood that in the original coal pile surface state feature map and the coal pile spectrum feature map, the feature elements are highly coupled, the boundaries of the semantic representations of each channel are unclear, and there is a large amount of redundant information. It is difficult to effectively extract key information by directly performing subsequent processing. In the technical solution of the present application, the coal pile surface state feature map and the coal pile spectrum feature map are feature decoupled along the channel dimension of the coal pile surface state feature map and the coal pile spectrum feature map, so as to preliminarily disassemble the complex information of the coal pile surface state and the spectral feature, separate the highly coupled feature elements in the features, and reduce redundant information, so as to obtain a set of local feature coding vectors of the coal pile surface state and a set of local feature coding vectors of the coal pile spectrum, thereby providing a basis for subsequent more accurate analysis. In a specific example, the coal pile surface state characteristic map and the coal pile spectrum characteristic map are feature decoupled along the channel dimension of the coal pile surface state characteristic map and the coal pile spectrum characteristic map according to the following formula to obtain a set of coal pile surface state local feature coding vectors and a set of coal pile spectrum local feature coding vectors. The formula is:

[0053] Decouple(F 1 )={v 11 ,v 12 ,...,v 1i ,...,v 1n}

[0054] Decouple(F 2 )={v 21 ,v 22 ,...,v 2j ,...,v2n}

[0055] Among them, F 1 The characteristic diagram of the coal pile surface state, F 2 represents the spectral characteristic graph of the coal pile, decouple(·) represents the characteristic decoupling operation, v 11 ,v 12 ,v 1i v 1n represents each local feature coding vector of the surface state of the coal pile in the set of local feature coding vectors of the surface state of the coal pile, v 21 ,v 22 ,v 2j ,v 2n They respectively represent each coal pile spectral local feature coding vector in the set of coal pile spectral local feature coding vectors.

[0056] Furthermore, considering that different feature coding vectors may come from feature maps with different semantic background dimensions, direct matching will lead to semantic mismatch or offset, and there may be noise feature pairs in the set of local feature coding vectors of the coal pile surface state and the set of local feature coding vectors of the coal pile spectrum, which interferes with subsequent analysis. Therefore, in order to accurately analyze the coal pile characteristics, the set of local feature coding vectors of the coal pile surface state and the set of local feature coding vectors of the coal pile spectrum are further input into the causal chain unit screening network to screen and integrate the local feature coding vectors of the coal pile surface state and the local feature coding vectors of the coal pile spectrum. Specifically, before causal screening, the semantic alignment modulation flow field is first used to accurately dynamically align the coal pile feature pairs to achieve harmony and consistency, thereby eliminating the semantic mismatch problem. Then, the screening algorithm calculates the causal chain closed loop strength factor between the feature pairs, captures those coal pile feature pairs that are highly correlated and causally related, filters out potential noise feature pairs, and retains those important identification features that are highly causal for coal pile identification, so as to more accurately reflect the intrinsic causal relationship between the surface state of the coal pile and the spectral features, and obtains a set of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} feature pairs that constitute the causal chain unit. This provides strong support for the subsequent more in-depth analysis of the relationship between the surface state of the coal pile and the spectral features, and the accurate judgment of the type of coal pile. In a specific example, the specific steps of inputting the set of the coal pile surface state local feature coding vector and the set of the coal pile spectrum local feature coding vector into the causal chain unit screening network are as follows: First, extract the first candidate local feature coding vector from the set of the coal pile surface state local feature coding vector. Next, extract the second candidate local feature coding vector from the set of the coal pile spectrum local feature coding vector. Then, construct a first candidate-second candidate semantic alignment modulation flow field between the first candidate local feature coding vector and the second candidate local feature coding vector. Then, based on the first alternative-second alternative semantic alignment modulation flow field, the first alternative local feature coding vector and the second alternative local feature coding vector are feature aligned to obtain the aligned first alternative local feature coding vector and the aligned second alternative local feature coding vector. The aligned first alternative local feature coding vector and the aligned second alternative local feature coding vector are input into the causal chain unit screening network to obtain the causal chain unit closed-loop strength factor. Furthermore, based on the comparison between the causal chain unit closed-loop strength factor and a preset threshold, it is determined whether the first alternative local feature coding vector and the second alternative local feature coding vector constitute a causal chain unit.More specifically, in this example, the set of local feature coding vectors of the coal pile surface state and the set of local feature coding vectors of the coal pile spectrum are input into the causal chain unit screening network according to the following formula to obtain a set of {local feature coding vectors of the coal pile surface state, local feature coding vectors of the coal pile spectrum} feature pairs constituting the causal chain unit. The formula is:

[0057]

[0058]

[0059] Among them, Conv 1×1 Represents the point convolution layer, Conv 3×3 (·) is a convolution layer with a convolution kernel of 3×3, v 2j T v 2j The transposed vector of represents vector multiplication, Δ represents the first candidate-second candidate semantic alignment modulation flow field, v 1i ′ represents the first candidate local feature encoding vector after alignment, v 2j ′ represents the second candidate local feature encoding vector after alignment, arccosh(·) is the Poincare distance function, c(v 1i ′,v 2j ′) represents the closed-loop strength factor of the causal chain unit, θ is the preset threshold, {v 1i ,v 2j} pair The feature pair of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} constituting the causal chain unit is represented.

[0060] Specifically, the coal pile surface state-spectral feature joint perception unit 332 is used to perform feature joint perception encoding on the set of {coal pile surface state features, coal pile spectral features} features containing the causal chain unit to obtain a set of coal pile surface state-spectral multimodal joint perception encoding feature vectors. That is, in the technical solution of the present application, each of the {coal pile surface state local feature encoding vector, coal pile spectrum local feature encoding vector} feature pairs constituting the causal chain unit in the set of {coal pile surface state local feature encoding vector, coal pile spectrum local feature encoding vector} feature pairs constituting the causal chain unit is input into the causal chain unit semantic dynamic response network to obtain the set of coal pile surface state-spectral multimodal joint perception encoding feature vectors. It should be understood that in a complex open-pit coal mine environment, there is a complex nonlinear relationship between the surface state features and spectral features of the coal pile, and the causal chain unit screening network only preliminarily screens out the relevant feature pairs, which is far from enough to capture the deeper semantic interaction and causal dynamic changes between the coal pile surface state features and the coal pile spectral feature pairs. Therefore, in the technical solution of the present application, each of the {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} feature pairs constituting the causal chain unit in the set of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} feature pairs constituting the causal chain unit is input into the causal chain unit semantic dynamic response network, so as to deeply explore the individual semantics and causal relationship between the {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} feature pairs by using high-order dynamic interaction modeling, and the obtained coal pile surface state-spectrum multimodal joint perception coding feature vector set can more accurately reflect the characteristics of different coal piles in open-pit coal mines. These feature vectors not only contain the basic information of the surface state and spectral characteristics of the coal pile, but also accurately capture the semantics and causal relationship between the feature pairs. In actual mining, for some coal pile types that are difficult to distinguish, these feature vectors can provide more detailed feature descriptions to help the system identify the coal pile type more accurately. It is worth mentioning that in this process, the causal chain unit semantic dynamic response network will generate a set of dynamic weights or implicit dynamic mapping functions according to the semantic context of the coal pile features. These dynamic weights or functions can capture the nonlinear and complex interactions between the feature pairs {local feature encoding vector of coal pile surface state, local feature encoding vector of coal pile spectrum}, and realize the organic integration of semantics and causal relationship of feature pairs. In practical applications, the characteristics of coal piles will change with factors such as mining progress and environmental conditions. When the semantics in the feature pairs change, the network can adjust the output in real time through dynamic weights, imitating the response mechanism in the actual causal chain, thereby effectively enriching the relationship modeling capabilities between feature pairs, and making the output feature vectors more expressive at a wider range of semantic and causal levels.Through this process, the intrinsic connection between the surface state of the coal pile and the spectral characteristics can be represented in a more accurate and comprehensive way, providing strong support for the subsequent accurate identification of the type of coal pile in different scenarios. In a specific example, the set of {coal pile surface state characteristics, coal pile spectral characteristics} features containing the causal chain unit is subjected to feature joint perception encoding using the following formula to obtain a set of coal pile surface state-spectral multimodal joint perception encoding feature vectors. Wherein, the formula is:

[0061]

[0062] in, represents point subtraction, α and β are weighted hyperparameters, and W 1 and W 2 is the coal pile surface state-spectral multimodal joint perception weight matrix, Indicates point plus, v p(1i-2j) Represents the coal pile surface state-spectral multimodal joint perception coding feature vector.

[0063] Specifically, the feature aggregation unit 333 is used to perform feature aggregation on the set of the coal pile surface state-spectral multimodal joint perception coding feature vectors to obtain the coal pile surface state-spectral multimodal joint perception coding feature map. It should be understood that although the set of the coal pile surface state-spectral multimodal joint perception coding feature vectors contains rich information about the surface state and spectral characteristics of the coal pile, this information is scattered in each feature vector and lacks integrity and coherence. For example, in different mining areas, the surface state and spectral characteristics of the coal pile may have slight differences, and a single feature vector may only reflect part of the information and cannot accurately depict the overall picture of the coal pile. Therefore, further, the set of the coal pile surface state-spectral multimodal joint perception coding feature vectors is feature aggregated to form a coal pile surface state-spectral multimodal joint perception coding feature map with strong causal joint perception characteristics. Through feature aggregation, the information about the surface state and spectral characteristics of the coal pile in each feature vector can be fused, the isolation between the information can be eliminated, and a comprehensive and systematic coal pile feature representation can be constructed, providing richer and more accurate information for subsequent coal pile type identification.

[0064] Exemplarily, the coal pile type identification module 340 is used to identify the type of coal pile for the set of the coal pile surface state-spectral multimodal joint perception coding feature map to obtain a set of coal pile type identification results. Specifically, in the technical solution of the present application, each coal pile surface state-spectral multimodal joint perception coding feature map in the set of the coal pile surface state-spectral multimodal joint perception coding feature map is respectively input into a coal pile type identifier based on a classifier to obtain a set of coal pile type identification results. That is, the type of coal pile is identified based on the coal pile surface state-spectral multimodal features in the coal pile surface state-spectral multimodal joint perception coding feature map. In this process, the classifier accurately determines the type of coal pile by analyzing the information of the coal pile surface state and spectral features contained in the coal pile surface state-spectral multimodal joint perception coding feature map, as well as the relationship between these information. In this way, complex coal pile feature information can be converted into specific coal pile types, providing an accurate basis for subsequent mining task planning.

[0065] Exemplarily, the excavation task instruction generation module 350 is used to generate an excavation task instruction based on the set of the coal pile type identification results. That is, in the technical solution of the present application, first, based on the set of the coal pile type identification results, the excavation priority of the open-pit coal mine excavator is determined to obtain a set of excavation priorities. It should be understood that in areas with different geological structures, the types of coal piles are different. If the excavation sequence is not reasonably arranged according to the actual situation, the excavation efficiency may be reduced, thereby affecting the overall mining efficiency. Therefore, in order to enable the excavator to operate more efficiently and orderly, in the technical solution of the present application, the excavation priority is determined based on the accurate coal pile type identification results. In this process, the system analyzes the coal pile type identification results and comprehensively considers various factors, such as the quality of the coal pile, the difficulty of mining, the market demand, etc., to reasonably determine the excavation priority of each coal pile. Further, based on the set of excavation priorities, an excavation task instruction is generated. For example, for coal piles with high quality, easy mining and large market demand, their excavation priority is increased, and excavation is carried out first to obtain higher economic benefits. For coal piles that are difficult to mine but of good quality, the excavation sequence can be reasonably arranged according to the equipment condition and mining plan to avoid equipment damage or delays due to blind excavation. Through reasonable excavation priority arrangement, equipment wear and energy consumption can be reduced, and mining costs can be reduced. In this way, the defects of the existing system in excavation priority decision-making are effectively overcome, and the operating efficiency and resource utilization of open-pit coal mine excavators are improved.

[0066] As described above, the automatic control system 300 of the open-pit coal mine excavator according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an automatic control algorithm of the open-pit coal mine excavator. In a possible implementation, the automatic control system 300 of the open-pit coal mine excavator according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the automatic control system 300 of the open-pit coal mine excavator can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal. Of course, the automatic control system 300 of the open-pit coal mine excavator can also be one of the many hardware modules of the wireless terminal.

[0067] Alternatively, in another example, the automatic control system 300 of the open-pit coal mine excavator and the wireless terminal may also be separate devices, and the automatic control system 300 of the open-pit coal mine excavator may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0068] Furthermore, the present application also provides an automatic control method for an open-pit coal mine excavator.

[0069] Figure 4 FIG. 1 is a flow chart of an automatic control method for an open-pit coal mine excavator according to an embodiment of the present application. Figure 4 As shown, the automatic control method of an open-pit coal mine excavator according to an embodiment of the present application includes the steps of: S1, obtaining the coal pile surface state image and the coal pile spectral image of each coal pile collected by the spectral sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectral image}. S2, extracting the coal pile surface state features and coal pile spectral features of the set of {coal pile surface state image, coal pile spectral image} to obtain a set of {coal pile surface state feature map, coal pile spectral feature map}. S3, performing feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectral feature map} to obtain a set of coal pile surface state-spectral multimodal joint perception coding feature maps. S4, performing coal pile type recognition on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type recognition results. S5, generating an excavation task instruction based on the set of coal pile type recognition results.

[0070] In summary, the automatic control method of the open-pit coal mine excavator according to the embodiment of the present application is explained, by obtaining the surface state image of the coal pile and the set of spectral images of the coal pile collected by the spectral sensor and the camera, and using the neural network model based on deep learning to extract features of the two respectively to capture the external morphological characteristics and intrinsic attribute characteristics of the coal pile, so as to obtain the surface state characteristics and spectral feature representation of the coal pile, and further perform multimodal comprehensive analysis on the surface state characteristics and spectral characteristics to dig out the essential correlation feature representation between the two, and accurately judge the type of the coal pile based on this. In this way, it is possible to achieve accurate identification of the coal pile, thereby intelligently generating excavation task instructions, improving excavation efficiency and resource utilization.

[0071] The various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the various embodiments described, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary technicians in the technical field to understand the various embodiments disclosed herein. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. An automatic control system for an open-pit coal mine excavator, characterized in that: include: An image acquisition module, used for acquiring the coal pile surface state image and the coal pile spectrum image of each coal pile acquired by the spectrum sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image}; A feature extraction module, used for extracting the coal pile surface state features and coal pile spectrum features of the set of {coal pile surface state image, coal pile spectrum image} to obtain a set of {coal pile surface state feature map, coal pile spectrum feature map}; A feature interaction analysis module, used for performing feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectrum feature map} to obtain a set of coal pile surface state-spectrum multimodal joint perception coding feature maps; A coal pile type identification module, used for performing coal pile type identification on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type identification results; The mining task instruction generation module is used to generate mining task instructions based on the set of coal pile type identification results.

2. The automatic control system of an open-pit coal mine excavator according to claim 1, characterized in that: The feature extraction module comprises: A coal pile surface state feature extraction unit, used for inputting each coal pile surface state image in the set of coal pile surface state images into a coal pile surface state feature extractor based on a depthwise separable convolution model to obtain a set of coal pile surface state feature maps; The coal pile spectral feature extraction unit is used to input each coal pile spectral image in the set of coal pile spectral images into a coal pile spectral feature extractor based on a deep separable convolution model to obtain a set of coal pile spectral feature maps.

3. The automatic control system of an open-pit coal mine excavator according to claim 2, characterized in that: The coal pile surface state feature extraction unit is also used for: Cascading each coal pile surface state image in the set of coal pile surface state images along the channel dimension to obtain a coal pile surface state feature tensor; Using each layer of the deep separable convolution model, in the forward pass of the layer, convolution processing, pooling processing and nonlinear activation processing are performed on the feature tensor of the surface state feature of the coal pile based on the separable convolution kernel, so that the last layer of the deep separable convolution model outputs a set of surface state feature maps of the coal pile.

4. The automatic control system of an open-pit coal mine excavator according to claim 3, characterized in that: The feature interaction analysis module includes: A feature pair construction unit, used to perform causal association analysis on the coal pile surface state feature map and the coal pile spectrum feature map, and obtain a set of {coal pile surface state feature, coal pile spectrum feature} feature pairs constituting a causal chain unit based on the analysis result; A coal pile surface state-spectral feature joint perception unit, used for performing feature joint perception encoding on the set of {coal pile surface state feature, coal pile spectral feature} features containing the causal chain unit to obtain a set of coal pile surface state-spectral multimodal joint perception encoding feature vectors; A feature aggregation unit is used to perform feature aggregation on the set of the coal pile surface state-spectral multimodal joint perception coding feature vectors to obtain the coal pile surface state-spectral multimodal joint perception coding feature map.

5. The automatic control system of an open-pit coal mine excavator according to claim 4, characterized in that: The feature pair construction unit is also used for: Performing feature decoupling on the coal pile surface state feature map and the coal pile spectrum feature map along the channel dimension of the coal pile surface state feature map and the coal pile spectrum feature map to obtain a set of coal pile surface state local feature coding vectors and a set of coal pile spectrum local feature coding vectors; The set of local feature coding vectors of the coal pile surface state and the set of local feature coding vectors of the coal pile spectrum are input into the causal chain unit screening network to obtain a set of feature pairs of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} constituting the causal chain unit as the set of feature pairs of {coal pile surface state feature, coal pile spectrum feature}.

6. The automatic control system of an open-pit coal mine excavator according to claim 5, characterized in that: The coal pile surface state-spectral feature joint perception unit is also used to: input each feature pair of {coal pile surface state local feature coding vector, coal pile spectrum local feature coding vector} constituting the causal chain unit in the set of feature pairs constituting the causal chain unit into the causal chain unit semantic dynamic response network to obtain the set of coal pile surface state-spectrum multimodal joint perception coding feature vectors.

7. The automatic control system of an open-pit coal mine excavator according to claim 6, characterized in that: The coal pile type identification module is also used for: Each coal pile surface state-spectral multimodal joint perceptual coding feature map in the set of coal pile surface state-spectral multimodal joint perceptual coding feature maps is respectively input into a classifier-based coal pile type identifier to obtain a set of coal pile type identification results.

8. The automatic control system of an open-pit coal mine excavator according to claim 7, characterized in that: The mining task instruction generating module is further used for: Based on the set of coal pile type identification results, determining the excavation priority of the open-pit coal mine excavator to obtain a set of excavation priorities; Based on the set of mining priorities, a mining task instruction is generated.

9. An automatic control method for an open-pit coal mine excavator, characterized in that: include: Acquire the coal pile surface state image and the coal pile spectrum image of each coal pile collected by the spectrum sensor and the camera to obtain a set of {coal pile surface state image, coal pile spectrum image}; Extracting the coal pile surface state features and coal pile spectrum features of the set of {coal pile surface state image, coal pile spectrum image} to obtain a set of {coal pile surface state feature map, coal pile spectrum feature map}; Performing feature interaction analysis based on strong causal interconnection on the set of {coal pile surface state feature map, coal pile spectrum feature map} to obtain a set of coal pile surface state-spectrum multimodal joint perception coding feature maps; Performing coal pile type recognition on the set of coal pile surface state-spectral multimodal joint perception coding feature maps to obtain a set of coal pile type recognition results; Based on the set of coal pile type identification results, an excavation task instruction is generated.

Images