Automatic control system and method for strip mine electric shovel

Abstract

The invention discloses a strip mine electric shovel automatic control system and method, and the system employs a neural network model based on deep learning to carry out the feature extraction of an obstacle target region, so as to excavate the deep semantic features in the obstacle target region, so as to improve the recognition precision of the specific type and characteristics of an obstacle. Further performing joint coding analysis on the type of the obstacle and the relative position of the obstacle to deeply understand the type of the obstacle and the distance relation between the obstacle and the electric shovel so as to obtain interactive coding characteristics capable of comprehensively reflecting the type of the obstacle and distance information, and determining the distance of the electric shovel on the basis of the interactive coding characteristics. Whether an emergency braking command is generated is determined. In this way, the obstacle detection precision is improved, more accurate and more intelligent recognition and response to obstacles are achieved, and the safety of strip mine electric shovel operation is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent control, and more specifically, to an automatic control system and method for an open-pit mine electric shovel. Background Art

[0002] In the operating environment of open-pit mines, electric shovels are key equipment, and their operating safety and efficiency are directly related to the smooth progress of the entire mining operation. However, in the current electric shovel operation practice, the detection and handling of obstacles still face many challenges and shortcomings.

[0003] Existing obstacle detection systems mainly rely on a single type of sensor or lidar to identify obstacles and their distance from the shovel. This reliance on a single data source is difficult to provide comprehensive and accurate information, especially in complex and changeable actual operating environments. For example, when encountering severe weather conditions such as rain, snow, fog, etc., traditional sensors may not work effectively, resulting in a significant decrease in the ability to perceive the surrounding environment, increasing the risk of misjudgment or missed detection. In addition, since different types of obstacles (such as rocks, trees, other mechanical equipment, etc.) have different physical properties, a single type of sensor often has difficulty accurately distinguishing the specific types of these obstacles, thereby affecting the effectiveness of subsequent decision-making.

[0004] In addition, the response speed of the existing system is also an issue that needs to be addressed. Especially when processing large amounts of image data, the algorithm complexity is high and the calculation time is long, which may lead to a failure to respond in time in emergency situations, greatly increasing the risk of accidents. For example, when an obstacle suddenly appears in front of a high-speed moving electric shovel, if the system cannot quickly identify and take action, it may cause a serious collision accident.

[0005] In order to overcome the above problems, an optimized open-pit mine electric shovel automatic control system is expected. Summary of the invention

[0006] The embodiments of the present application aim to solve at least one of the technical problems existing in the prior art, and provide an open-pit mine electric shovel automatic control system and method, which uses a neural network model based on deep learning to perform feature extraction on the obstacle target area to mine the deep semantic features in the obstacle target area, so as to improve the recognition accuracy of the specific type and characteristics of the obstacle; further, by performing joint coding analysis on the obstacle type and the relative position of the obstacle, a deep understanding of the obstacle type and the distance relationship between the obstacle type and the electric shovel is achieved, so as to obtain interactive coding features that can comprehensively reflect the obstacle type and distance information, and determine whether to generate an emergency braking instruction based on the interactive coding features. In this way, the detection accuracy of the obstacle is improved, and then more accurate and intelligent identification and response to the obstacle is achieved, which greatly improves the safety of the open-pit mine electric shovel operation.

[0007] On the one hand, the present application provides an open-pit mine electric shovel automatic control system, comprising:

[0008] A surrounding environment image acquisition module is used to acquire surrounding environment images through a camera deployed on an open-pit mine shovel;

[0009] An image filtering module, used for performing an adaptive median filtering process on the surrounding environment image to obtain a filtered surrounding environment image;

[0010] An obstacle target area recognition module is used to perform obstacle recognition on the filtered surrounding environment image to obtain a recognition result, wherein the recognition result is used to indicate whether there is an obstacle;

[0011] A distance detection module, configured to detect the distance between the obstacle and the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel in response to the recognition result that there is an obstacle;

[0012] The emergency braking module is used to determine whether to generate an emergency braking instruction based on the type of the obstacle and the distance of the obstacle from the open-pit mining shovel.

[0013] Optionally, the obstacle target area identification module is further used to:

[0014] Inputting the filtered surrounding environment image into an obstacle target identifier based on a YOLO model to obtain an obstacle target area image;

[0015] The obstacle target area image is input into a classifier-based obstacle recognition model to obtain the recognition result, and the recognition result is used to indicate whether there is an obstacle.

[0016] Optionally, the emergency braking module comprises:

[0017] An obstacle image feature extraction unit, used for performing obstacle image feature extraction on the obstacle target area image to obtain an obstacle image coding feature map;

[0018] A one-hot encoding unit, used for one-hot encoding the distance of the open-pit mining shovel to obtain a distance one-hot encoding vector;

[0019] A multimodal feature fusion unit, used for performing multimodal feature fusion based on core clue guidance on the obstacle image encoding feature map and the distance one-hot encoding vector to obtain an obstacle type-distance cross-modal significant interaction encoding feature;

[0020] The emergency braking instruction generating unit is used to determine whether to generate an emergency braking instruction based on the obstacle type-distance cross-modal significant interaction coding feature.

[0021] Optionally, the obstacle image feature extraction unit is further used to:

[0022] The obstacle target area image is input into an obstacle image feature extractor based on a Mobile-Former model to obtain the obstacle image encoding feature map.

[0023] Optionally, the multimodal feature fusion unit includes:

[0024] A core clue extraction subunit, configured to construct an obstacle type-distance core clue weaving template matrix based on the obstacle image encoding feature map and the core clue encoding features of the distance one-hot encoding vector;

[0025] A cross-modal fine-grained interactive encoding subunit, configured to perform cross-modal fine-grained interactive encoding on the distance one-hot encoding vector and the obstacle image encoding feature map based on the obstacle type-distance core clue weaving template matrix to obtain a set of obstacle type-distance cross-modal fine-grained interactive encoding vectors;

[0026] The position mean calculation subunit is used to calculate the position mean vector of the set of the obstacle type-distance cross-modal fine-grained interaction coding vectors to obtain the obstacle type-distance cross-modal significant interaction coding vector as the obstacle type-distance cross-modal significant interaction coding feature.

[0027] Optionally, the core clue extraction subunit is further used for:

[0028] Performing point convolution coding-based core clue extraction on the distance one-hot encoding vector to obtain a distance core clue encoding vector;

[0029] Extracting an obstacle type core clue coding vector from the obstacle image coding feature map;

[0030] A core clue weaving template matrix between the distance core clue encoding vector and the obstacle type core clue encoding vector is constructed to obtain the obstacle type-distance core clue weaving template matrix.

[0031] Optionally, the cross-modal fine-grained interaction encoding subunit is further used to:

[0032] Performing feature decoupling along the channel dimension on the obstacle image encoding feature map to obtain a set of obstacle type local feature matrices;

[0033] The distance core clue encoding vector is used as a query vector, each obstacle type local feature matrix in the set of obstacle type local feature matrices is used as a key matrix, and the obstacle type-distance core clue weaving template matrix is ​​used as a prior information constraint matrix, which is input into a cross-modal template constraint encoder based on a heterogeneous converter to obtain the set of cross-modal fine-grained interaction encoding vectors.

[0034] Optionally, the emergency braking instruction generating unit is further used to:

[0035] The obstacle type-distance cross-modal significant interaction encoding vector is input into a fully connected layer and a Softmax classification function to obtain a control result, and the control result is used to determine whether to generate an emergency braking instruction.

[0036] On the other hand, the present application provides an open-pit mine electric shovel automatic control method, comprising:

[0037] The camera deployed on the open-pit mine shovel collects images of the surrounding environment;

[0038] Performing adaptive median filtering on the surrounding environment image to obtain a filtered surrounding environment image;

[0039] Performing obstacle recognition on the filtered surrounding environment image to obtain a recognition result, wherein the recognition result is used to indicate whether an obstacle exists;

[0040] In response to the identification result that an obstacle exists, detecting the distance between the obstacle and the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel;

[0041] A determination is made whether to generate an emergency braking command based on the type of the obstacle and the distance of the obstacle from the open pit mining shovel.

[0042] Compared with the related art, the automatic control system and method of an open-pit mine shovel provided by the present application uses a neural network model based on deep learning to extract features from the obstacle target area to mine the deep semantic features in the obstacle target area, so as to improve the recognition accuracy of the specific type and characteristics of the obstacle; further, by performing joint coding analysis on the obstacle type and the relative position of the obstacle, a deep understanding of the obstacle type and the distance relationship between the obstacle type and the electric shovel is achieved, so as to obtain interactive coding features that can comprehensively reflect the obstacle type and distance information, and determine whether to generate an emergency braking command based on the interactive coding features. In this way, the detection accuracy of obstacles is improved, thereby achieving more accurate and intelligent identification and response to obstacles, greatly improving the safety of open-pit mine shovel operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a block diagram of an open-pit mine electric shovel automatic control system according to an embodiment of the present application;

[0044] Figure 2 A schematic diagram of data flow of an open-pit mine electric shovel automatic control system according to an embodiment of the present application;

[0045] Figure 3 is a block diagram of an emergency brake module in an open-pit mine electric shovel automatic control system according to an embodiment of the present application;

[0046] Figure 4 is a block diagram of a multi-modal feature fusion unit in an open-pit mine electric shovel automatic control system according to an embodiment of the present application;

[0047] Figure 5 It is a flow chart of the automatic control method of the open-pit mine electric shovel according to the embodiment of the present application. DETAILED DESCRIPTION

[0048] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0049] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0050] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0051] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0052] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0053] In the technical solution of the present application, an open-pit mine electric shovel automatic control system and method are proposed, which first uses a camera to capture the surrounding environment image and removes the image noise through adaptive filtering technology. Then, an efficient YOLO model is used for obstacle recognition to quickly determine whether there is an obstacle on the working path. Once an obstacle is found, the laser radar will be activated to accurately measure its distance. For obstacles that may pose a threat, the system combines its type and distance information to determine whether emergency braking needs to be initiated. In this way, the safety of electric shovel operation can be improved and collision accidents can be avoided. Figure 1 It is a block diagram of an open-pit mine electric shovel automatic control system according to an embodiment of the present application.

[0054] Figure 2 Schematic diagram of data flow of open pit mine electric shovel automatic control system according to the embodiment of the present application. Figure 1 and Figure 2As shown, according to an embodiment of the present application, an open-pit mine shovel automatic control system 300 includes: a surrounding environment image acquisition module 310, which is used to collect surrounding environment images through a camera deployed on the open-pit mine shovel. An image filtering module 320, which is used to perform adaptive median filtering on the surrounding environment image to obtain a filtered surrounding environment image. An obstacle target area recognition module 330, which is used to perform obstacle recognition on the filtered surrounding environment image to obtain a recognition result, and the recognition result is used to indicate whether there is an obstacle. A distance detection module 340, which is used to detect the distance between the obstacle and the open-pit mine shovel through a laser radar deployed on the open-pit mine shovel in response to the recognition result that there is an obstacle. An emergency braking module 350, which is used to determine whether to generate an emergency braking instruction based on the type of the obstacle and the distance between the obstacle and the open-pit mine shovel.

[0055] Exemplarily, the surrounding environment image acquisition module 310 is used to collect surrounding environment images through a camera deployed on an open-pit mine shovel. It should be understood that the operating environment of an open-pit mine is complex and changeable, and traditional single sensor systems often perform poorly when dealing with severe weather conditions or complex terrain, resulting in limited perception of the surrounding environment. Collecting surrounding environment images through a camera deployed on an open-pit mine shovel can obtain richer visual information, thereby making up for the shortcomings of a single sensor. The deployment of cameras can not only provide high-resolution image data, but also maintain relatively stable performance under a variety of lighting conditions, which is crucial to improving the overall perception ability of the system.

[0056] Exemplarily, the image filtering module 320 is used to perform adaptive median filtering on the surrounding environment image to obtain a filtered surrounding environment image. It should be understood that when the camera captures the original image of the surrounding environment, these images may contain various types of noise, such as noise caused by dust, bad weather conditions or equipment vibration. In order to ensure that the surrounding environment image can accurately reflect the actual environmental information and reduce the impact of noise on subsequent processing steps (such as obstacle recognition), the surrounding environment image is adaptively median filtered in the technical solution of the present application. Adaptive median filtering is a nonlinear image processing method that can effectively remove impulse noise and protect the details of the image from being blurred to a certain extent. It is worth mentioning that the core of adaptive median filtering lies in its "adaptive" characteristics, which means that the size of the filter will be dynamically adjusted according to the specific situation of the local area of ​​the image. In specific implementation, the algorithm calculates the median in a window centered at each pixel position and compares it with the central pixel value. If the central pixel is considered to be noise (i.e., it is significantly different from most other pixel values), the central pixel value is replaced by the median in the window. Otherwise, the original value remains unchanged. In this process, the window size can be changed according to the actual situation, starting from a small range and gradually expanding until a suitable size is found, so that the detailed features of the image can be retained as much as possible while effectively removing noise. In a specific example, when applying adaptive median filtering, an initial small window size is first selected as the starting point, and then the window size is gradually increased until the preset maximum value is reached or specific conditions are met. This strategy allows the filter to better adapt to different types of noise and image structures. For areas with small but important details, a smaller window size can avoid over-smoothing, while for areas with more severe noise, a larger window size can be used to more effectively remove noise. The image processed by adaptive median filtering will be clearer and smoother, reducing unnecessary noise interference, and providing a high-quality data foundation for subsequent obstacle target area recognition.

[0057] Exemplarily, the obstacle target area recognition module 330 is used to perform obstacle recognition on the filtered surrounding environment image to obtain a recognition result, and the recognition result is used to indicate whether there is an obstacle. That is, in the technical solution of the present application, first, the filtered surrounding environment image is input into the obstacle target identifier based on the YOLO model to obtain an obstacle target area image. It should be understood that by using the YOLO model to analyze the filtered surrounding environment image, all obstacles in the working area that may threaten the safety of the electric shovel can be accurately and quickly identified, and the specific location and range of the obstacles can be determined. In this way, it is ensured that the system can respond in the shortest time, discover potential dangers in time, and greatly improve the response speed and safety of the system. It is worth mentioning that the YOLO model can predict the location and category of all obstacle targets by processing the entire image at a time. Compared with other models, the YOLO model is more efficient when processing large amounts of data, and is particularly suitable for application scenarios with high real-time requirements such as automatic control of open-pit mine electric shovels. Next, the obstacle target area image is input into the classifier-based obstacle recognition model to obtain the recognition result, and the recognition result is used to indicate whether there is an obstacle. Wherein, the classifier completes the obstacle recognition based on the obstacle target area features in the obstacle target area image. That is, in the technical solution of the present application, the labels of the classifier include the existence of obstacles (first label), and the absence of obstacles (second label), wherein the classifier determines which classification label the obstacle target area image belongs to through a soft maximum function. It is worth noting that the first label p1 and the second label p2 here do not contain artificially set concepts. In fact, during the training process, the computer model does not have the concept of "whether there is an obstacle". It only has two classification labels and the probability of the output feature under these two classification labels, that is, the sum of p1 and p2 is one. Therefore, the classification result of whether there is an obstacle is actually converted into a binary classification class probability distribution that conforms to the laws of nature through the classification label. In essence, the physical meaning of the natural probability distribution of the label is used, rather than the linguistic text meaning of "whether there is an obstacle".

[0058] Exemplarily, the distance detection module 340 is used to detect the distance of the obstacle from the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel in response to the recognition result that there is an obstacle. That is, once it is determined that there are potential obstacles in the surrounding environment, the laser radar system will be triggered to start working. Among them, the laser radar equipment is installed at different key positions of the shovel to ensure that all possible directions and angles can be covered, thereby achieving all-round monitoring. In this process, the laser radar sensor continuously emits laser pulses outward to form a dense laser point cloud to scan the area around the shovel. When the laser pulse encounters an obstacle, it will be reflected back to the laser radar sensor. According to the principle of constant speed of light, by calculating the time difference required for the laser pulse to be emitted and received, combined with the speed of light value, the distance between the shovel and the obstacle can be accurately calculated. It is worth noting that since the laser radar can provide high-resolution distance information, it can not only determine the existence of obstacles, but also depict the outline of the obstacles and their specific position relative to the shovel. In this way, the emergency braking module of the control system can quickly assess the risk level of the current situation based on the distance information obtained and the obstacle type previously obtained through image recognition, and decide whether to take emergency braking measures or other appropriate action strategies to avoid potential collision accidents and ensure the safety of operations. In this way, combined with advanced image processing technology and the distance measurement capability of LiDAR, the open-pit mine shovel automatic control system achieves a high degree of perception and intelligent response to the surrounding environment, greatly improving the safety and efficiency of mine operations.

[0059] Exemplarily, the emergency braking module 350 is used to determine whether to generate an emergency braking instruction based on the type of the obstacle and the distance of the obstacle from the open-pit mining shovel. In particular, in a specific example of the present application, Figure 3 As shown, the emergency braking module 350 includes: an obstacle image feature extraction unit 351, which is used to extract obstacle image features from the obstacle target area image to obtain an obstacle image coding feature map. A one-hot encoding unit 352, which is used to one-hot encode the distance of the open-pit mine shovel to obtain a distance one-hot encoding vector. A multimodal feature fusion unit 353, which is used to perform multimodal feature fusion based on core clue guidance on the obstacle image coding feature map and the distance one-hot encoding vector to obtain an obstacle type-distance cross-modal significant interaction coding feature. An emergency braking instruction generation unit 354, which is used to determine whether to generate an emergency braking instruction based on the obstacle type-distance cross-modal significant interaction coding feature.

[0060] Specifically, the obstacle image feature extraction unit 351 is used to extract obstacle image features from the obstacle target area image to obtain an obstacle image coding feature map. Specifically, the obstacle target area image is input into an obstacle image feature extractor based on the Mobile-Former model to obtain the obstacle image coding feature map. Considering that for an environment such as an open-pit mine, the system often needs to process a large amount of data with limited computing resources and ensure real-time and accuracy, the Mobile-Former model combines the lightweight advantages of MobileNet and the powerful representation capabilities of Transformer, and is particularly suitable for application scenarios with limited resources but high performance requirements. Therefore, in the technical solution of the present application, the obstacle target area image is input into an obstacle image feature extractor based on the Mobile-Former model to mine the deep semantic expression of obstacles in the obstacle target area, so as to improve the recognition accuracy of obstacle types, thereby enhancing the system's ability to distinguish subtle differences between different obstacles. In addition, the Mobile-Former model captures detail information by introducing a local attention mechanism and uses a global attention mechanism to obtain contextual information. This enables the model to not only accurately extract the key features of the obstacle type when processing obstacle images, but also effectively understand the relationship between the obstacle and the environment, providing high-quality data support for subsequent obstacle detection.

[0061] Specifically, the one-hot encoding unit 352 is used to perform one-hot encoding on the distance of the open-pit mine shovel to obtain a one-hot encoding vector of the distance. Among them, one-hot encoding is a method of converting categorical data into numerical data by mapping each possible value to an independent dimension and using 0 or 1 to indicate whether the category exists. In the technical solution of the present application, by performing one-hot encoding on the distance between the open-pit mine shovel and the obstacle, the continuous distance value can be discretized into a series of binary vectors, so that the distance information can be better understood and processed by the machine learning model. In addition, considering that in the operating environment of the open-pit mine, the distance between the shovel and the obstacle changes dynamically, and this change is crucial to the decision-making process. For example, when approaching an obstacle, even a slight change in distance may require immediate adoption of different action strategies. By performing one-hot encoding on the distance information, not only can the importance of different distance intervals be accurately captured, but also the sensitivity of the model to subtle changes in distance can be enhanced, thereby ensuring the response speed and accuracy of the subsequent system to obstacle identification.

[0062] Specifically, the multimodal feature fusion unit 353 is used to perform core clue-guided multimodal feature fusion on the obstacle image encoding feature map and the distance one-hot encoding vector to obtain obstacle type-distance cross-modal significant interaction encoding features. Considering that the obstacle type and its relative position (i.e., distance) are two interrelated but independent information sources, it is difficult to achieve a comprehensive understanding of the environmental state by relying on any one of the information alone, which leads to errors in obstacle detection. Traditional multimodal fusion methods often rely on explicit alignment or simple feature splicing, which leads to the problem that deep semantic dependencies between different modalities are difficult to fully capture. For example, when dealing with obstacle detection in complex environments, the information provided by a single type of sensor or lidar may not be sufficient to fully and accurately describe the specific situation of the obstacle, especially in the face of natural environmental changes (such as rain, snow, fog, etc.), this limitation is more obvious, and it is easy to misjudge or miss the situation. In order to enhance the collaborative expression ability between obstacle type and distance modalities, so as to further improve the detection accuracy of obstacles by the system, in the technical solution of the present application, the obstacle image coding feature map and the distance unique hot coding vector are subjected to multimodal feature fusion based on core clue guidance, that is, based on the core clue features of obstacle type and distance, the semantic relevance and complementarity of multimodal features under obstacle detection are fused by constructing a correlation template and a cross-domain interaction mechanism between obstacle type and distance, so as to generate a cross-modal fine-grained interactive coding feature representation with high semantic consistency and significant interactivity. In this way, the system can not only accurately identify the specific type of obstacle, but also make a more accurate risk assessment based on the actual distance between it and the electric shovel. For example, when a large stone is detected in front and the distance is close, the system can quickly determine the potential danger and take immediate action, such as starting the emergency braking procedure, so as to avoid the occurrence of a collision accident. In this way, the system can improve the response speed while ensuring high accuracy of obstacle detection, thereby improving the accuracy of automatic control of the electric shovel. Specifically, in a specific example of the present application, such as Figure 4As shown, the multimodal feature fusion unit 353 includes: a core clue extraction subunit 3531, which is used to construct an obstacle type-distance core clue weaving template matrix based on the core clue coding features of the obstacle image coding feature map and the distance one-hot coding vector. A cross-modal fine-grained interaction coding subunit 3532 is used to perform cross-modal fine-grained interaction coding on the distance one-hot coding vector and the obstacle image coding feature map based on the obstacle type-distance core clue weaving template matrix to obtain a set of obstacle type-distance cross-modal fine-grained interaction coding vectors. A position mean calculation subunit 3533 is used to calculate the position mean vector of the set of obstacle type-distance cross-modal fine-grained interaction coding vectors to obtain an obstacle type-distance cross-modal significant interaction coding vector as the obstacle type-distance cross-modal significant interaction coding feature.

[0063] More specifically, the core clue extraction subunit 3531 is used to construct an obstacle type-distance core clue weaving template matrix based on the core clue encoding features of the obstacle image encoding feature map and the distance one-hot encoding vector. In an embodiment of the present application, first, the distance one-hot encoding vector is subjected to core clue extraction based on point convolution encoding to obtain a distance core clue encoding vector. That is, the most critical part, that is, those important features that can directly reflect the relative position between the electric shovel and the obstacle, is extracted from the original distance information through the point convolution operation. In this way, it is ensured that even in a complex mining environment, the system can quickly and accurately capture the location information of the obstacle, providing a solid foundation for subsequent decision-making. Next, the obstacle type core clue encoding vector is extracted from the obstacle image encoding feature map. Considering that the obstacle image usually carries rich spatial dimension information, the core features of the obstacle image encoding feature map are extracted to emphasize the key parts in the image using the regional attention mechanism to maintain the integrity of the structured information. In this way, it helps the system to identify obstacles more accurately. Furthermore, a core clue weaving template matrix is ​​constructed between the distance core clue encoding vector and the obstacle type core clue encoding vector to create an explicit association relationship between the two modalities, so that the distance information and the obstacle type information can complement and support each other, thereby generating more accurate interactive coding features. This approach not only alleviates the problem of difficulty in direct interaction due to differences in data formats in traditional methods, but also captures the fine-grained dependency between distance and obstacle type through a template constraint learning mechanism, thereby enhancing the expression capability of obstacle type-distance cross-modal collaboration. In a specific example of the present application, based on the core clue encoding features of the obstacle image encoding feature map and the distance one-hot encoding vector, the obstacle type-distance core clue weaving template matrix is ​​constructed using the following core clue weaving template construction formula. Among them, the core clue weaving template construction formula is:

[0064] v c1 =LeakyReLU{Conv 1×1 (v1)+b1}

[0065]

[0066] Where v1 represents the distance one-hot encoding vector, Conv 1×1 represents the point convolution layer, b1 is the bias vector, LeakyReLU represents the LeakyReLU activation function, and v c1 represents the distance core clue coding vector, F2 represents the obstacle image coding feature map, Partition(·) is the grid partitioning operation, represents each obstacle image encoding local feature map in the set of obstacle image encoding local feature maps, sigmoid(·) is the sigmoid function, Conv 3×3 (·) is a convolution layer with a convolution kernel of 3×3. represents the obstacle image encoding local enhanced feature map corresponding to the i-th obstacle image encoding local feature map, Globalpool represents the global pooling operation, Indicates the The corresponding obstacle image encodes the local enhanced feature vector, max(·) and min(·) represent the maximum and minimum values ​​extracted from the feature vector respectively. for The variance of , γ is a hyperparameter, D i express The corresponding obstacle type core clue local energy significance description factor, v c2 represents the obstacle type core clue encoding vector, exp(·) represents the exponential function with the natural constant e as the base, M tem A template matrix weaving a core cue representing the obstacle type-distance.

[0067] More specifically, the cross-modal fine-grained interactive encoding subunit 3532 is used to perform cross-modal fine-grained interactive encoding on the distance unique hot encoding vector and the obstacle image encoding feature map based on the obstacle type-distance core clue weaving template matrix to obtain a set of obstacle type-distance cross-modal fine-grained interactive encoding vectors. In an embodiment of the present application, first, the obstacle image encoding feature map is feature decoupled along the channel dimension to obtain a set of obstacle type local feature matrices. Here, the obstacle image encoding feature map is feature decoupled along the channel dimension so that the system can capture various details of the obstacle more finely, thereby improving the accuracy of recognition. Further, the distance core clue encoding vector is used as a query vector, each obstacle type local feature matrix in the set of obstacle type local feature matrices is used as a key matrix, and the obstacle type-distance core clue weaving template matrix is ​​used as a prior information constraint matrix, which is input into a cross-modal template constraint encoder based on a heterogeneous converter to obtain a set of cross-modal fine-grained interactive encoding vectors. That is, a cross-modal template constraint encoder based on a heterogeneous converter is used, the distance core clue encoding vector is used as the query vector, and the response is obtained from the set of local feature matrices of the obstacle type through the attention mechanism, and the obstacle type-distance core clue weaving template matrix is ​​used as a priori information constraint to perform cross-modal fine-grained interaction encoding. This constraint mechanism further enhances the characterization ability of the results by strengthening the standardization and accuracy in modal interaction. In a specific example of the present application, based on the obstacle type-distance core clue weaving template matrix, the distance one-hot encoding vector and the obstacle image encoding feature map are cross-modally fine-grained interactively encoded using the following cross-modal fine-grained interaction encoding formula to obtain a set of obstacle type-distance cross-modal fine-grained interaction encoding vectors. Among them, the cross-modal fine-grained interaction encoding formula is:

[0068] Decompose(F2)={M1,M2,...,M n}

[0069]

[0070] Wherein, Decompose(·) is a feature decoupling operation, M1, M2, ..., M2 represent local feature matrices of each obstacle type in the set of local feature matrices of the obstacle type, softmax(·) is a softmax function, Represents matrix multiplication, M i T represents the transposed matrix of the local feature matrix of the ith obstacle type in the set of local feature matrices of the obstacle type, v tiRepresents the i-th cross-modal fine-grained interaction coding vector in the set of cross-modal fine-grained interaction coding vectors.

[0071] More specifically, the position mean calculation subunit 3533 is used to calculate the position mean vector of the set of the obstacle type-distance cross-modal fine-grained interaction coding vectors to obtain the obstacle type-distance cross-modal significant interaction coding vector as the obstacle type-distance cross-modal significant interaction coding feature. In a specific example of the present application, the position mean vector of the set of the obstacle type-distance cross-modal fine-grained interaction coding vectors is calculated by the following formula to obtain the obstacle type-distance cross-modal significant interaction coding vector. Wherein, the formula is:

[0072]

[0073] Where n represents the scale of the set of cross-modal fine-grained interaction encoding vectors, v f Represents the obstacle type-distance cross-modal significant interaction encoding vector.

[0074] Specifically, the emergency braking instruction generation unit 354 is used to determine whether to generate an emergency braking instruction based on the obstacle type-distance cross-modal significant interaction coding feature. In the technical solution of the present application, the obstacle type-distance cross-modal significant interaction coding vector is input into the fully connected layer and the Softmax classification function to obtain a control result, and the control result is used to determine whether to generate an emergency braking instruction. In this process, the fully connected layer captures the complex correlation relationship between different modes from the input obstacle type-distance cross-modal significant interaction global feature representation, and converts it into a form, and further uses the Softmax function to determine whether to generate an emergency braking instruction. In an example, if the system detects that there is an obstacle ahead, the fully connected layer will comprehensively consider all relevant information (type, size, distance, etc.) of the obstacle and generate a comprehensive feature representation. Then, the Softmax function calculates the probability distribution of different control instructions based on this feature representation, such as continuing to move forward, slowing down, or immediately performing emergency braking, so as to achieve automatic control of the electric shovel. In this way, the obstacle detection accuracy during the operation of the electric shovel is improved, allowing the electric shovel to operate more intelligently and efficiently in complex and changeable working environments, greatly improving the safety and efficiency of mining operations.

[0075] As described above, the open-pit mine shovel automatic control system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an open-pit mine shovel automatic control algorithm. In a possible implementation, the open-pit mine shovel automatic control system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the open-pit mine shovel automatic control system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal. Of course, the open-pit mine shovel automatic control system 300 can also be one of the many hardware modules of the wireless terminal.

[0076] Alternatively, in another example, the open-pit mine shovel automatic control system 300 and the wireless terminal may also be separate devices, and the open-pit mine shovel automatic control system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0077] Furthermore, the present application also provides an automatic control method for an open-pit mine electric shovel.

[0078] Figure 5 FIG. 1 is a flow chart of an automatic control method for an open pit mine electric shovel according to an embodiment of the present application. Figure 5 As shown, the automatic control method of an open-pit mine shovel according to an embodiment of the present application includes the following steps: S1, collecting surrounding environment images by a camera deployed on the open-pit mine shovel. S2, performing adaptive median filtering on the surrounding environment image to obtain a filtered surrounding environment image. S3, performing obstacle recognition on the filtered surrounding environment image to obtain a recognition result, and the recognition result is used to indicate whether there is an obstacle. S4, in response to the recognition result that there is an obstacle, detecting the distance between the obstacle and the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel. S5, determining whether to generate an emergency braking command based on the type of the obstacle and the distance between the obstacle and the open-pit mine shovel.

[0079] In summary, the open-pit mine electric shovel automatic control method according to the embodiment of the present application is explained, which uses a neural network model based on deep learning to perform feature extraction on the obstacle target area to mine the deep semantic features in the obstacle target area, so as to improve the recognition accuracy of the specific type and characteristics of the obstacle. Further, by performing joint coding analysis on the obstacle type and the relative position of the obstacle, a deep understanding of the obstacle type and the relationship between the distance between the obstacle type and the electric shovel is achieved, so as to obtain interactive coding features that can comprehensively reflect the obstacle type and distance information, and based on the interactive coding features, determine whether to generate an emergency braking command. In this way, the detection accuracy of the obstacle is improved, thereby achieving more accurate and intelligent identification and response to the obstacle, greatly improving the safety of the open-pit mine electric shovel operation.

[0080] The various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the various embodiments described, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary technicians in the technical field to understand the various embodiments disclosed herein. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. An open-pit mine electric shovel automatic control system, characterized in that: include: A surrounding environment image acquisition module is used to acquire surrounding environment images through a camera deployed on an open-pit mine shovel; An image filtering module, used for performing an adaptive median filtering process on the surrounding environment image to obtain a filtered surrounding environment image; An obstacle target area recognition module is used to perform obstacle recognition on the filtered surrounding environment image to obtain a recognition result, wherein the recognition result is used to indicate whether there is an obstacle; A distance detection module, configured to detect the distance between the obstacle and the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel in response to the recognition result that there is an obstacle; The emergency braking module is used to determine whether to generate an emergency braking instruction based on the type of the obstacle and the distance of the obstacle from the open-pit mining shovel.

2. The open-pit mine electric shovel automatic control system according to claim 1 is characterized in that: The obstacle target area recognition module is also used for: Inputting the filtered surrounding environment image into an obstacle target identifier based on a YOLO model to obtain an obstacle target area image; The obstacle target area image is input into a classifier-based obstacle recognition model to obtain the recognition result, and the recognition result is used to indicate whether there is an obstacle.

3. The open-pit mine electric shovel automatic control system according to claim 2 is characterized in that: The emergency brake module comprises: An obstacle image feature extraction unit, used for performing obstacle image feature extraction on the obstacle target area image to obtain an obstacle image coding feature map; A one-hot encoding unit, used for one-hot encoding the distance of the open-pit mining shovel to obtain a distance one-hot encoding vector; A multimodal feature fusion unit, used for performing multimodal feature fusion based on core clue guidance on the obstacle image encoding feature map and the distance one-hot encoding vector to obtain an obstacle type-distance cross-modal significant interaction encoding feature; The emergency braking instruction generating unit is used to determine whether to generate an emergency braking instruction based on the obstacle type-distance cross-modal significant interaction coding feature.

4. The open-pit mine electric shovel automatic control system according to claim 3 is characterized in that: The obstacle image feature extraction unit is further used for: The obstacle target area image is input into an obstacle image feature extractor based on a Mobile-Former model to obtain the obstacle image encoding feature map.

5. The open-pit mine electric shovel automatic control system according to claim 4, characterized in that: The multimodal feature fusion unit comprises: A core clue extraction subunit, configured to construct an obstacle type-distance core clue weaving template matrix based on the obstacle image encoding feature map and the core clue encoding features of the distance one-hot encoding vector; A cross-modal fine-grained interactive encoding subunit, configured to perform cross-modal fine-grained interactive encoding on the distance one-hot encoding vector and the obstacle image encoding feature map based on the obstacle type-distance core clue weaving template matrix to obtain a set of obstacle type-distance cross-modal fine-grained interactive encoding vectors; The position mean calculation subunit is used to calculate the position mean vector of the set of the obstacle type-distance cross-modal fine-grained interaction coding vectors to obtain the obstacle type-distance cross-modal significant interaction coding vector as the obstacle type-distance cross-modal significant interaction coding feature.

6. The open-pit mine electric shovel automatic control system according to claim 5, characterized in that: The core clue extraction subunit is further used for: Performing point convolution coding-based core clue extraction on the distance one-hot encoding vector to obtain a distance core clue encoding vector; Extracting an obstacle type core clue coding vector from the obstacle image coding feature map; A core clue weaving template matrix between the distance core clue encoding vector and the obstacle type core clue encoding vector is constructed to obtain the obstacle type-distance core clue weaving template matrix.

7. The open-pit mine electric shovel automatic control system according to claim 6, characterized in that: The cross-modal fine-grained interaction encoding subunit is further used for: Performing feature decoupling along the channel dimension on the obstacle image encoding feature map to obtain a set of obstacle type local feature matrices; The distance core clue encoding vector is used as a query vector, each obstacle type local feature matrix in the set of obstacle type local feature matrices is used as a key matrix, and the obstacle type-distance core clue weaving template matrix is ​​used as a prior information constraint matrix, which is input into a cross-modal template constraint encoder based on a heterogeneous converter to obtain the set of cross-modal fine-grained interaction encoding vectors.

8. The open-pit mine electric shovel automatic control system according to claim 7, characterized in that: The emergency braking instruction generating unit is further used for: The obstacle type-distance cross-modal significant interaction encoding vector is input into a fully connected layer and a Softmax classification function to obtain a control result, and the control result is used to determine whether to generate an emergency braking instruction.

9. An open-pit mine electric shovel automatic control method, characterized in that: include: The camera deployed on the open-pit mine shovel collects images of the surrounding environment; Performing adaptive median filtering on the surrounding environment image to obtain a filtered surrounding environment image; Performing obstacle recognition on the filtered surrounding environment image to obtain a recognition result, wherein the recognition result is used to indicate whether an obstacle exists; In response to the identification result that an obstacle exists, detecting the distance between the obstacle and the open-pit mine shovel by a laser radar deployed on the open-pit mine shovel; A determination is made whether to generate an emergency braking command based on the type of the obstacle and the distance of the obstacle from the open pit mining shovel.

Images