Mine dump truck automatic unloading method and system based on visual positioning

Abstract

The application discloses a mine dump truck automatic unloading method and system based on visual positioning. The method comprises the following steps: obtaining historical overhead images of a dump truck and a mechanical arm; labeling the historical overhead images to obtain a first training data set; constructing an initial recognition model of the dump truck and the mechanical arm based on YOLOv8s; training the first training data set to obtain the recognition model of the dump truck and the mechanical arm; constructing a dump truck safety pin recognition model based on a traditional machine vision ORB feature point detection algorithm; establishing a stereo visual positioning system by using the recognition model of the dump truck and the mechanical arm and the dump truck safety pin recognition model to realize automatic unloading of the mine dump truck. The method introduces artificial intelligence recognition and cooperates with a self-adaptive control strategy, improves the unloading efficiency and safety performance of the mine dump truck, and reduces the labor cost.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of automatic control technology, and particularly relates to a mine dump truck automatic unloading method and system based on visual positioning. BACKGROUND

[0002] The dump truck is a key equipment for transporting materials such as ore and gangue in mine exploitation, and the automatic unloading device aims to solve the problems of low efficiency and poor safety of traditional manual or semi-automatic unloading, and is an important part of mine mechanical equipment. Modern mine exploitation tends to be large-scale and efficient, and most dump trucks use traditional manual or semi-automatic unloading methods (such as manual overturning and track inclination unloading). The traditional unloading methods all use manual methods, which have the problems of low parking positioning accuracy and inability to realize automatic unloading, and are difficult to match the production capacity requirements of continuous transportation lines. The previous manual unloading method requires personnel to operate at close range, which has safety hazards such as truck overturning and material sliding, and the method is low in efficiency and high in labor cost.

[0003] The existing dump truck automatic unloading technology has great defects, such as CN 220410552, which designs a new type of dump truck, modifies the body structure of the existing dump truck, realizes automatic unloading under external force loading, but cannot automatically unload the existing mine trucks, and is difficult to meet the needs of different mines for different structures of trucks. SUMMARY

[0004] In view of the deficiencies of the prior art, one of the purposes of the present application is to provide a mine dump truck automatic unloading method based on visual positioning, so as to achieve the purposes of high unloading efficiency, low labor cost and high safety performance.

[0005] The second purpose of the present application is to provide a system for realizing the mine dump truck automatic unloading method based on visual positioning.

[0006] The present application provides a mine dump truck automatic unloading method based on visual positioning, comprising the following steps:

[0007] S1. Obtain historical aerial images of the dump truck and the mechanical arm;

[0008] S2. Label the historical aerial images to obtain a first training data set;

[0009] S3. Construct an initial dump truck and mechanical arm recognition model based on YOLOv8s, and train using the first training data set to obtain a dump truck and mechanical arm recognition model;

[0010] S4. Construct a dump truck safety pin recognition model based on a traditional machine vision ORB feature point detection algorithm;

[0011] S5. Using the recognition model of the mine car and the mechanical arm obtained in step S3 and the recognition model of the mine car safety pin obtained in step S4, a stereoscopic vision positioning system is established to realize automatic unloading of the mine dump truck.

[0012] Further, the application is provided with two mechanical arms and a mechanical arm slide rail on the side of the mine dump truck track, the mine car is positioned, fixed, turned over and unloaded by the mechanical arms, the two mechanical arms are placed on different bases, the bases are on the mechanical arm slide rail, an electromagnet is arranged at the end of each mechanical arm, the first mechanical arm is responsible for fixing the mine car and turning over the mine car for unloading, the second mechanical arm is responsible for opening the locking mechanism, and the first mechanical arm is also responsible for pushing down the mine car and righting the mine car through the adsorption of the electromagnet after unloading is completed.

[0013] The second mechanical arm is responsible for opening the locking mechanism, specifically, the second mechanical arm is responsible for moving to the safety pin position, electrifying the electromagnet to adsorb the safety pin, rotating the safety pin to open the safety, and then re-electrifying the electromagnet to adsorb and rotate the safety pin to insert into the safety slot after the mine car is righted.

[0014] An eye-in-hand camera is arranged at the end of the second mechanical arm to identify the position of the safety pin.

[0015] A top camera is arranged at the top of the operation area of the mine dump truck to identify the relative position of the mine car and the mechanical arm.

[0016] In step S3, the initial recognition model of the mine car and the mechanical arm comprises a backbone network, a first metal anti-glare attention module, a second metal anti-glare attention module, a third metal anti-glare attention module, a feature fusion module, a secondary attention enhancement module, a first detection head module, a second detection head module, a third detection head module, and a result output module.

[0017] The input image is first processed by the backbone network module to obtain three first to third feature maps of high resolution, medium resolution and low resolution of different scales; the three feature maps are input into the first to third metal anti-glare attention modules for processing, and the processing results are input into the feature fusion module; the feature fusion module performs feature fusion on the received feature maps and outputs fourth to sixth feature maps of high resolution, medium resolution and low resolution of different scales; the fourth feature map is input into the secondary attention enhancement module for processing, and the processing result is input into the first detection head module; the fifth feature map is directly input into the second detection head module; the sixth feature map is directly input into the third detection head module; the first to third detection head modules perform mine car bounding box positioning and mechanical arm bounding box positioning according to the input feature maps, and input the positioning results into the result output module; the result output module obtains a mine dump truck positioning map and a mechanical arm positioning map as the output of the model according to the input data.

[0018] The recognition model of the mine car and the mechanical arm is specifically: based on YOLOv8s, replacing the backbone network with ShuffleNetV2, and embedding a metal anti-glare attention module after each of the three feature layers output by the network; the metal anti-glare attention module comprises a channel attention module, a spatial attention module and a glare suppression module connected in series; the metal anti-glare attention module strengthens the metal texture feature channel through the channel attention module, focuses on the mechanical arm contact area through the spatial attention module, and actively reduces the glare interference through the glare suppression module; the feature fusion module adopts an improved FPN+PAN structure, applies a 1x1 convolution kernel to each of the three metal anti-glare attention module output feature layers to be fused, unifies the channels to 256 dimensions (all input feature maps are compressed / expanded to 256 channels through a 1x1 convolution), and finally outputs three fourth to ninth feature maps of different scales through the FPN and PAN double-path transmission mechanisms (in the feature fusion process, the FPN and PAN are retained and the feature flows of the two paths are transmitted at the same time); wherein, the fourth feature map with high resolution is obtained through the FPN pathway fusion; the fifth feature map with high resolution is obtained through the PAN pathway fusion; the sixth feature map with medium resolution is obtained through the FPN pathway fusion; the seventh feature map with medium resolution is obtained through the PAN pathway fusion; the eighth feature map with low resolution is obtained through the FPN pathway fusion; and the ninth feature map with low resolution is obtained through the PAN pathway fusion; a secondary attention enhancement is applied to the highest resolution feature map before the detection head, and finally the precise positioning of the dump truck is realized through a three-scale detection head, and the final result is output through a result output module.

[0019] The channel attention module is specifically: a channel descriptor is generated through global average pooling from a given feature map , wherein is the cth channel of the input channel, and the dimension is (height x width); is the spatial resolution of the feature map, corresponding to the feature size output by the neural network; is a spatial coordinate index, which traverses the entire two-dimensional feature map plane; is the global feature descriptor of the cth channel, which is obtained through spatial dimension compression; a two-layer MLP (multi-layer perceptron) is used to calculate the weight , wherein is a Sigmoid function; is a ReLU activation function; is the first layer of full connection weight matrix, and the dimension is ( is the compression ratio), is the input dimension, ​For output dimensions; The weight matrix of the second fully connected layer ; This is the final generated channel attention weight vector; by learning the contribution of each channel, the representation of key feature channels such as the metal texture of the mining truck is enhanced. The final output is... ,in .

[0020] The spatial attention module specifically includes: input as... ,against Generate by applying max pooling and average pooling along the channel axis fused through convolutional layers: ,in It has a 7×7 convolution kernel; The feature map is generated by max pooling operation on the channel dimension, with dimension 1. ; The feature map is generated by average pooling along the channel dimension, with dimension 1. ;[;] represents a tensor concatenation operation, merging two feature maps along the channel dimension; this focuses the network on the contact area between the robotic arm's end effector and the safety pin, suppressing background interference. The output is... .

[0021] The reflection suppression module specifically comprises: given an input feature map and ,in For the number of channels, and The height and width of the feature map are given respectively. First, the statistical properties of the feature map are calculated. Then, a specular mask is generated by analyzing the statistical properties of the feature map. The specific steps include:

[0022] The channel average feature map is calculated using the following formula: ,in This represents the number of channels in the feature map of the neural network. This represents the average feature intensity at each spatial location. For the first Feature maps of each channel;

[0023] The following formula is used to calculate the channel contrast feature map: ,in The characteristic variance of each spatial location;

[0024] The mean plot and variance plot are concatenated along the channel dimension and then fused using a convolutional layer. The following formula is used for calculation: ,in: This represents a tensor concatenation operation, generating a dimension of... Feature map; for Convolution kernel, outputs single-channel feature map Used to encode specular response;

[0025] The specular response is mapped to the range [0, 1] using the Sigmoid function to generate a specular mask, calculated using the following formula: in This is a highlight mask; values ​​close to 1 represent highlight areas, and values ​​close to 0 represent non-highlight areas; the Sigmoid function. Ensure the mask is smooth and differentiable;

[0026] A specular mask is applied to the original feature map to reduce the feature response of the specular regions, calculated using the following formula: in The feature map output by the spatial attention module. Represents element-wise multiplication. To generate suppression weights, the weights for highlight regions are close to 0, and the features are suppressed; the weights for non-highlight regions are close to 1, and the features are preserved.

[0027] The mine car safety pin identification model described in step S5 includes the following steps: Based on the mechanical structure knowledge of the safety pin (key areas such as thread junctions and positioning hole edges), a region of interest (ROI) is preset, guiding the algorithm to focus on effective areas with rich and stable textures, reducing background interference; within the preset ROI, the FAST algorithm is used to detect candidate corner points, and the feature points with the highest response values ​​are selected through non-maximum suppression, retaining corner points with strong geometric stability.

[0028] A Gaussian difference pyramid was constructed to handle the detection requirements of safety pins of different sizes. Each pyramid layer was generated with a downsampling rate of 1.2, achieving multi-scale feature coverage; the number of pyramid layers... Based on the distance between the camera on the second robotic arm and the target depth Dynamic calculation of the nominal dimensions of the safety pin: ;in For camera sensor pixel size, For the smallest identifiable physical size, For safety factor;

[0029] Calculate the image moments within a 15px radius neighborhood of the feature point to determine the principal orientation angle. The direction correction range is ±15°, to accommodate the deflection angle during the assembly of the safety pin; among which... The main direction angle of the feature point (range 0-360°); The image moment represents the centroid of the gray-level distribution in the neighborhood of a pixel; for The grayscale value at that location; , and It is the order of the moment; the feature description is performed sequentially using the steps of neighborhood selection, Gaussian smoothing, and binary descriptor construction.

[0030] The neighborhood selection covers the entire safety pin and matches the physical size of the safety pin; the Gaussian smoothing suppresses surface reflection noise of the metal; the binary descriptor is set in 256-bit Steered BRIEF mode to support the presence of rotational errors.

[0031] The binary descriptor is constructed using the following formula: ,in These are the rotated pixel coordinate pairs; For binary indexes (ORB uses 256-bit descriptors, i=0~255); This is a grayscale query function; It is a rotation matrix;

[0032] The DBSCAN clustering algorithm is applied to the ORB feature points extracted from the entire image, with adaptive clustering parameters; key parameters are dynamically set, including neighborhood radius. and minimum points The dynamic setting specifically refers to: based on camera intrinsic parameters (…). , Main point , ) and current distance Calculate the minimum physical diameter of the safety pin. The minimum pixel size corresponding to the current viewpoint set up ( (For empirical coefficients); set Statistical lower bound based on the effective texture region of the insurance pin;

[0033] The candidate clusters obtained from DBSCAN are geometrically verified to ensure they conform to the mechanical structure of the safety pin. Based on the fact that the safety pin is a slender cylinder with a typical aspect ratio ≥ 4:1 and the mechanical structure constraints of the safety pin, the covariance matrix of the points within the feature clusters is calculated and principal component analysis is performed. Clusters with an aspect ratio greater than 3:1 and a variance of the second principal component less than 15% are retained.

[0034] The clusters that pass the above clustering and verification are identified as target insurance chips. If multiple candidate clusters exist, the cluster with the largest size and highest confidence level is selected.

[0035] After selecting the target feature cluster, extract its convex hull center point as the target position reference, and calculate the depth distance to the target using the following formula: The moving objective of robotic arm two is to minimize... Value; among which Determined through calibration experiments, Scaling factor This is the offset. This is the distance from the target depth.

[0036] Step S6 specifically includes the following steps:

[0037] S61. After the mine car stops, take a picture of the relative position of the mine car and the robotic arm through the top camera, determine the relative position of the robotic arm as a whole with the mine car, and control the slide rail to move the robotic arm so that the second robotic arm is aligned with the horizontal position of the safety pin based on the recognition result of the mine car and robotic arm recognition model.

[0038] S62. The first robotic arm extends, bringing the electromagnet at its tip into contact with the body of the mine car. When energized, the electromagnet adheres to the mine car, maintaining the relative position of the mine car and the robotic arm as a whole. The second robotic arm takes an image of the safety pin using a camera, determining whether the second robotic arm and the mine car's safety pin are aligned in the vertical direction. The target detection algorithm is used to locate the position of the safety pin, and the second robotic arm is controlled to align with the mine car's safety pin based on the result.

[0039] S63. After both cameras determine the arrival position, the electromagnet is powered on and the pin is located by shaking using a random perturbation search algorithm. It is determined whether the pin is correctly attracted to the pin. If not, the electromagnet is turned off and the robotic arm is fine-tuned by using the random perturbation search algorithm again. If the robotic arm moves to the appropriate position, the second robotic arm extends so that the electromagnet at the top contacts the locking mechanism. After attracting the locking mechanism, the second robotic arm rotates to open the locking mechanism. After the locking mechanism is opened, the second robotic arm maintains the attraction of the electromagnet to the pin and performs synchronous operation with the first robotic arm.

[0040] S64. The first robotic arm continues to extend, pushing the mine car to flip it over and complete the unloading. After unloading, the first robotic arm retracts, returning the mine car to its original position. Then the second robotic arm rotates to close the locking mechanism. Finally, the power is cut off, causing the electromagnet to lose its magnetic force, and both robotic arms return to their initial positions.

[0041] Step S61 specifically involves: using the top camera and the relative position recognition model of the mine car to perform target detection on the mine car, obtaining the coordinates of the mine car's bounding box, calculating the distance between the mine car's position and the standard tipping position, and using the following formula: ;in For the target location ; The top edge of the minecart's bounding box. This represents the bottom edge of the minecart's bounding box. For mining carts in The distance between the axis and the standard unloading position; by synchronously moving robotic arm one and robotic arm two along the track, the distance between robotic arm one and robotic arm two is... Axis coordinates and mine car Axis coordinate alignment, using a robotic arm to attract a tipping mine car via an electromagnet and then minimizing it. , put the mine car in Move to the specified target area on the coordinate system.

[0042] The random perturbation search algorithm uses a sine wave superimposed with random noise to generate the swaying trajectory. Where A is the amplitude, which is less than the tolerance of the safety pin hole; To avoid the resonant frequency of the robotic arm; Zero-mean Gaussian noise enhances robustness; when the contact force at the end of the robotic arm... Exceeding the threshold The shaking stops, and the position of the robotic arm and the pin is determined.

[0043] Furthermore, in practical applications, a protection mechanism that uses multiple sensors to collaboratively trigger power-off is also included, specifically: real-time detection of the motor's operating current. When the rated value is exceeded Protection is triggered when the resistance reaches 120%; abnormal resistance torque is detected by the joint torque sensor. The threshold is set to 150% of the rated torque, forming a dual verification with current monitoring to avoid false triggering.

[0044] The present invention also provides a system for implementing the vision-based automatic unloading method for mine tipper trucks, including an image acquisition module, an image annotation module, a model building and training module, a safety pin recognition model building module, and an automatic unloading module for mine tipper trucks;

[0045] The image acquisition module acquires historical overhead images of the tipper mine truck and the robotic arm, and uploads the data to the image annotation module;

[0046] The image annotation module annotates historical aerial images based on the received data, obtains a training dataset, and uploads the data to the model building and training module;

[0047] The model building and training module builds an initial recognition model of the mining truck and robotic arm based on YOLOv8s according to the received data, and trains it using the training dataset to obtain the recognition model of the mining truck and robotic arm, and uploads the data to the automatic unloading module of the mine dump truck.

[0048] The safety pin recognition model building module is based on the traditional machine vision ORB feature point detection algorithm to build a safety pin recognition model for mining trucks and upload the data to the automatic unloading module of the mine dump truck.

[0049] The automatic unloading module for mine dump trucks uses the received data and the recognition models of the mine truck and the robotic arm, as well as the recognition model of the mine truck's safety pin, to establish a stereo vision positioning system, thereby realizing automatic unloading of the mine dump truck.

[0050] This invention discloses an automatic unloading method and system for mine tipper trucks based on visual positioning. By introducing artificial intelligence recognition and combining it with an adaptive control strategy, the unloading efficiency and safety performance of mine tipper trucks are improved, while reducing labor costs. Attached Figure Description

[0051] Figure 1 This is a schematic flowchart of the method of the present invention;

[0052] Figure 2 This is a schematic diagram of the robotic arm and mine car device in the method of the present invention; wherein, 1 is robotic arm one; 2 is robotic arm two; 3 is a top camera;

[0053] Figure 3 This is a schematic diagram of the structure of the identification model between the mining truck and the robotic arm in this invention;

[0054] Figure 4 This is a schematic diagram illustrating the operation of the reflectivity suppression module in the identification model of the mining truck and robotic arm of the present invention.

[0055] Figure 5 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0056] This invention provides a visual positioning-based automatic unloading method for dump trucks in mines, the process of which is shown in the schematic diagram below. Figure 1 As shown, it includes the following steps:

[0057] S1. Obtain overhead images of the tipper minecart and the robotic arm;

[0058] This invention features two robotic arms and a sliding rail for the robotic arms mounted on the side of a tipper mine car track. The robotic arms are used to position, secure, tip, unload, and right the mine car. A schematic diagram of the robotic arm and mine car assembly is shown below. Figure 2 As shown; two robotic arms are placed on different bases, which are located on the robotic arm slide rails. Each robotic arm is equipped with an electromagnet at its end; robotic arm one is responsible for fixing the mine car and tilting the mine car for unloading; robotic arm two is responsible for opening the locking mechanism; robotic arm one is also responsible for pushing the mine car over and righting the mine car by electromagnet attraction after unloading is completed.

[0059] The robotic arm 2 is responsible for opening the locking mechanism as follows: after moving to the position of the safety pin, the robotic arm 2 will energize the electromagnet to attract the safety pin, and then rotate the safety pin to open the safety. After the mine car returns to the center, it will energize again to attract the safety pin and then rotate it into the safety slot.

[0060] The end of the second robotic arm is equipped with a hand-eye integrated camera to identify the position of the safety pin;

[0061] A top camera is installed at the top of the working area of ​​the tipper mine car to identify the relative position of the mine car and the robotic arm.

[0062] S2. Label the historical aerial images to obtain the first training dataset;

[0063] S3. Construct an initial recognition model for mining trucks and robotic arms based on YOLOv8s; and train it using the first training dataset to obtain the recognition model for mining trucks and robotic arms.

[0064] In step S3, the structural diagram of the initial mining truck and robotic arm recognition model is shown below. Figure 3 As shown, it includes a backbone network, a first metal anti-reflective attention module, a second metal anti-reflective attention module, a third metal anti-reflective attention module, a feature fusion module, a secondary attention enhancement module, a first detection head module, a second detection head module, a third detection head module, and a result output module;

[0065] The input image is first processed by the backbone network module to obtain first to third feature maps at three different scales: high resolution, medium resolution, and low resolution. These three feature maps are then processed by the first to third metal anti-reflective attention modules, and the results are input to the feature fusion module. The feature fusion module fuses the received feature maps and outputs fourth to sixth feature maps at three different scales: high resolution, medium resolution, and low resolution. The fourth feature map is then processed by the secondary attention enhancement module, and the results are input to the first detection head module. The fifth feature map is directly input to the second detection head module, and the sixth feature map is directly input to the third detection head module. The first to third detection head modules perform bounding box localization for the mine car and the robotic arm based on the input feature maps, and the localization results are input to the result output module. The result output module outputs the dump truck localization map and the robotic arm localization map as the model's output based on the input data.

[0066] The specific recognition model for the mining truck and robotic arm is as follows: Based on YOLOv8s, the backbone network is replaced with ShuffleNetV2, and a metal anti-reflective attention module is embedded after each of its three output feature layers. The metal anti-reflective attention module includes a channel attention module, a spatial attention module, and a reflection suppression module connected in series. The metal anti-reflective attention module enhances the metal texture feature channels through the channel attention module, focuses the contact area of ​​the robotic arm through the spatial attention module, and actively reduces high-light interference through the reflection suppression module. The feature fusion module adopts an improved FPN+PAN structure, applying a 1x1 convolution kernel to each feature layer to be fused from the three metal anti-reflective attention modules, unifying the channels to 256 dimensions (compressing / expanding all input feature maps to 256 dimensions through 1×1 convolution). The system integrates multi-scale metal features through a dual-path transfer mechanism of FPN and PAN (simultaneously retaining the feature flows of both FPN and PAN paths during feature fusion) to output three different scale feature maps (fourth to ninth). The high-resolution fourth feature map is obtained through FPN fusion; the high-resolution fifth feature map is obtained through PAN fusion; the medium-resolution sixth feature map is obtained through FPN fusion; the medium-resolution seventh feature map is obtained through PAN fusion; the low-resolution eighth feature map is obtained through FPN fusion; and the low-resolution ninth feature map is obtained through PAN fusion. A secondary attention enhancement is applied to the highest-resolution feature map before the detection head. Finally, the three-scale detection head achieves accurate positioning of the tipper mine car, and the final result is output through the result output module.

[0067] The channel attention module specifically comprises: given a feature map Channel descriptors are generated using global average pooling: ,in Let c be the c-th channel of the input channel, with dimension 1. (Height × Width); This represents the spatial resolution of the feature map, corresponding to the feature size output by the neural network. Use spatial coordinates as indices to traverse the entire two-dimensional feature map plane; The global feature descriptor for the c-th channel is obtained through spatial dimension compression; the weights are calculated using a two-layer MLP (Multilayer Perceptron). ,in For the Sigmoid function; It is the ReLU activation function; The first fully connected layer weight matrix has dimensions of [dimensionality]. ( (Compression ratio) For input dimensions, For output dimensions; The weight matrix of the second fully connected layer ; This is the final generated channel attention weight vector; by learning the contribution of each channel, the representation of key feature channels such as the metal texture of the mining truck is enhanced. The final output is... ,in .

[0068] The spatial attention module specifically includes: input as... ,against Generate by applying max pooling and average pooling along the channel axis fused through convolutional layers: ,in It has a 7×7 convolution kernel; The feature map is generated by max pooling operation on the channel dimension, with dimension 1. ; The feature map is generated by average pooling along the channel dimension, with dimension 1. ;[;] represents a tensor concatenation operation, merging two feature maps along the channel dimension; this focuses the network on the contact area between the robotic arm's end effector and the safety pin, suppressing background interference. The output is... .

[0069] The reflection suppression module specifically comprises: given an input feature map and ,in For the number of channels, and The height and width of the feature map are respectively defined. First, the statistical properties of the feature map are calculated. By analyzing the statistical properties of the feature map, a specular mask is generated. The processing flow diagram is shown below. Figure 4 As shown, the specific steps include:

[0070] The channel average feature map is calculated using the following formula: ,in This represents the number of channels in the feature map of the neural network. This represents the average feature intensity at each spatial location. For the first Feature maps of each channel;

[0071] The following formula is used to calculate the channel contrast feature map: ,in The characteristic variance of each spatial location;

[0072] The mean plot and variance plot are concatenated along the channel dimension and then fused using a convolutional layer. The following formula is used for calculation: ,in: This represents a tensor concatenation operation, generating a dimension of... Feature map; for Convolution kernel, outputs single-channel feature map Used to encode specular response;

[0073] The specular response is mapped to the range [0, 1] using the Sigmoid function to generate a specular mask, calculated using the following formula: in This is a highlight mask; values ​​close to 1 represent highlight areas, and values ​​close to 0 represent non-highlight areas; the Sigmoid function. Ensure the mask is smooth and differentiable;

[0074] A specular mask is applied to the original feature map to reduce the feature response of the specular regions, calculated using the following formula: in The feature map output by the spatial attention module. Represents element-wise multiplication. To generate suppression weights, the weights for highlight regions are close to 0, and the features are suppressed; the weights for non-highlight regions are close to 1, and the features are preserved.

[0075] S4. Construct a mine truck safety pin recognition model based on the traditional machine vision ORB feature point detection algorithm;

[0076] The mine car safety pin identification model described in step S4 includes the following steps: Based on the mechanical structure knowledge of the safety pin (key areas such as thread junctions and positioning hole edges), a region of interest (ROI) is preset, guiding the algorithm to focus on effective areas with rich and stable textures, reducing background interference; within the preset ROI, the FAST algorithm is used to detect candidate corner points, and the feature points with the highest response values ​​are selected through non-maximum suppression, retaining corner points with strong geometric stability;

[0077] A Gaussian difference pyramid was constructed to handle the detection requirements of safety pins of different sizes. Each pyramid layer was generated with a downsampling rate of 1.2, achieving multi-scale feature coverage; the number of pyramid layers... Based on the distance between the camera on the second robotic arm and the target depth Dynamic calculation of the nominal dimensions of the safety pin: ;in For camera sensor pixel size, For the smallest identifiable physical size, For safety factor;

[0078] Calculate the image moments within a 15px radius neighborhood of the feature point to determine the principal orientation angle. The direction correction range is ±15°, to accommodate the deflection angle during the assembly of the safety pin; among which... The main direction angle of the feature point (range 0-360°); The image moment represents the centroid of the gray-level distribution in the neighborhood of a pixel; for The grayscale value at that location; , and It is the order of the moment; the feature description is performed sequentially using the steps of neighborhood selection, Gaussian smoothing, and binary descriptor construction.

[0079] The neighborhood selection covers the entire safety pin and matches the physical size of the safety pin; the Gaussian smoothing suppresses surface reflection noise of the metal; the binary descriptor is set in 256-bit Steered BRIEF mode to support the presence of rotational errors.

[0080] The binary descriptor is constructed using the following formula: ,in These are the rotated pixel coordinate pairs; For binary indexes (ORB uses 256-bit descriptors, i=0~255); This is a grayscale query function; It is a rotation matrix;

[0081] The DBSCAN clustering algorithm is applied to the ORB feature points extracted from the entire image, with adaptive clustering parameters; key parameters are dynamically set, including neighborhood radius. and minimum points The dynamic setting specifically refers to: based on camera intrinsic parameters (…). , Main point , ) and current distance Calculate the minimum physical diameter of the safety pin. The minimum pixel size corresponding to the current viewpoint set up ( (For empirical coefficients); set Statistical lower bound based on the effective texture region of the insurance pin;

[0082] The candidate clusters obtained from DBSCAN are geometrically verified to ensure they conform to the mechanical structure of the safety pin. Based on the fact that the safety pin is a slender cylinder with a typical aspect ratio ≥ 4:1 and the mechanical structure constraints of the safety pin, the covariance matrix of the points within the feature clusters is calculated and principal component analysis is performed. Clusters with an aspect ratio greater than 3:1 and a variance of the second principal component less than 15% are retained.

[0083] The clusters that pass the above clustering and verification are identified as target insurance chips. If multiple candidate clusters exist, the cluster with the largest size and highest confidence level is selected.

[0084] After selecting the target feature cluster, extract its convex hull center point as the target position reference, and calculate the depth distance to the target using the following formula: The moving objective of robotic arm two is to minimize... Value; among which Determined through calibration experiments, Scaling factor This is the offset. This is the distance from the target depth.

[0085] S5. Using the identification models of the mine car and the robotic arm obtained in step S3 and the identification model of the mine car safety pin obtained in step S4, a stereo vision positioning system is established to realize automatic unloading of mine dump trucks.

[0086] Step S5 specifically includes the following steps:

[0087] S51. After the mine car stops, take a picture of the relative position of the mine car and the robotic arm through the top camera, determine the relative position of the robotic arm as a whole with the mine car, and control the slide rail to move the robotic arm so that the second robotic arm is aligned with the horizontal position of the safety pin based on the recognition result of the mine car and robotic arm recognition model.

[0088] S52. The first robotic arm extends, bringing the electromagnet at its tip into contact with the body of the mine car. When energized, the electromagnet adheres to the mine car, maintaining the relative position of the mine car and the robotic arm as a whole. The second robotic arm takes an image of the safety pin using a camera, determining whether the second robotic arm and the mine car's safety pin are aligned in the vertical direction. The target detection algorithm is used to locate the position of the safety pin, and the second robotic arm is controlled to align with the mine car's safety pin based on the result.

[0089] S53. After both cameras determine the arrival position, the electromagnet is powered on and the pin is located by shaking using a random perturbation search algorithm. It is determined whether the pin is correctly attracted to the pin. If not, the electromagnet is turned off and the robotic arm is fine-tuned by using the random perturbation search algorithm again. If the robotic arm moves to the appropriate position, the second robotic arm extends so that the electromagnet at the top contacts the locking mechanism. After attracting the locking mechanism, the second robotic arm rotates to open the locking mechanism. After the locking mechanism is opened, the second robotic arm maintains the attraction of the electromagnet to the pin and performs synchronous operation with the first robotic arm.

[0090] S54. The first robotic arm continues to extend, pushing the mine car to flip it over and complete the unloading. After unloading, the first robotic arm retracts, returning the mine car to its original position. Then the second robotic arm rotates to close the locking mechanism. Finally, the power is cut off, causing the electromagnet to lose its magnetic force, and both robotic arms return to their initial positions.

[0091] Step S51 specifically involves: using the top camera and the relative position recognition model of the mine car to perform target detection on the mine car, obtaining the coordinates of the mine car's bounding box, calculating the distance between the mine car's position and the standard tipping position, and using the following formula: ;in For the target location ; The top edge of the minecart's bounding box. This represents the bottom edge of the minecart's bounding box. For mining carts in The distance between the axis and the standard unloading position; by synchronously moving robotic arm one and robotic arm two along the track, the distance between robotic arm one and robotic arm two is... Axis coordinates and mine car Axis coordinate alignment, using a robotic arm to attract a tipping mine car via an electromagnet and then minimizing it. , put the mine car in Move to the specified target area on the coordinate system.

[0092] The random perturbation search algorithm uses a sine wave superimposed with random noise to generate the swaying trajectory. Where A is the amplitude, which is less than the tolerance of the safety pin hole; To avoid the resonant frequency of the robotic arm; Zero-mean Gaussian noise enhances robustness; when the contact force at the end of the robotic arm... Exceeding the threshold The shaking stops, and the position of the robotic arm and the pin is determined.

[0093] Furthermore, in practical applications, a protection mechanism that uses multiple sensors to collaboratively trigger power-off is also included, specifically: real-time detection of the motor's operating current. When the rated value is exceeded Protection is triggered when the resistance reaches 120%; abnormal resistance torque is detected by the joint torque sensor. The threshold is set to 150% of the rated torque, forming a dual verification with current monitoring to avoid false triggering.

[0094] The present invention also provides a system for implementing the aforementioned vision-based automatic unloading method for dump trucks in mines, the structural schematic diagram of which is shown below. Figure 5 As shown, it includes an image acquisition module, an image annotation module, a model building and training module, a safety pin recognition model building module, and an automatic unloading module for mine dump trucks;

[0095] The image acquisition module acquires historical overhead images of the tipper mine truck and the robotic arm, and uploads the data to the image annotation module;

[0096] The image annotation module annotates historical aerial images based on the received data, obtains a training dataset, and uploads the data to the model building and training module;

[0097] The model building and training module builds an initial recognition model of the mining truck and robotic arm based on YOLOv8s according to the received data, and trains it using the training dataset to obtain the recognition model of the mining truck and robotic arm, and uploads the data to the automatic unloading module of the mine dump truck.

[0098] The safety pin recognition model building module is based on the traditional machine vision ORB feature point detection algorithm to build a safety pin recognition model for mining trucks and upload the data to the automatic unloading module of the mine dump truck.

[0099] The automatic unloading module for mine dump trucks uses the received data and the recognition models of the mine truck and the robotic arm, as well as the recognition model of the mine truck's safety pin, to establish a stereo vision positioning system, thereby realizing automatic unloading of the mine dump truck.

Claims

1. A method for automatic unloading of dump trucks in mines based on vision positioning, characterized in that, Includes the following steps: S1. Obtain historical overhead images of the tipper mine car and robotic arm; S2. Label the historical aerial images to obtain the first training dataset; S3. Construct an initial recognition model for mining trucks and robotic arms based on YOLOv8s; and train it using the first training dataset to obtain the recognition model for mining trucks and robotic arms. The specific identification model for the mining truck and robotic arm is as follows: based on YOLOv8s, the backbone network is replaced with ShuffleNetV2, and metal anti-reflective attention modules are embedded after the three feature layers of its output; an improved FPN+PAN structure is adopted, and multi-scale metal features are fused through the dual-path transmission mechanism of FPN and PAN respectively. S4. Construct a mine truck safety pin recognition model based on the traditional machine vision ORB feature point detection algorithm; S5. Using the identification models of the mine car and the robotic arm obtained in step S3 and the identification model of the mine car safety pin obtained in step S4, a stereo vision positioning system is established to realize automatic unloading of mine dump trucks.

2. The automatic unloading method for mine tipper cars based on vision positioning according to claim 1, characterized in that, Two robotic arms and a slide rail are installed on the side of the tipping mine car track. The robotic arms are used to position, fix, tip over for unloading, and straighten the mine car. The two robotic arms are placed on different bases, which are located on the slide rails for the robotic arms. Each robotic arm is equipped with an electromagnet at its end. Robotic arm one is responsible for fixing the mine car and tipping it over for unloading. Robotic arm two is responsible for opening the locking mechanism. Robotic arm one is also responsible for pushing the mine car over and straightening it by attracting it with the electromagnet after unloading. The robotic arm 2 is responsible for opening the locking mechanism as follows: after moving to the position of the safety pin, the robotic arm 2 will energize the electromagnet to attract the safety pin, and then rotate the safety pin to open the safety. After the mine car returns to the center, it will energize again to attract the safety pin and then rotate it into the safety slot. The end of the second robotic arm is equipped with a hand-eye integrated camera to identify the position of the safety pin; A top camera is installed at the top of the working area of ​​the tipper mine car to identify the relative position of the mine car and the robotic arm.

3. The automatic unloading method for mine tipper cars based on vision positioning according to claim 2, characterized in that, In step S3, the initial identification model of the mining car and the robotic arm includes a backbone network, a first metal anti-reflective attention module, a second metal anti-reflective attention module, a third metal anti-reflective attention module, a feature fusion module, a secondary attention enhancement module, a first detection head module, a second detection head module, a third detection head module, and a result output module. The input image is first processed by the backbone network module to obtain first to third feature maps of three different scales: high resolution, medium resolution, and low resolution. The three feature maps are then input into the first to third metal anti-reflective attention modules for processing, and the processing results are input into the feature fusion module. The feature fusion module fuses the received feature maps and outputs three feature maps of different scales: high-resolution fourth and fifth feature maps, medium-resolution sixth and seventh feature maps, and low-resolution eighth and ninth feature maps. The fourth and fifth feature maps are input into the secondary attention enhancement module for processing, and the processing result is input into the first detection head module. The sixth and seventh feature maps are directly input into the second detection head module; The eighth and ninth feature maps are directly input into the third detection head module; the first to third detection head modules perform mine car bounding box localization and robotic arm bounding box localization respectively based on the input feature maps, and input the localization results into the result output module; the result output module obtains the dump truck localization map and robotic arm localization map as the model output based on the input data; The specific recognition model for the mining truck and robotic arm is as follows: Based on YOLOv8s, the backbone network is replaced with ShuffleNetV2, and a metal anti-reflective attention module is embedded after the three feature layers of its output. The metal anti-reflective attention module includes a channel attention module, a spatial attention module, and a reflection suppression module connected in series. The metal anti-reflective attention module enhances the metal texture feature channels through the channel attention module, focuses the contact area of ​​the robotic arm through the spatial attention module, and actively reduces specular interference through the reflection suppression module. The feature fusion module adopts an improved FPN+PAN structure. For each feature layer to be fused output by the three metal anti-reflective attention modules, a 1x1 convolution kernel is applied to unify the channels to 256 dimensions. Multi-scale metal features are fused through the dual-path transfer mechanism of FPN and PAN. The final output consists of three feature maps of different scales, the fourth to the ninth. The high-resolution fourth feature map is obtained by fusion through the FPN path, and the high-resolution fifth feature map is obtained by fusion through the PAN path. The sixth feature map of medium resolution is obtained by fusion using the FPN approach; the seventh feature map of medium resolution is obtained by fusion using the PAN approach; the eighth feature map of low resolution is obtained by fusion using the FPN approach; the ninth feature map of low resolution is obtained by fusion using the PAN approach; secondary attention enhancement is applied to the high-resolution feature map in front of the detection head; finally, the three-scale detection head is used to achieve accurate positioning of the tipper mine car, and the final result is output through the result output module.

4. The automatic unloading method for mine tipper cars based on vision positioning according to claim 3, characterized in that, The channel attention module specifically comprises: given a feature map Channel descriptors are generated using global average pooling: ,in Let c be the c-th channel of the input channel, with dimension 1. ; This represents the spatial resolution of the feature map, corresponding to the feature size output by the neural network. Use spatial coordinates as indices to traverse the entire two-dimensional feature map plane; The global feature descriptor for the c-th channel is obtained through spatial dimension compression; weights are calculated using a two-layer MLP: ,in For the Sigmoid function; It is the ReLU activation function; The first fully connected layer weight matrix has dimensions of [dimensionality]. , The compression ratio is... For input dimensions, For output dimensions; The weight matrix of the second fully connected layer ; This is the final generated channel attention weight vector; by learning the contribution of each channel, the expression of key feature channels of the mining truck's metal texture is strengthened; the final output is... ,in .

5. The automatic unloading method for mine tipper cars based on visual positioning according to claim 3, characterized in that, The spatial attention module specifically includes: input as... ,against Generate by applying max pooling and average pooling along the channel axis fused through convolutional layers: ,in It has a 7×7 convolution kernel; The feature map is generated by max pooling operation on the channel dimension, with dimension 1. ; The feature map is generated by average pooling along the channel dimension, with dimension 1. ; [;] represents a tensor concatenation operation, merging two feature maps along the channel dimension; the output of the spatial attention module is... .

6. The automatic unloading method for mine tipper cars based on vision positioning according to claim 3, characterized in that, The reflection suppression module specifically comprises: given an input feature map and ,in For the number of channels, and The height and width of the feature map are given respectively. First, the statistical properties of the feature map are calculated. Then, a specular mask is generated by analyzing the statistical properties of the feature map. The specific steps include: The channel average feature map is calculated using the following formula: ,in This represents the number of channels in the feature map of the neural network. This represents the average feature intensity at each spatial location. For the first Feature maps of each channel; The following formula is used to calculate the channel contrast feature map: ,in The characteristic variance of each spatial location; The mean plot and variance plot are concatenated along the channel dimension and then fused using a convolutional layer. The following formula is used for calculation: ,in: This represents a tensor concatenation operation, generating a dimension of... Feature map; for Convolution kernel, outputs single-channel feature map Used to encode specular response; The specular response is mapped to the range [0, 1] using the Sigmoid function to generate a specular mask, calculated using the following formula: in This is a highlight mask; values ​​close to 1 represent highlight areas, and values ​​close to 0 represent non-highlight areas; the Sigmoid function. Ensure the mask is smooth and differentiable; A specular mask is applied to the original feature map to reduce the feature response of the specular regions, calculated using the following formula: in The feature map output by the spatial attention module. Represents element-wise multiplication. To generate suppression weights, the weights for highlight regions are close to 0, and the features are suppressed; the weights for non-highlight regions are close to 1, and the features are preserved.

7. The automatic unloading method for mine tipper cars based on visual positioning according to claim 2, wherein the mine car safety pin recognition model in step S4 includes the following steps: Based on the mechanical structure knowledge of the safety pin, the region of interest is preset to guide the algorithm to focus on the effective region with rich and stable texture, thereby reducing background interference; Within the preset ROI, the FAST algorithm is used to detect candidate corner points, and non-maximum suppression is used to filter the feature points with the highest response values, retaining corner points with strong geometric stability. A Gaussian difference pyramid was constructed to handle the detection requirements of safety pins of different sizes. Each pyramid layer was generated with a downsampling rate of 1.2, achieving multi-scale feature coverage; the number of pyramid layers... Based on the distance between the camera on the second robotic arm and the target depth Dynamic calculation of the nominal dimensions of the safety pin: ;in For camera sensor pixel size, For the smallest identifiable physical size, For safety factor; Calculate the image moments within a 15px radius neighborhood of the feature point to determine the principal orientation angle. The direction correction range is ±15°, to accommodate the deflection angle during the assembly of the safety pin; among which... The principal direction angle of the feature point; The image moment represents the centroid of the gray-level distribution in the neighborhood of a pixel; for The grayscale value at that location; , and It is the order of the moment; the feature description is performed sequentially using the steps of neighborhood selection, Gaussian smoothing, and binary descriptor construction. The neighborhood selection covers the entire safety pin and matches the physical size of the safety pin; the Gaussian smoothing suppresses surface reflection noise of the metal; the binary descriptor is set in 256-bit Steered BRIEF mode to support the presence of rotational errors. The binary descriptor is constructed using the following formula: ,in These are the rotated pixel coordinate pairs; Binary index; This is a grayscale query function; It is a rotation matrix; The DBSCAN clustering algorithm is applied to the ORB feature points extracted from the whole image, and the clustering parameters are adapted. Key parameters are dynamically set; these key parameters include the radius of the domain. and minimum points The dynamic setting specifically refers to: based on camera intrinsic parameters (…). , Main point , ) and current distance Calculate the minimum physical diameter of the safety pin. The minimum pixel size corresponding to the current viewpoint set up , For empirical coefficients; set Statistical lower bound based on the effective texture region of the insurance pin; The candidate clusters obtained from DBSCAN are geometrically verified to ensure they conform to the mechanical structure of the safety pin. Based on the fact that the safety pin is a slender cylinder with a typical length-to-diameter ratio of ≥4:1, and considering the mechanical structural constraints of the safety pin, the covariance matrix of the points within the feature clusters is calculated and principal component analysis is performed. Clusters with a length-to-width ratio greater than 3:1 and a second principal component variance ratio of less than 15% are retained. Clusters that pass the above clustering and verification are identified as target insurance chips; if multiple candidate clusters exist, the cluster with the size that best matches the expectation and the highest confidence level is selected. After selecting the target feature cluster, extract its convex hull center point as the target position reference, and calculate the depth distance to the target using the following formula: The movement objective of robotic arm two is to minimize Value; among which Determined through calibration experiments, Scaling factor This is the offset. This is the distance from the target depth.

8. The automatic unloading method for mine tipper cars based on vision positioning according to claim 2, step S5 specifically includes the following steps: S51. After the mine car stops, take a picture of the relative position of the mine car and the robotic arm through the top camera, determine the relative position of the robotic arm as a whole with the mine car, and control the slide rail to move the robotic arm so that the second robotic arm is aligned with the horizontal position of the safety pin based on the recognition result of the mine car and robotic arm recognition model. S52. The first robotic arm extends, bringing the electromagnet at its tip into contact with the body of the mine car. When energized, the electromagnet adheres to the mine car, maintaining the relative position of the mine car and the robotic arm as a whole. The second robotic arm takes an image of the safety pin using a camera, determining whether the second robotic arm and the mine car's safety pin are aligned in the vertical direction. The target detection algorithm is used to locate the position of the safety pin, and the second robotic arm is controlled to align with the mine car's safety pin based on the result. S53. After both cameras determine the arrival position, the electromagnet is powered on and the pin is located by shaking using a random perturbation search algorithm. It is determined whether the pin is correctly attracted to the pin. If not, the electromagnet is turned off and the robotic arm is fine-tuned by using the random perturbation search algorithm again. If the robotic arm moves to the appropriate position, the second robotic arm extends so that the electromagnet at the top contacts the locking mechanism. After attracting the locking mechanism, the second robotic arm rotates to open the locking mechanism. After the locking mechanism is opened, the second robotic arm maintains the attraction of the electromagnet to the pin and performs synchronous operation with the first robotic arm. S54. The first robotic arm continues to extend, pushing the mine car to flip it over and complete the unloading. After unloading, the first robotic arm retracts, returning the mine car to its original position. Then the second robotic arm rotates to close the locking mechanism. Finally, the power is cut off, causing the electromagnet to lose its magnetic force, and both robotic arms return to their initial positions. Step S51 specifically involves: using the top camera and the relative position recognition model of the mine car to perform target detection on the mine car, obtaining the coordinates of the mine car's bounding box, calculating the distance between the mine car's position and the standard tipping position, and using the following formula: ;in For the target location ; The top edge of the minecart's bounding box. This represents the bottom edge of the minecart's bounding box. For the mine car in The distance between the axis and the standard unloading position; by synchronously moving robotic arm one and robotic arm two along the track, the distance between robotic arm one and robotic arm two is... Axis coordinates and mine car Axis coordinate alignment, using a robotic arm to attract a tipping mine car via an electromagnet and then minimizing it. , put the mine cart in Move along the coordinates to the specified target area; The random perturbation search algorithm uses a sine wave superimposed with random noise to generate the swaying trajectory. Where A is the amplitude, which is less than the tolerance of the safety pin hole; To avoid the resonant frequency of the robotic arm; Zero-mean Gaussian noise enhances robustness; when the contact force at the end of the robotic arm... Exceeding the threshold The shaking stops, and the position of the robotic arm and the pin is determined.

9. The automatic unloading method for mine tipper cars based on vision positioning according to claim 8, characterized in that, Step S5 also includes a protection mechanism that uses multiple sensors to collaboratively trigger power-off, specifically: real-time detection of the motor's operating current. When the rated value is exceeded Protection is triggered when the value reaches 120%. Abnormal resistance torque is detected by a joint torque sensor. The threshold is set to 150% of the rated torque, forming a dual verification with current monitoring to avoid false triggering.

10. A system for implementing the vision-based positioning-based automatic unloading method for mine tipper cars according to any one of claims 1 to 9, characterized in that, It includes an image acquisition module, an image annotation module, a model building and training module, a safety pin recognition model building module, and an automatic unloading module for mine dump trucks; The image acquisition module acquires historical overhead images of the tipper mine truck and the robotic arm, and uploads the data to the image annotation module; The image annotation module annotates historical aerial images based on the received data, obtains a training dataset, and uploads the data to the model building and training module; The model building and training module builds an initial recognition model of the mining truck and robotic arm based on YOLOv8s according to the received data, and trains it using the training dataset to obtain the recognition model of the mining truck and robotic arm, and uploads the data to the automatic unloading module of the mine dump truck. The safety pin recognition model building module is based on the traditional machine vision ORB feature point detection algorithm to build a safety pin recognition model for mining trucks and upload the data to the automatic unloading module of the mine dump truck. The automatic unloading module for mine dump trucks uses the received data and the recognition models of the mine truck and the robotic arm, as well as the recognition model of the mine truck's safety pin, to establish a stereo vision positioning system, thereby realizing automatic unloading of the mine dump truck.

Images