Method for identifying continuous shoveling position of autonomous operation loader based on deep learning

Abstract

The invention provides an autonomous operation loader continuous shoveling position identification method based on deep learning, and relates to the technical field of image processing, and the method comprises the steps: obtaining laser point cloud data of the surface of a material pile, and processing the laser point cloud data to obtain a plurality of corresponding material pile shoveling area features; constructing an executive device model of the loader, and performing simulation stress analysis on the executive device model and the material pile shoveling area features to obtain corresponding material pile shoveling position features; the material pile shoveling position features are analyzed and evaluated, and the optimal material pile shoveling position features are obtained; constructing a detection model, acquiring a to-be-identified material pile image, inputting the to-be-identified material pile image into the detection model for processing, and detecting and adjusting the shoveling position of the loader according to an image processing result and the optimal material pile shoveling position feature; according to the method, the model parameter quantity and the calculation quantity are reduced while the detection precision is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to a method for identifying continuous shoveling positions of an autonomous loader based on deep learning. Background Art

[0002] At present, loaders are widely used in construction sites, mining sites and other places, and are important tools in the field of modern construction and engineering. At present, most loaders still rely on manual operation for shoveling operations. There are many problems in the entire shoveling process of the loader: 1) The loader operating environment is relatively complex, which is easy to pose a certain threat to the personal safety of the driver; 2) The shoveling process depends entirely on the driver's experience to judge, and there is no unified standard, which will have a certain impact on the efficiency and quality of the project to a large extent. In order to improve the operating efficiency of the loader, the autonomous operation of the loader has become an important research topic. As the first step in the shoveling process, the judgment of the shoveling position has an extremely important influence. The correct shoveling position can ensure that the loader effectively collects and fills materials. In addition, it can also ensure the safety of the loading operation and avoid accidents such as collapse of the shoveling or excessive resistance to shoveling, which causes the rear wheels to leave the ground.

[0003] However, the existing technology for determining the loading position has the following main shortcomings: (1) Most loading position selection methods only consider the first loading position study on the regular material pile, ignoring the problems of irregular height of the material pile surface and uneven ground contact edge during continuous operation; (2) There are various reasons why the loading position affects the loading efficiency of the loader, which requires comprehensive consideration of multiple factors; (3) Dust is inevitable in the actual working environment, and the laser radar cannot effectively penetrate, resulting in information loss.

[0004] Therefore, how to provide a method for identifying the continuous shoveling position of an autonomous loader that can solve the above problems is an issue that those skilled in the art urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a continuous shoveling position recognition method for an autonomous loader based on deep learning, which ensures detection accuracy while reducing the amount of model parameters and calculation.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for identifying continuous shoveling positions of an autonomous loader based on deep learning, comprising:

[0008] Acquire laser point cloud data of the material pile surface, and process the laser point cloud data to obtain corresponding features of multiple material pile shoveling areas;

[0009] Constructing an execution device model of the loader, and performing a simulated force analysis on the execution device model and the material pile shoveling area characteristics to obtain corresponding material pile shoveling position characteristics;

[0010] Analyzing and evaluating the material pile shoveling position characteristics to obtain the optimal material pile shoveling position characteristics;

[0011] A detection model is constructed, and an image of a material pile to be identified is obtained at the same time. The image of the material pile to be identified is input into the detection model for processing, and the shoveling position of the loader is detected and adjusted according to the image processing result and the characteristics of the optimal material pile shoveling position.

[0012] Preferably, the specific process of obtaining the corresponding multiple stockpile shoveling area features includes:

[0013] Acquire laser point cloud data of the surface of the stockpile, and preprocess the laser point cloud data;

[0014] The pre-processed laser point cloud data is segmented to obtain convex area features and concave area features of the pile, and the convex area features and concave area features are the shoveling area features of the pile.

[0015] Preferably, the specific process of obtaining the optimal pile shoveling position characteristics includes:

[0016] Inputting the actuator model into simulation software;

[0017] Simulating the actuator model, the convex region features, and the concave region features while analyzing the force of the material on the side of the actuator model;

[0018] Compare the forces on the side of the actuator model and select the one with the smallest force as the best pile shoveling position feature, that is, the convex area feature;

[0019] The convex area characteristics are analyzed to obtain the optimal stockpile shoveling position characteristics.

[0020] Preferably, the specific process of analyzing the convex area characteristics to obtain the optimal pile shoveling position characteristics includes:

[0021] Acquire multiple edge convex regions included in the convex region feature;

[0022] Calculate the edge curvature of each edge convex area in the simulation software, and calculate the maximum insertion resistance, maximum lifting resistance and maximum filling mass of each edge convex area;

[0023] The maximum insertion resistance, maximum lifting resistance, and maximum filling mass were normalized;

[0024] The spider graph evaluation method is used to calculate the degree value formed by each edge convex area on each axis of the spider graph;

[0025] By comprehensively comparing multiple degree values, the optimal stockpile shoveling position characteristics are obtained.

[0026] Preferably, the specific process of building the detection model includes:

[0027] A detection model is constructed, wherein the detection model is a Yolov5s model, and the Yolov5s model is improved.

[0028] Preferably, the specific process of improving the Yolov5s model includes:

[0029] The backbone network of the Yolov5s model is replaced with the improved GhostNet network, the RFB module is introduced into the GhostNet network, the CBAM module is added, and EIoU is selected as the loss function.

[0030] Preferably, the specific process of building the detection model also includes:

[0031] Construct a training dataset of piled material images and use it to train the detection model.

[0032] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for identifying continuous shoveling positions of an autonomous loader based on deep learning, which is mainly divided into three parts:

[0033] (1) Simulation analysis of the shovel loading position characteristics: Analyze the influence of concave and convex features on the position selection on the two-dimensional features, establish relevant mathematical models and discrete element simulation, analyze the influence of concave and convex features on the shovel loading position selection, and optimize the shovel loading position;

[0034] (2) After the characteristic shape of the shoveling position is preliminarily determined, how to distinguish different characteristic areas and determine the shoveling position is mainly established through simulation to establish the impact of different degrees of edge curvature and size of the pile on shoveling. A comprehensive evaluation method is used to reflect the pros and cons of the shoveling conditions at different positions, which helps to determine the best shoveling position, thereby improving shoveling efficiency and safety;

[0035] (3) After completing the analysis and identification of the shoveling position characteristics, in order to facilitate the deployment of the network model on mobile devices, a series of lightweight improvement measures are used to ensure the detection accuracy while reducing the model parameters and calculation amount, thereby reducing the operation cost and promoting the intelligent development of autonomous loader shoveling operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 An overall flow chart of a method for identifying continuous shoveling positions of an autonomous loader based on deep learning provided by the present invention;

[0038] Figure 2 A schematic diagram of a point cloud acquisition simulation of a shovel excavation position provided in an embodiment of the present invention, wherein 1-support, 2-area array solid-state laser radar, 3-stock pile;

[0039] Figure 3 A schematic diagram of a point cloud segmentation result provided in an embodiment of the present invention, wherein (a) is a schematic diagram of a 3D modeling of original data, and (b) is a schematic diagram of a point cloud segmentation result;

[0040] Figure 4 A schematic diagram of a simulation of a convex portion of shoveling crushed stone provided in an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a simulated recess for shoveling crushed stones provided by an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the axial lateral force acting on the bucket at the concave and convex part of the crushed stone provided by an embodiment of the present invention;

[0043] Figure 7 A diagram of the accumulation of gravel after shoveling provided by an embodiment of the present invention;

[0044] Figure 8 A schematic diagram of simulation of gravel pile generation for different buckets provided by an embodiment of the present invention;

[0045] FIG9( a) is a schematic diagram of a Roberts operator template provided in an embodiment of the present invention;

[0046] FIG9( b ) is a schematic diagram of a Roberts operator provided in an embodiment of the present invention;

[0047] Fig.10 A schematic diagram for comparing three edge curvatures of 3375 kg of material provided by an embodiment of the present invention;

[0048] Fig.11 A schematic diagram for comparing three edge curvatures of 5063 kg of material provided in an embodiment of the present invention;

[0049] Fig.12A schematic diagram for comparing three edge curvatures of 6750kg of material provided in an embodiment of the present invention;

[0050] Fig.13 A spider graph evaluation comparison effect diagram provided by an embodiment of the present invention;

[0051] Fig.14 A schematic diagram of an improved Yolov5s network model provided in an embodiment of the present invention;

[0052] Fig.15 A schematic diagram of an RFB module provided in an embodiment of the present invention;

[0053] Fig.16 A schematic diagram of a CBAM provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 As shown, an embodiment of the present invention discloses a method for identifying continuous shoveling positions of an autonomous loader based on deep learning, comprising:

[0056] Acquire laser point cloud data of the material pile surface, and process the laser point cloud data to obtain corresponding features of multiple material pile shoveling areas;

[0057] The loader's actuator model was constructed, and the actuator model and the pile shoveling area characteristics were simulated and analyzed to obtain the corresponding pile shoveling position characteristics. The loader's actuator model was based on a 1.2T loader, and the pile material simulation model was based on crushed stone.

[0058] Analyze and evaluate the characteristics of the material pile shoveling position to obtain the optimal material pile shoveling position characteristics;

[0059] A detection model is constructed, and images of the material pile to be identified are obtained at the same time. The images of the material pile to be identified are input into the detection model for processing. The loader's shoveling position is detected and adjusted according to the image processing results and the characteristics of the optimal material pile shoveling position.

[0060] In a specific embodiment, see Figure 2-3 As shown, the specific process of obtaining the corresponding multiple material pile shoveling area features includes:

[0061] Obtain laser point cloud data of the stockpile surface and pre-process the laser point cloud data;

[0062] The pre-processed laser point cloud data is segmented to obtain the convex area features and concave area features of the pile, and the convex area features and concave area features are the shoveling area features of the pile.

[0063] See also Figure 4-5 As shown, in a specific embodiment, the specific process of obtaining the optimal pile shoveling position feature includes:

[0064] Inputting the actuator model into simulation software, wherein the simulation software may be EDEM;

[0065] Simulating the actuator model, the convex region features, and the concave region features while analyzing the force of the material on the side of the actuator model;

[0066] Compare the forces on the side of the actuator model and select the one with the smallest force as the best pile shoveling position feature, that is, the convex area feature;

[0067] The convex area characteristics are analyzed to obtain the optimal stockpile shoveling position characteristics.

[0068] Specifically, the forces acting on the sides of the bucket were extracted by post-processing in EDEM. The specific results can be found in Figure 6 As shown, 0 to 3 seconds is the shoveling stage, 3 to 6 seconds is the lifting stage, and 6 seconds later is the bucket retreat.

[0069] from Figure 6 It can be concluded that: during the insertion stage of shoveling materials, the convex position has a much lower lateral force on the bucket's X-axis than the concave position. When the loader is performing shoveling operations, especially when facing uneven material surfaces, there may be phenomena such as increased resistance, tire slippage, and reduced engine speed. According to the "Loader Safety Technical Operation Regulations", shoveling operations should be stopped when encountering these situations, because forced operations may cause the loader to lose balance and tip over. Therefore, the convex area feature of shoveling can effectively prevent the bucket from being subjected to excessive force on both sides during shoveling, and can be used as a characteristic shape to determine the shoveling position, thereby ensuring the safety of the shoveling process, and also laying the foundation for the subsequent evaluation of different convex feature positions in the material pile.

[0070] In a specific embodiment, the specific process of analyzing the convex area features to obtain the optimal pile shoveling position features includes:

[0071] Acquire multiple edge convex regions included in the convex region feature;

[0072] Calculate the edge curvature of each edge convex area in the simulation software, and calculate the maximum insertion resistance, maximum lifting resistance and maximum filling mass of each edge convex area;

[0073] The maximum insertion resistance, maximum lifting resistance, and maximum filling mass were normalized;

[0074] The spider graph evaluation method is used to calculate the degree value formed by each edge convex area on each axis of the spider graph;

[0075] By comprehensively comparing multiple degree values, the optimal stockpile shoveling position characteristics are obtained.

[0076] Specifically, after initially determining the characteristic shape of the shoveling position, it is found that the area after the bucket is shoveled presents a shape that is concave in the middle and convex on both sides, such as Figure 7 As shown. Based on this situation, it can be judged that the next shoveling location should be selected in the raised area, that is, Figure 7 The three different colored boxes represent the areas. Therefore, distinguishing different convex areas is crucial for the selection of shovel loading positions and is an important link in ensuring energy saving and high efficiency of loaders in production. Since the embodiment of the present invention adopts an image recognition method, the image features of different convex areas are used as the basis for identification. Figure 8 It can be seen that the curvature of the edges of convex regions varies. The specific process of calculating the edge curvature is as follows:

[0077] 1. Establish a cross-sectional area of ​​2.54m 2 The cylindrical barrel and rectangular barrel are filled with 3375kg, 5063kg and 6750kg of crushed stone particles respectively. The particle generation process is as follows Figure 8 As shown. By lifting the barrel column, the pile with different edges can be obtained;

[0078] 2. In Matlab, binarize the stacked image to get the edge shape, and then use the Roberts edge detection method to extract the pixel coordinate points of the edge contour curve;

[0079] The Roberts operator has a good effect on the rough edge and low noise of the pile. It mainly estimates the possible edges in the image by calculating the difference between adjacent pixels in the diagonal direction. For example, Figure 9 (a) and (b) show the template of the operator. The gradient size g in the X and Y directions at the pixel point P2 is x and g y The specific calculation formula is:

[0080]

[0081] 3. Fit the curve and calculate the average curvature value, the formula is shown below (4). Assume that the equation of the curve is shown in (3), and use the parametric equation derivation method to obtain the curvature matrix K divided by the number of points to get the average curvature H; in EDEM, three different masses of crushed stone particles are placed in three different material drop tubes to generate a total of 9 different curvatures of the pile edge, and the edge curvature is extracted as follows: Figure 10-12 shown.

[0082]

[0083] The maximum insertion resistance, maximum lifting resistance and maximum filling mass values ​​of 9 piles with different curvature edges were directly obtained by coupling simulation with Adams in EDEM, and used as three indicators to judge the quality of different convex area positions;

[0084] 4. Then, the maximum insertion resistance, maximum lifting resistance and maximum filling mass values ​​of the 9 different curvature edge piles were directly obtained in EDEM through coupling simulation with Adams, and used as three indicators to judge the quality of different convex area positions. Since there are different measurement units and scales between the three indicators, which affects the results of modeling and analysis, it is necessary to normalize the data so that the data is mapped to the range of (0-1) to make the different features comparable. The formula for the maximum value normalization is shown in formula (5):

[0085]

[0086] According to the loader excavation process, the maximum pressure in the bucket insertion stage and lifting stage, the maximum pressure in each direction, and the mass of the material excavated by the bucket are selected as three indicators to judge the quality of different excavation positions in the post-processing of EDEM. Then, the spider diagram evaluation method is used to calculate the corresponding degree value. Finally, the degree of quality is reflected by comparing the size of the degree value. The effect is as follows: Fig.13 shown.

[0087] In a specific embodiment, the specific process of building a detection model includes:

[0088] A detection model is constructed, wherein the detection model is the Yolov5s model, and the Yolov5s model is improved.

[0089] See also Fig.14 As shown, in a specific embodiment, the specific process of improving the Yolov5s model includes:

[0090] The backbone network of the Yolov5s model is replaced with the improved GhostNet network, the RFB module is introduced into the GhostNet network, the CBAM module is added, and EIoU is selected as the loss function.

[0091] Specifically, the specific process of improving the Yolov5s model is as follows:

[0092] (1) The backbone network is replaced with the GhostNet network

[0093] GhostNet was originally designed to improve computational efficiency. Compared with traditional convolutional networks, GhostNet effectively reduces the amount of computation by introducing the Ghost module, which means that on mobile devices or embedded systems, the improved Yolov5s model runs more smoothly than Yolov5s, which helps to achieve real-time target detection.

[0094] In addition, GhostNet also shows superiority in terms of parameter quantity, significantly reducing the number of parameters of the model. This feature not only reduces memory usage, but also greatly reduces the computing resources required for network reasoning, making the overall model more lightweight, which is of great significance for devices with limited resources, especially mobile devices such as autonomous loaders. At the same time, GhostNet's efficient feature extraction capability makes Yolov5s expected to improve its reasoning speed.

[0095] (2) Introducing the RFB module into the backbone GhostNet network

[0096] The RFB module enhances the network's multi-scale feature extraction capabilities by using convolution operations of different scales. Traditional convolutional layers have certain limitations in terms of receptive field, while the RFB module effectively captures contextual information of different scales by combining multiple convolution kernel sizes. This multi-scale feature learning is particularly important for target detection, which can help the model better identify small targets and objects in complex scenes and improve overall detection accuracy. Combining the GhostNet network with the RFB module can not only achieve a balance between parameters and computational complexity, but also improve the accuracy of the model. Especially in the face of object occlusion, complex environment and other situations, this combination can significantly improve detection capabilities. In addition, due to the lightweight nature of GhostNet and the efficient design of the RFB module, the overall model can achieve fast reasoning while maintaining high accuracy, which is extremely important for application scenarios that require real-time feedback during autonomous loader shoveling operations. Its structure is as follows Fig.15 shown.

[0097] (3) Add CBAM attention mechanism

[0098] The core idea of ​​the CBAM attention mechanism is to introduce adaptive weighting of spatial features and channel features in the convolutional neural network, thereby enhancing the network's effective extraction of key information that focuses on the input data. CBAM mainly consists of two parts. The channel attention calculates global statistics for the activation map of each channel, and then generates channel weights through normalization operations, and then applies these weights to the original feature map to emphasize important channel features; spatial attention obtains global spatial attention through convolution and pooling operations, and then uses a similar method to generate position weights. These weights represent the importance of different areas of the input image. Finally, these two weights will be multiplied element by element to adjust the spatial dimension of the original feature map. Its structure is as follows Fig.16 shown.

[0099] In a specific embodiment, the specific process of building the detection model also includes:

[0100] Construct a training dataset of piled material images and use it to train the detection model.

[0101] Specifically, in the process of constructing a training data set of images of stacked materials, in order to introduce different transformations so that the model can see more data under different angles and conditions and help the model learn a wider range of features, the embodiment of the present invention adopts data enhancement. Finally, after collecting and enhancing the data, a total of 2400 images of stacked materials with a resolution of 7296×5472 pixels are obtained.

[0102] In the process of selecting the loss function, the original loss function of the model is the CIOU loss function, which has the following problem: if the aspect ratio of the predicted box and the true box is the same, then the penalty term of the aspect ratio will be 0, which will obviously make the predicted box unable to perform regression optimization.

[0103] Therefore, the embodiment of the present invention replaces the original loss function with the EIoU loss function. EIoU separates the aspect ratio influencing factors of the predicted box and the true box based on the penalty term of CIoU, and calculates the length and width of the two respectively, thereby solving the problem of CIoU. The formula of the EIoU loss function is shown in formula (6):

[0104]

[0105] In the formula, ρ represents the distance between the center point of the predicted box A and the real box B, A ctr , B ctr represents the center point of the predicted box A and the real box B, c represents the diagonal length of the minimum closure box containing the predicted box A and the real box B, ω gt 、h gt are the width and height of the real box, ω and h are the width and height of the predicted box, C ωand C h It is the width and height of the minimum enclosing box containing two boxes, and IoU represents the overlap ratio of the “predicted box” and the “real box”.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying continuous shoveling positions of an autonomous loader based on deep learning, characterized in that: include: Acquire laser point cloud data of the material pile surface, and process the laser point cloud data to obtain corresponding features of multiple material pile shoveling areas; Constructing an execution device model of the loader, and performing a simulated force analysis on the execution device model and the material pile shoveling area characteristics to obtain corresponding material pile shoveling position characteristics; Analyzing and evaluating the material pile shoveling position characteristics to obtain the optimal material pile shoveling position characteristics; A detection model is constructed, and an image of a material pile to be identified is obtained at the same time. The image of the material pile to be identified is input into the detection model for processing, and the shoveling position of the loader is detected and adjusted according to the image processing result and the characteristics of the optimal material pile shoveling position.

2. The method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 1, characterized in that: The specific process of obtaining the corresponding multiple stockpile shoveling area features includes: Acquire laser point cloud data of the surface of the stockpile, and preprocess the laser point cloud data; The pre-processed laser point cloud data is segmented to obtain convex area features and concave area features of the pile, and the convex area features and concave area features are the shoveling area features of the pile.

3. The method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 2, characterized in that: The specific process of obtaining the optimal stockpile shoveling position characteristics includes: Inputting the actuator model into simulation software; Simulating the actuator model, the convex region features, and the concave region features while analyzing the force of the material on the side of the actuator model; Compare the forces on the side of the actuator model and select the one with the smallest force as the best pile shoveling position feature, that is, the convex area feature; The convex area characteristics are analyzed to obtain the optimal stockpile shoveling position characteristics.

4. According to the method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 3, the specific process of analyzing the convex area features to obtain the optimal pile shoveling position features includes: Acquire multiple edge convex regions included in the convex region feature; Calculate the edge curvature of each edge convex area in the simulation software, and calculate the maximum insertion resistance, maximum lifting resistance and maximum filling mass of each edge convex area; The maximum insertion resistance, maximum lifting resistance, and maximum filling mass were normalized; The spider graph evaluation method is used to calculate the degree value formed by each edge convex area on each axis of the spider graph; By comprehensively comparing multiple degree values, the optimal stockpile shoveling position characteristics are obtained.

5. The method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 1, characterized in that: The specific process of building a detection model includes: A detection model is constructed, wherein the detection model is a Yolov5s model, and the Yolov5s model is improved.

6. The method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 5, characterized in that: The specific process of improving the Yolov5s model includes: The backbone network of the Yolov5s model is replaced with the improved GhostNet network, the RFB module is introduced into the GhostNet network, the CBAM module is added, and EIoU is selected as the loss function.

7. The method for identifying continuous shoveling positions of an autonomous loader based on deep learning according to claim 5, characterized in that: The specific process of building a detection model also includes: Construct a training dataset of piled material images and use it to train the detection model.

Images