A loader driving control method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and storage medium for controlling the driving of a loader, which relates to the technical field of loaders. The environmental image and environmental point cloud image of the scene where the loader is located are respectively obtained from the vision sensor and lidar sensor configured on the loader. According to the environmental image and the pre-trained target detection model, the category of at least one target entity in the environmental image and the pixel coordinates of each target entity are obtained. According to the category of at least one target entity, it is determined whether at least one target entity includes a loading target. If a loading target is included, the position of the loading target in the scene is determined according to the pixel coordinates of the loading target and the environmental point cloud image. According to the position of the loading target in the scene, path planning is performed on the loader to obtain driving control parameters. According to the driving control parameters, the loader is controlled to drive to the position where the loading target is located along the driving path and perform a loading operation on the loading target, reducing the labor cost.

Description

Technical Field

[0001] This application relates to the technical field of loaders, and in particular, to a method, device, equipment and storage medium for controlling the driving of a loader. Background Art

[0002] Loaders are mainly used as earthwork construction machinery in construction projects such as highways, railways, buildings, hydropower, ports, mines, etc. They can perform loading operations. For example, loaders can shovel and load bulk materials such as soil, sand, gravel, lime, coal, etc., and can also perform light shoveling operations on ores, hard soil, etc.

[0003] In the prior art, loaders are driven manually to perform loading operations on materials. With the continuous increase in the number of construction projects and the continuous expansion of the project area, multiple loading operations on multiple materials are required, resulting in a large traffic flow of loaders. Since each loader needs to be driven manually to the material location, the labor cost is relatively high. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, device, equipment and storage medium for controlling the driving of a loader to solve the technical problem of relatively high labor cost in the prior art where each loader needs to be driven manually to the material location.

[0005] To achieve the above object, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for controlling the driving of a loader, the method including:

[0007] Obtain an environmental image and an environmental point cloud image of the scene where the loader is located from a vision sensor and a lidar sensor configured on the loader respectively;

[0008] According to the environmental image and a pre-trained target detection model, obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity;

[0009] According to the category of the at least one target entity, determine whether the at least one target entity includes a loading target;

[0010] If the loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image;

[0011] According to the position of the loading target in the scene, perform path planning on the loader to obtain driving control parameters;

[0012] Based on the driving control parameters, control the loader to drive to the position where the loading target is located along the driving path and perform a loading operation on the loading target.

[0013] Optionally, the determining the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image includes:

[0014] According to the pixel coordinates of the loading target, using the preset coordinate mapping relationship between the vision sensor and the lidar sensor, determine the three-dimensional point cloud data of the loading target from the environmental point cloud image;

[0015] According to the three-dimensional point cloud data of the loading target, determine the position of the loading target in the scene.

[0016] Optionally, if there are multiple loading targets, before performing path planning on the loader according to the positions of the loading targets in the scene to obtain driving control parameters, the method further includes:

[0017] According to the three-dimensional point cloud data of multiple loading targets, determine the sizes of the multiple loading targets and the distances between each loading target and the loader;

[0018] According to the sizes of the multiple loading targets and the distances between each loading target and the loader, determine the loading targets that meet the preset conditions from the multiple loading targets as path planning targets;

[0019] The performing path planning on the loader according to the positions of the loading targets in the scene to obtain driving control parameters includes:

[0020] According to the positions of the path planning targets in the scene, perform path planning on the loader to obtain the driving control parameters;

[0021] Optionally, the determining the loading targets that meet the preset conditions from the multiple loading targets as path planning targets according to the sizes of the multiple loading targets and the distances between each loading target and the loader includes:

[0022] According to the sizes of the multiple loading targets, determine the capacities of the multiple loading targets;

[0023] According to the capacities of the multiple loading targets and the distances between each loading target and the loader, select the loading target with the capacity meeting the preset minimum loading amount and the closest distance from the multiple loading targets as the path planning target.

[0024] Optionally, determining the sizes of the plurality of loading targets according to the three-dimensional point cloud data of the plurality of loading targets includes:

[0025] Mapping the three-dimensional point cloud data of the plurality of loading targets to a preset vertical mapping plane to obtain a vertical image of the three-dimensional point cloud data on the preset vertical mapping plane, and determining the size of the vertical image.

[0026] Mapping the three-dimensional point cloud data of the plurality of loading targets to a preset horizontal mapping plane to obtain a horizontal image of the three-dimensional point cloud data on the preset horizontal mapping plane, and determining the size of the horizontal image.

[0027] Optionally, determining the distance between each of the loading targets and the loader according to the three-dimensional point cloud data of the plurality of loading targets includes:

[0028] Obtaining the three-dimensional coordinates of the three-dimensional point cloud data of the plurality of loading targets according to the three-dimensional point cloud data of the plurality of loading targets and a preset coordinate system;

[0029] Calculating the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target;

[0030] Calculating the distance between each loading target and the loader by taking the average according to the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader.

[0031] Optionally, the method further includes:

[0032] Obtaining the frequency of the intermediate frequency signal of the obstacle collected by the millimeter wave radar sensor configured on the loader;

[0033] Determining the distance between the obstacle and the loader according to the frequency of the intermediate frequency signal;

[0034] If the distance is less than a preset threshold, controlling the loader to brake.

[0035] In a second aspect, an embodiment of the present application provides a loader travel control device, including:

[0036] An acquisition module, configured to respectively acquire an environmental image and an environmental point cloud image of the scene where the loader is located from a vision sensor and a lidar sensor configured on the loader;

[0037] A first obtaining module, configured to obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and a pre-trained target detection model;

[0038] A first determination module, configured to determine whether the at least one target entity includes a loading target according to the category of the at least one target entity;

[0039] A second determination module, configured to, if the loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image;

[0040] A second obtaining module, configured to perform path planning on the loader according to the position of the loading target in the scene, so as to obtain driving control parameters;

[0041] A control module, configured to control the loader to travel to the position where the loading target is located based on the driving path according to the driving control parameters, and perform a loading operation on the loading target.

[0042] In a third aspect, an embodiment of the present application provides a computer device, including: a storage medium and a processor, where the storage medium stores a computer program executable by the processor, and when the processor executes the computer program, a loader driving control method in the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is read and executed, a loader driving control method in the first aspect is implemented.

[0044] Compared with the prior art, the present application has the following beneficial effects:

[0045] A loader driving control method, device, equipment and storage medium provided by the present application obtain an environmental image and an environmental point cloud image of the scene where the loader is located respectively from a vision sensor and a lidar sensor configured on the loader, obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and a pre-trained target detection model, determine whether the at least one target entity includes a loading target according to the category of the at least one target entity, if the loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image, perform path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters, and control the loader to travel to the position where the loading target is located based on the driving path according to the driving control parameters and perform a loading operation on the loading target, realizing that the loader automatically identifies the loading target and automatically travels to the loading target to perform the loading operation. In this process, the loader does not need to be self-driving and does not require manual driving, which can reduce labor costs. Description of the Drawings

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0047] Figure 1 Schematic diagram of a loader driving control system provided by an embodiment of the present application;

[0048] Figure 2 Schematic flow chart of a loader driving control method provided by an embodiment of the present application;

[0049] Figure 3 Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0050] Figure 4 Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0051] Figure 5 Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0052] Figure 6 Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0053] Figure 7 [[ID=Z29]]Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0054] Figure 8 Schematic flow chart of another loader driving control method provided by an embodiment of the present application;

[0055] Figure 9 Schematic diagram of a loader driving control device provided by an embodiment of the present application;

[0056] Figure 10 Schematic diagram of a control device provided by an embodiment of the present application. Detailed implementation manners

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0058] Accordingly, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts fall within the scope of protection of the present application.

[0059] It should be noted that, without conflict, the features in the embodiments of the present invention can be combined with each other.

[0060] Before performing the loading operation, the loader needs to drive to the material location first. Generally, each loader is driven to the corresponding material location by manual driving, resulting in a relatively high labor cost. Therefore, the loader can be automatically controlled to drive to the corresponding material location automatically, which does not require manual driving, that is, does not require the participation of humans, and can reduce the labor cost.

[0061] In order to solve the technical problem in the prior art that each loader needs to be driven to the material location by manual driving, resulting in a relatively high labor cost, in the solution of the present application, a loader driving control system is provided. First, through a specific example, an explanation of a loader driving control system provided by the embodiments of the present application is given below. Figure 1 It is a schematic diagram of a loader driving control system provided by an embodiment of the present application, as Figure 1 shown, the loader driving control system includes: a control device 10, a vision sensor 20, a lidar sensor 30, and a millimeter-wave radar sensor 40.

[0062] The control device 10 is communicatively connected to the vision sensor 20, so that the control device 10 can obtain the environmental image information collected by the vision sensor 20 and process the environmental image information. The control device 10 is also communicatively connected to the lidar sensor 30, so that the control device 10 can obtain the environmental point cloud image information collected by the lidar sensor 30 and process the environmental point cloud image information. The control device 10 is also communicatively connected to the millimeter-wave radar sensor 40, so that the control device 10 can obtain the information collected by the millimeter-wave radar sensor 40 and process the information.

[0063] The control device 10 is arranged on the loader. The control device 10 can calculate the driving control parameters of the vehicle according to at least one of the environmental image information collected by the vision sensor 20, the environmental point cloud image information collected by the lidar sensor 30, and the information collected by the millimeter-wave radar sensor 40, so as to control the driving of the loader and perform the loading operation through the driving control parameters.

[0064] The application of the vision sensor 20 on the loader can be in the form of a camera. In the embodiment of the present application, the vision sensor 20 can be a monocular camera sensor, a binocular camera sensor, a trinocular camera sensor, or a surround-view camera sensor. Of course, it can also be other camera sensors, and no specific limitation is made in the embodiment of the present application.

[0065] The lidar sensor 30 is a sensor used to accurately obtain three-dimensional position information. It can detect targets using laser beams and obtain point cloud data. In the embodiment of the present application, the lidar sensor 30 can be a mechanical lidar sensor, a solid-state lidar sensor, or a hybrid solid-state lidar sensor. Of course, it can also be other lidar sensors, and no specific limitation is made in the embodiment of the present application.

[0066] The millimeter-wave radar sensor 40 realizes information acquisition by transmitting and receiving millimeter-wave electromagnetic waves. Among them, millimeter waves are electromagnetic waves with wavelengths between 1 and 10 mm. They have short wavelengths, wide frequency bands, are relatively easy to achieve narrow beams, have high radar resolution, and the emitted beams have excellent penetration and are not easily affected by smoke.

[0067] Optionally, both the vision sensor 20 and the lidar sensor 30 can be set on the top of the loader, and the millimeter-wave radar sensor 40 is installed in the bottom area of the loader for emergency obstacle avoidance, and its action range covers the blind areas of the lidar sensor 30 and the vision sensor 20. Of course, the installation positions of these multiple sensors are only examples, and they can also be other positions, and no specific limitation is made in the embodiment of the present application.

[0068] A loader driving control system provided by the present application. The control device can calculate the driving control parameters of the vehicle based on at least one of the environmental image information collected by the vision sensor, the environmental point cloud image information collected by the lidar sensor, and the information collected by the millimeter-wave radar sensor, so as to control the loader to perform automatic driving and loading operations through the driving control parameters, without the need for manual driving, reducing labor costs.

[0069] In order to reduce the labor cost of driving a loader, in the solution of the present application, a loader driving control method is provided. The following is an explanation of a loader driving control method provided by the embodiment of the present application through specific examples. Figure 2 It is a schematic flow chart of a loader driving control method provided by an embodiment of the present application. As Figure 2 shown, the execution subject of this method is the control device in the loader driving control system. The control device can be a device with computing, processing, and control functions. This method includes:

[0070] S201. Obtain the environmental image and environmental point cloud image of the scene where the loader is located from the vision sensor and lidar sensor configured on the loader respectively.

[0071] In the embodiment of the present application, the vision sensor can be used to collect the environmental image of the scene where the loader is located. The environmental image can be an image of a building, a utility pole, a person, etc. in the scene where the loader is located. The control device can communicate with the vision sensor to obtain the environmental image of the scene where the loader is located collected by the vision sensor in real time.

[0072] The lidar sensor can be used to collect the environmental point cloud image of the scene where the loader is located. The environmental point cloud image can be the point cloud data of a building, a utility pole, a person, etc. in the scene where the loader is located. The control device can communicate with the lidar sensor to obtain the environmental point cloud image of the scene where the loader is located collected by the lidar sensor in real time.

[0073] S202. Obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and the pre-trained target detection model.

[0074] A pre-trained target detection model is set inside the vision sensor. The target detection model can be a deep network detection model based on a convolutional neural network, such as the YOLO series. For example, it can be the YOLOV5 detection model. After training the model using the gradient descent algorithm, the trained target detection model is used for identifying target entities in the scene. Of course, the deep network detection model can be any detection model that meets the target detection task and is not specifically limited in the embodiment of the present application.

[0075] In the embodiment of the present application, inputting the environmental image into the pre-trained target detection model can obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity.

[0076] Among them, the training process of the target detection model is as follows: Initialize the target detection model, collect environmental images of different entities in the current scene for deep learning modeling, that is, deep training, and implant the established target detection model into the vision sensor for target entity recognition.

[0077] Optionally, the category of the target entity, the confidence level of the category, and the pixel coordinates of the target entity on the environmental image can be given for the image to be recognized. The pixel coordinates can be displayed as the minimum bounding rectangle of the target entity on the environmental image. When an image with low confidence or significantly incorrect position coordinates appears, the visual sensor sends the environmental image and the predicted results to the background server through a communication protocol, and the server adjusts according to the predicted results. If the confidence level meets the preset conditions, the environmental image can be added to the model training to optimize and improve the target detection model. If the confidence level is less than the minimum threshold, that is, the confidence level is extremely low, indicating that the target detection model does not currently include the recognition task of this category, the target detection model can be automatically optimized by adding classification. For example, the target category and coordinates can be adjusted manually.

[0078] Optionally, when the target detection model regularly receives sample images, the sampling module can be used to screen and clean the sample images, select the sample images required by the target detection model, and avoid the impact of unbalanced sample images on the detection effect of the target detection model. The sampling module mainly screens based on the similarity of image features. The features used to calculate the similarity are the feature values learned by the current target detection model, and these feature values are stored in a database, such as a MongoDB database.

[0079] Optionally, when the target detection model increases the sample images, the model is retrained and the model is verified according to the verified data. If the model verification result meets the preset conditions, the retrained target detection model is updated to the visual sensor.

[0080] S203. According to the category of at least one target entity, determine whether at least one target entity includes a loading target.

[0081] After obtaining the category of at least one target entity in the environmental image and the pixel coordinates of each target entity through a pre-trained target detection model, it can be determined whether there is a target entity with the category of loading target according to the category information. A target entity with the category of loading target can be called a loading target.

[0082] For example, if there are four target entities in the environmental image, and the categories of the four target entities are a, b, c, and d respectively. If the category of the loading target is e, then at least one target entity does not include a loading target. If the category of the loading target is b, then at least one target entity includes a loading target, and the loading target is the target entity corresponding to category b.

[0083] S204. If a loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image.

[0084] When at least one target entity includes a loading target, the position of the loading target in the scene can be determined according to the relevant position information of the loading target. The relevant position information can be the pixel coordinates of the loading target and the environmental point cloud image.

[0085] S205. Perform path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters.

[0086] According to the position of the loading target in the scene and the current position of the loader, obtain the driving path of the loader to the loading target. According to the driving path, driving control parameters can be obtained. The driving control parameters can be parameters such as the real-time front wheel angle and real-time acceleration of the loader on the driving path, which are not specifically limited in the embodiments of the present application.

[0087] S206. According to the driving control parameters, control the loader to drive to the position where the loading target is located based on the driving path and perform a loading operation on the loading target.

[0088] Optionally, when controlling the loader to drive to the position where the loading target is located based on the driving path according to the driving control parameters, the obstacles around the loader on the driving path can also be detected in real time by a vision sensor. The obstacles are other target entities except the loading target. If the distance of the obstacle is less than or equal to the minimum distance threshold, control the loader to brake. If the distance of the obstacle is greater than the minimum distance threshold and less than or equal to the maximum distance threshold, control the loader to decelerate.

[0089] A loader driving control method provided by the present application obtains the environmental image and environmental point cloud image of the scene where the loader is located, obtains the category of at least one target entity in the environmental image and the pixel coordinates of each target entity, determines whether at least one target entity includes a loading target. If it includes a loading target, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image, and then perform path planning on the loader to obtain driving control parameters. According to the driving control parameters, control the loader to drive to the position where the loading target is located based on the driving path and perform a loading operation on the loading target, realizing that the loader automatically identifies the loading target and automatically drives to the loading target to perform the loading operation. In this process, the loader does not need to be for autonomous driving and does not need manual driving, which can reduce labor costs.

[0090] In the above Figure 2 On the basis of a loader driving control method shown, the embodiments of the present application also provide an implementation method of another loader driving control method. Optionally, Figure 3 This is a schematic flowchart of another loader driving control method provided by the embodiments of the present application, as Figure 3As shown above, the above method S204 determines the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image, including:

[0091] S301. According to the pixel coordinates of the loading target, using the preset coordinate mapping relationship between the visual sensor and the lidar sensor, determine the three-dimensional point cloud data of the loading target from the environmental point cloud image.

[0092] According to the multi-sensor fusion technology, the data of the visual sensor and the lidar sensor can be fused to better perceive the surrounding environment.

[0093] In the embodiment of the present application, according to the multi-sensor fusion technology, a mapping relationship can be established between the pixel coordinates of the target entity in the 2D environmental image of the visual sensor and the 3D environmental point cloud coordinates of the lidar sensor.

[0094] If there is a loading target in the 2D environmental image of the visual sensor, then according to the pixel coordinates of the loading target, using the mapping relationship between the pixel coordinates of the target entity in the 2D environmental image of the visual sensor and the 3D environmental point cloud coordinates of the lidar sensor, determine the three-dimensional point cloud data of the loading target in the environmental point cloud image corresponding to the pixel coordinates of the loading target.

[0095] S302. According to the three-dimensional point cloud data of the loading target, determine the position of the loading target in the scene.

[0096] After obtaining the three-dimensional point cloud data of the loading target, the position of the center of the loader can be used as the origin of the coordinate system to determine the three-dimensional coordinates of all points in the three-dimensional point cloud data of the loading target. Of course, the position of the center of the lidar sensor or the visual sensor can also be used as the origin of the coordinate system, which is not specifically limited in the embodiment of the present application.

[0097] After determining the three-dimensional coordinates of all points in the three-dimensional point cloud data of the loading target, according to the three-dimensional coordinates of all points, the distance from all points to the origin of the coordinate system can be calculated. The three-dimensional coordinates of the point with the smallest distance can be selected as the position of the loading target in the scene, or the average distance can be calculated, and the three-dimensional coordinates of the point closest to the average distance can be found as the position of the loading target in the scene. Other appropriate three-dimensional coordinates of points can also be selected as the position of the loading target in the scene, which is not specifically limited in the embodiment of the present application.

[0098] A loader driving control method provided by the present application determines the three-dimensional point cloud data of a loading target from an environmental point cloud image according to the pixel coordinates of the loading target and by using a preset coordinate mapping relationship between a vision sensor and a lidar sensor. According to the three-dimensional point cloud data of the loading target, the position of the loading target in the scene is determined. Furthermore, the path of the loader can be planned based on the position of the loading target, and the loader is controlled to drive to the position where the loading target is located and perform a loading operation on the loading target, thereby realizing the unmanned automatic driving of the loader and reducing the labor cost.

[0099] Based on the above Figure 3 shown loader driving control method, the embodiment of the present application also provides an implementation method of another loader driving control method. Optionally, Figure 4 is a flowchart of another loader driving control method provided by the embodiment of the present application. As Figure 4 shown, if there are multiple loading targets, before the above method S205, that is, before planning the path of the loader according to the position of the loading target in the scene and obtaining the driving control parameters, it further includes:

[0100] S401, determine the sizes of multiple loading targets and the distances between each loading target and the loader according to the three-dimensional point cloud data of the multiple loading targets.

[0101] If there are multiple loading targets, one of them needs to be selected as the final loading target.

[0102] In the embodiment of the present application, the sizes of multiple loading targets and the distances between each loading target and the loader can be used as conditions for selecting the final loading target. Among them, the size of each loading target and the distance between each loading target and the loader can be calculated through the three-dimensional point cloud data of each loading target.

[0103] Optionally, before determining the sizes of multiple loading targets and the distances between each loading target and the loader according to the three-dimensional point cloud data of the multiple loading targets, the three-dimensional point cloud data of the multiple loading targets can also be processed. This processing can be to filter out noise points and isolated points, and then obtain effective three-dimensional point cloud data of the loading target. Among them, the mean and variance of the distances between points in the three-dimensional point cloud data of the loading target can be calculated, and according to the empirical rule, noise points and isolated points can be filtered out. This empirical rule can also be called the 3-sigma rule.

[0104] S402, determine the loading target that meets the preset conditions as the path planning target from multiple loading targets according to the sizes of multiple loading targets and the distances between each loading target and the loader.

[0105] The preset conditions are the size conditions of the loading target and the distance conditions between the loading target and the loader. According to the preset conditions among multiple loading targets, one of the loading targets is selected as the final loading target, which can also be called the path planning target.

[0106] In the above method S205, path planning is performed on the loader according to the position of the loading target in the scene to obtain driving control parameters, including: path planning is performed on the loader according to the position of the path planning target in the scene to obtain driving control parameters.

[0107] A loader driving control method provided by the present application. If there are multiple loading targets, the sizes of the multiple loading targets are determined according to the three-dimensional point cloud data of the multiple loading targets, and the distance between each loading target and the loader is determined. According to the sizes of the multiple loading targets and the distance between each loading target and the loader, the loading target that meets the preset conditions is determined from the multiple loading targets as the path planning target, and path planning is performed on the loader according to the position of the path planning target in the scene to obtain driving control parameters, which solves the problem of how to select one of the multiple loading targets by the loader when there are multiple loading targets in the scene.

[0108] In the above Figure 4 Based on the above-mentioned loader driving control method, the embodiment of the present application also provides an implementation method of another loader driving control method. Optionally, Figure 5 is a schematic flowchart of another loader driving control method provided by the embodiment of the present application. As Figure 5 shown, the above method S402, determining the loading target that meets the preset conditions from multiple loading targets as the path planning target according to the sizes of the multiple loading targets and the distance between each loading target and the loader, includes:

[0109] S501, determining the capacities of the multiple loading targets according to the sizes of the multiple loading targets.

[0110] In the embodiment of the present application, the size of each loading target includes the size of the vertical image corresponding to each loading target and the size of the horizontal image corresponding to each loading target.

[0111] Exemplarily, when the loading target is a material pile, the size of the vertical image corresponding to the loading target is the size of the image corresponding to the cross-section of the material pile, which may include dimensions such as the height of the material pile. The size of the horizontal image corresponding to the loading target is the size of the image corresponding to the bottom of the material pile, and the size can be determined according to the shape of the bottom of the material pile. Exemplarily, if the shape of the bottom of the material pile is circular, the size of the horizontal image may include dimensions such as the diameter of the circular bottom of the material pile.

[0112] According to the size of each loading target, the capacity of each loading target can be determined. By way of example, when the loading target is a material pile, the size of the vertical image corresponding to the loading target includes the height of the material pile, and the size of the horizontal image corresponding to the loading target includes the diameter of the bottom of the material pile. At this time, the material pile is a cone. According to the volume calculation formula of the cone, by substituting the height information and diameter information of the material pile, the volume of the material pile can be calculated, that is, the capacity of the loading target.

[0113] The above is only an example of determining the capacity of the loading target when the loading target is a material pile. The loading target can be of other shapes, and there can also be other methods for calculating the capacity, which are not specifically limited in the embodiments of the present application.

[0114] S502. According to the capacity of multiple loading targets and the distance between each loading target and the loader, select, from the multiple loading targets, the loading target whose capacity meets the preset minimum loading amount and is the closest as the path planning target.

[0115] After obtaining the capacity of each loading target and the distance between each loading target and the loader, according to the principle of "nearest and sufficient quantity", select the loading target that meets the preset minimum loading amount of the loader and is the closest as the path planning target.

[0116] According to the principle of "nearest and sufficient quantity", the automatic driving path of the loader can be made the shortest, reducing the time consumed by the loader during driving and saving time resources.

[0117] A loader driving control method provided by the present application determines the capacities of multiple loading targets according to the sizes of the multiple loading targets, and selects, from the multiple loading targets, the loading target whose capacity meets the preset minimum loading amount and is the closest as the path planning target according to the capacities of the multiple loading targets and the distance between each loading target and the loader. That is, it solves the problem of how the loader selects one of the multiple loading targets in the scenario, and also makes the automatic driving path of the loader the shortest while meeting the loading amount, saving time resources and improving the efficiency of autonomous driving.

[0118] In the above Figure 4 On the basis of the shown loader driving control method, the embodiments of the present application also provide an implementation method of another loader driving control method. Optionally, Figure 6 is a schematic flowchart of another loader driving control method provided by the embodiments of the present application. As Figure 6 shown, the above method S401, according to the three-dimensional point cloud data of multiple loading targets, determines the sizes of the multiple loading targets, including:

[0119] S601. Map the three-dimensional point cloud data of multiple loading targets to a preset vertical mapping plane to obtain a vertical image of the three-dimensional point cloud data on the preset vertical mapping plane, and determine the size of the vertical image.

[0120] Mapping the three-dimensional point cloud data of multiple loading targets to the same preset vertical mapping plane can obtain multiple two-dimensional images corresponding to the three-dimensional point cloud data of multiple loading targets. These multiple two-dimensional images are located in the preset vertical mapping plane, and these multiple two-dimensional images can also be referred to as the vertical images of multiple loading targets.

[0121] Based on the vertical images of multiple loading targets, the size of the vertical image can be determined. For example, the size of the vertical image corresponding to a loading target can be the size of the image corresponding to the cross-section of the loading target, and can include dimensions such as the height of the loading target.

[0122] S602. Map the three-dimensional point cloud data of multiple loading targets to a preset horizontal mapping plane to obtain a horizontal image of the three-dimensional point cloud data on the preset horizontal mapping plane, and determine the size of the horizontal image.

[0123] Mapping the three-dimensional point cloud data of multiple loading targets to the same preset horizontal mapping plane can obtain multiple two-dimensional images corresponding to the three-dimensional point cloud data of multiple loading targets. These multiple two-dimensional images are located in the preset horizontal mapping plane, and these multiple two-dimensional images can also be referred to as the horizontal images of multiple loading targets.

[0124] Based on the horizontal images of multiple loading targets, the size of the horizontal image can be determined. For example, the size of the horizontal image corresponding to a loading target can be the size of the image corresponding to the bottom of the loading target, and can include dimensions such as the diameter or length or width corresponding to the shape of the bottom of the loading target.

[0125] It can be understood that the order of the above S601 and S602 is not limited to performing S601 first and then S602. It can also perform S602 first and then S601, or the two steps can be performed simultaneously, and there is no specific limitation in the embodiments of this application.

[0126] A loader driving control method provided in this application maps the three-dimensional point cloud data of multiple loading targets to a preset vertical mapping plane to obtain a vertical image of the three-dimensional point cloud data on the preset vertical mapping plane, determines the size of the vertical image, maps the three-dimensional point cloud data of multiple loading targets to a preset horizontal mapping plane to obtain a horizontal image of the three-dimensional point cloud data on the preset horizontal mapping plane, determines the size of the horizontal image, and based on the obtained size information, further can determine the capacity of multiple loading targets based on the size information of multiple loaders, select one of the loaders as the path planning target, so that the path planning target meets the requirement of the minimum loading amount.

[0127] Based on the above Figure 4 shown method for controlling the travel of a loader, the embodiments of the present application further provide an implementation method for another method for controlling the travel of a loader. Optionally, Figure 7 is a schematic flowchart of another method for controlling the travel of a loader provided by the embodiments of the present application. As Figure 7 shown, in the above method S401, according to the three-dimensional point cloud data of multiple loading targets, determining the distance between each of the loading targets and the loader includes:

[0128] S701, obtaining the three-dimensional coordinates of the three-dimensional point cloud data of multiple loading targets according to the three-dimensional point cloud data of multiple loading targets and a preset coordinate system.

[0129] Exemplarily, when the origin of the preset coordinate system is the position where the center of the loader is located, the loader can set the origin of the preset coordinate system. According to the preset coordinate system and the three-dimensional point cloud data of multiple loading targets, the three-dimensional coordinates of the three-dimensional point cloud data of multiple loading targets in the preset coordinate system can be obtained. Of course, the position where the center of the lidar sensor or the vision sensor is located can also be used as the origin of the coordinate system, which is not specifically limited in the embodiments of the present application.

[0130] Optionally, when the origin of the preset coordinate system is at other positions, according to the preset coordinate system and the three-dimensional point cloud data of multiple loading targets, the three-dimensional coordinates of the three-dimensional point cloud data of multiple loading targets in the preset coordinate system can be obtained. According to the preset coordinate system and the position of the loader, the three-dimensional coordinates of the loader in the preset coordinate system can be obtained. Of course, the three-dimensional coordinates of the lidar sensor or the vision sensor in the preset coordinate system can also be obtained.

[0131] S702, calculating the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target.

[0132] When the origin of the preset coordinate system is the position where the center of the loader is located, the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader can be calculated according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target. When the origin of the preset coordinate system is not the position where the center of the loader is located, the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader can be calculated according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the three-dimensional coordinates of the center position of the loader.

[0133] S703, selecting the minimum distance between the three-dimensional point cloud data of each loading target and the loader as the distance between each loading target and the loader.

[0134] Optionally, the average value between the 3D point cloud data of each loading target and the loader can also be calculated first, and this average value can be used as the distance between each loading target and the loader. Other distance information can also be calculated as the distance between each loading target and the loader, which is not specifically limited in the embodiments of the present application.

[0135] A loader driving control method provided by the present application obtains the 3D coordinates of the 3D point cloud data of multiple loading targets according to the 3D point cloud data of multiple loading targets and a preset coordinate system. According to the 3D coordinates of the 3D point cloud data of each loading target, the distance between the 3D coordinates of the 3D point cloud data of each loading target and the loader is calculated. The minimum distance between the 3D point cloud data of each loading target and the loader is selected as the distance between each loading target and the loader, and the distance between each loading target and the loader is obtained. Furthermore, according to this distance, one of the loaders can be selected as the path planning target, so that the distance between the path planning target and the loader is relatively close, and the automatic driving path is shorter, which can save time resources.

[0136] In the above Figure 2 Based on the shown loader driving control method, the embodiments of the present application also provide an implementation method of another loader driving control method. Optionally, Figure 8 is a schematic flowchart of another loader driving control method provided by the embodiments of the present application. As Figure 8 shown, this method further includes:

[0137] S801, obtain the frequency of the intermediate frequency signal of the obstacle collected by the millimeter-wave radar sensor configured on the loader.

[0138] The millimeter-wave radar sensor detects the surrounding obstacles at the bottom of the vehicle in real time, and controls the loader according to the relevant information of the obstacles to ensure the safety of the bottom of the loader during driving.

[0139] The millimeter-wave radar sensor uses a chirp signal as the transmitted signal. The chirp signal can be generated by a synthesizer, transmitted through an antenna, and received by a receiving antenna after the signal is transmitted back. The received signal and the transmitted signal are mixed to generate an intermediate frequency signal.

[0140] For the millimeter-wave radar sensor, the distance between the millimeter-wave radar sensor and the obstacle can be calculated through the frequency of the intermediate frequency signal. In the embodiments of the present application, the distance between the millimeter-wave radar sensor and the obstacle can be used as the distance between the obstacle and the loader.

[0141] S802, determine the distance between the obstacle and the loader according to the frequency of the intermediate frequency signal.

[0142] In the embodiments of the present application, the frequency of the intermediate frequency signal is proportional to the distance between the millimeter-wave radar sensor and the obstacle.

[0143] The distance between the obstacle and the millimeter-wave radar sensor can be determined by calculation according to the frequency modulation slope of the chirp signal generated by the synthesizer in the millimeter-wave radar sensor, the frequency of the intermediate frequency signal, and the speed of light, and used as the distance between the obstacle and the loader.

[0144] Exemplarily, after multiplying the frequency of the intermediate frequency signal by the speed of light and dividing it by the product of the frequency modulation slope of the chirp signal and 2, the distance between the obstacle and the millimeter-wave radar sensor can be obtained and used as the distance between the obstacle and the loader.

[0145] S803, if the distance is less than the preset threshold, control the loader to brake.

[0146] The millimeter-wave radar detects the obstacles around the vehicle in real time.

[0147] If the distance between the obstacle and the loader is less than the specified threshold, the loader performs emergency braking to ensure driving safety and prevent the occurrence of the "ghost probe" event. If the distance between the obstacle and the loader is not less than the specified threshold, no other control operations are performed on the loader.

[0148] A loader driving control method provided by the present application obtains the frequency of the intermediate frequency signal of the obstacle collected by the millimeter-wave radar sensor configured on the loader, determines the distance between the obstacle and the loader according to the frequency of the intermediate frequency signal, and if the distance is less than the preset threshold, controls the loader to brake, improving the driving safety of the loader.

[0149] The following describes a loader driving control device, a control device, and a storage medium provided by the present application for execution. The specific implementation process and technical effects are as described above and will not be repeated below.

[0150] Figure 9 It is a schematic diagram of a loader driving control device provided by an embodiment of the present application, as Figure 9 shown. The loader driving control device includes:

[0151] An acquisition module 901, configured to acquire an environmental image and an environmental point cloud image of the scene where the loader is located from a vision sensor and a lidar sensor configured on the loader respectively.

[0152] A first obtaining module 902, configured to obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and a pre-trained target detection model.

[0153] The first determination module 903 is configured to determine whether at least one target entity includes a loading target according to the category of at least one target entity.

[0154] The second determination module 904 is configured to, if a loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image.

[0155] The second obtaining module 905 is configured to perform path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters.

[0156] The control module 906 is configured to control the loader to travel to the position where the loading target is located based on the driving path according to the driving control parameters and perform a loading operation on the loading target.

[0157] Optionally, the second determination module 904 is specifically configured to determine the three-dimensional point cloud data of the loading target from the environmental point cloud image according to the pixel coordinates of the loading target by using a preset coordinate mapping relationship between a vision sensor and a lidar sensor; and determine the position of the loading target in the scene according to the three-dimensional point cloud data of the loading target.

[0158] Optionally, the second obtaining module 905 is further configured to determine the sizes of multiple loading targets and the distances between each loading target and the loader according to the three-dimensional point cloud data of the multiple loading targets; determine a loading target that meets a preset condition from the multiple loading targets as a path planning target according to the sizes of the multiple loading targets and the distances between each loading target and the loader; and perform path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters, including: performing path planning on the loader according to the position of the path planning target in the scene to obtain driving control parameters.

[0159] Optionally, the second obtaining module 905 is specifically configured to determine the capacities of the multiple loading targets according to the sizes of the multiple loading targets; and select a loading target with a capacity that meets a preset minimum loading amount and is the closest in distance from the multiple loading targets as the path planning target according to the capacities of the multiple loading targets and the distances between each loading target and the loader.

[0160] Optionally, the second obtaining module 905 is specifically configured to map the three-dimensional point cloud data of the multiple loading targets to a preset vertical mapping plane to obtain a vertical image of the three-dimensional point cloud data on the preset vertical mapping plane and determine the size of the vertical image; and map the three-dimensional point cloud data of the multiple loading targets to a preset horizontal mapping plane to obtain a horizontal image of the three-dimensional point cloud data on the preset horizontal mapping plane and determine the size of the horizontal image.

[0161] Optionally, the second obtaining module 905 is specifically configured to obtain the three-dimensional coordinates of the three-dimensional point cloud data of multiple loading targets according to the three-dimensional point cloud data of the multiple loading targets and a preset coordinate system; calculate the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target; and select the minimum distance between the three-dimensional point cloud data of each loading target and the loader as the distance between each loading target and the loader.

[0162] The third determining module 907 is configured to obtain the frequency of the intermediate frequency signal of the obstacle collected by the millimeter wave radar sensor configured on the loader; determine the distance between the obstacle and the loader according to the frequency of the intermediate frequency signal; and if the distance is less than a preset threshold, control the loader to brake.

[0163] The above modules may be one or more integrated circuits configured to implement the above methods. For example: one or more application specific integrated circuits (ASICs), or, one or more microprocessors (digital singnal processors, DSPs), or, one or more field programmable gate arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0164] Figure 10 A schematic diagram of a control device provided by an embodiment of the present application. The control device may be a control device with computing, processing, and control functions.

[0165] The control device includes: a processor 100, a storage medium 1002, and a bus 1003. The processor 1001 and the storage medium 1002 are connected through the bus 1003.

[0166] The storage medium 1002 is used to store a program, and the processor 1001 calls the program stored in the storage medium 1002 to execute the above method embodiments. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0167] Optionally, the present invention further provides a program product, such as a computer-readable storage medium, including a program that is used to execute the above method embodiments when executed by a processor.

[0168] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0171] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units are stored in a storage medium, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute some steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (English: Read-Only Memory, abbreviated as: ROM), random access memories (English: Random Access Memory, abbreviated as: RAM), magnetic disks or optical discs that can store program codes.

[0172] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A method for controlling the travel of a loader, characterized in that, Including: Obtaining an environmental image and an environmental point cloud image of the scene where the loader is located from a vision sensor and a lidar sensor configured on the loader respectively, where the environmental point cloud image is the environmental point cloud image of the scene where the loader is located collected by the lidar sensor; Obtaining the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and a pre-trained target detection model; Determining whether the at least one target entity includes a loading target according to the category of the at least one target entity; If the loading target is included, determining the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image; Performing path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters; Controlling the loader to drive to the position where the loading target is located based on the driving path and perform a loading operation on the loading target according to the driving control parameters.

2. The loader travel control method according to claim 1, wherein, The determining the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image includes: Determining the three-dimensional point cloud data of the loading target from the environmental point cloud image according to the pixel coordinates of the loading target by using a preset coordinate mapping relationship between the vision sensor and the lidar sensor; Determining the position of the loading target in the scene according to the three-dimensional point cloud data of the loading target.

3. The loader travel control method according to claim 2, wherein If there are multiple loading targets, before performing path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters, the method further includes: Determining the sizes of the multiple loading targets and the distances between each loading target and the loader according to the three-dimensional point cloud data of the multiple loading targets; Determining a loading target that meets a preset condition from the multiple loading targets as a path planning target according to the sizes of the multiple loading targets and the distances between each loading target and the loader; The performing path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters includes: Performing path planning on the loader according to the position of the path planning target in the scene to obtain the driving control parameters.

4. The loader travel control method according to claim 3, characterized in that, The determining a loading target that meets a preset condition from the multiple loading targets as a path planning target according to the sizes of the multiple loading targets and the distances between each loading target and the loader includes: Determining the capacities of the multiple loading targets according to the sizes of the multiple loading targets; Selecting, from the multiple loading targets, a loading target with a capacity meeting a preset minimum loading amount and the shortest distance as the path planning target according to the capacities of the multiple loading targets and the distances between each loading target and the loader.

5. The loader travel control method according to claim 3, characterized in that The determining the sizes of the multiple loading targets according to the three-dimensional point cloud data of the multiple loading targets includes: Map the three-dimensional point cloud data of multiple said loading targets to a preset vertical mapping plane to obtain a vertical image of the three-dimensional point cloud data on the preset vertical mapping plane, and determine the size of the vertical image; Map the three-dimensional point cloud data of multiple said loading targets to a preset horizontal mapping plane to obtain a horizontal image of the three-dimensional point cloud data on the preset horizontal mapping plane, and determine the size of the horizontal image.

6. The loader travel control method according to claim 3, characterized in that, The determining the distance between each said loading target and the loader according to the three-dimensional point cloud data of multiple said loading targets includes: Obtain the three-dimensional coordinates of the three-dimensional point cloud data of multiple said loading targets according to the three-dimensional point cloud data of multiple said loading targets and a preset coordinate system; Calculate the distance between the three-dimensional coordinates of the three-dimensional point cloud data of each loading target and the loader according to the three-dimensional coordinates of the three-dimensional point cloud data of each loading target; Select the minimum distance between the three-dimensional point cloud data of each loading target and the loader as the distance between each said loading target and the loader.

7. The loader travel control method according to claim 1, characterized in that, The method further includes: Obtain the frequency of the intermediate frequency signal of an obstacle collected by a millimeter wave radar sensor configured on the loader; Determine the distance between the obstacle and the loader according to the frequency of the intermediate frequency signal; If the distance is less than a preset threshold, control the loader to brake.

8. A traveling control device for a loader, characterized in that, including: An acquisition module, configured to respectively obtain an environmental image and an environmental point cloud image of the scene where the loader is located from a vision sensor and a lidar sensor configured on the loader, and the environmental point cloud image is the environmental point cloud image of the scene where the loader is located collected by the lidar sensor; A first obtaining module, configured to obtain the category of at least one target entity in the environmental image and the pixel coordinates of each target entity according to the environmental image and a pre-trained target detection model; A first determining module, configured to determine whether the at least one target entity includes a loading target according to the category of the at least one target entity; A second determining module, configured to, if the loading target is included, determine the position of the loading target in the scene according to the pixel coordinates of the loading target and the environmental point cloud image; A second obtaining module, configured to perform path planning on the loader according to the position of the loading target in the scene to obtain driving control parameters; A control module, configured to control the loader to travel to the position where the loading target is located and perform a loading operation on the loading target based on a driving path according to the driving control parameters.

9. A control device, characterized in that, including: A storage medium and a processor, the storage medium stores a computer program executable by the processor, and when the processor executes the computer program, it implements the loader driving control method according to any one of claims 1-7 above.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is read and executed, it implements the loader driving control method according to any one of claims 1-7 above.

Images