Mining point recommendation method and device based on multi-sensor fusion, equipment and medium

Abstract

The application provides a kind of based on multi-sensor fusion's excavation point recommendation method, device, equipment and medium, it is related to excavation point recommendation technical field, including: obtaining the visual data of excavator working scene;Wherein, visual data includes scene image data and scene point cloud data;Through the target area prediction model obtained by pre-training, the excavatable area range in excavator working scene is determined based on scene image data;Based on the excavatable area range, determine the alternative point cloud data from scene point cloud data;According to the preset excavator limiting parameter, determine the target recommended excavation point in excavator working scene from alternative point cloud data.The application can accurately and quickly recommend excavation point, better assist excavator to carry out excavation work, so as to reduce loading time and improve loading efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of excavation point recommendation, in particular to a multi-sensor fusion-based excavation point recommendation method, device, equipment and medium. BACKGROUND

[0002] Most of the current remote control excavator systems are simulated cockpits, which remotely control the excavator by simulating the real excavator cockpit and integrating video transmission. This technology not only allows the operator to work in a comfortable environment, but also ensures the safety of the operator and improves the operation quality and efficiency. However, for tasks such as excavation or side dumping that require repeated operations, the operator still needs to constantly perform complex operations in the simulated cockpit, which requires high skills of the excavator operator and is prone to fatigue and errors.

[0003] As one of the routine operations of the excavator, loading is also a construction operation that the excavator operator must complete with high concentration in actual operation. Therefore, it is of practical significance to integrate the function of automatic loading operation in the image transmission interface of the remote control excavator. This technology allows the operator to only input the excavation direction of the loading operation on the remote interface, and the excavator can automatically complete the loading operation, reducing the safety risks caused by the need for high concentration during remote operation, releasing the labor of the operator, and improving the overall operation efficiency.

[0004] Planning the trajectory path of the loading operation requires an algorithm module to provide an optimal excavation point in each operation. The current common excavation point prediction schemes are divided into rule-based schemes and key point detection-based schemes. Among them, the rule-based scheme is simple and easy to implement, but it is prone to failure of motion control planning due to inaccurate excavation points, thereby interrupting the regional loading process. The key point detection-based scheme needs to select a large number of digging sequences and label the digging point positions for the digging sequences, which not only makes the labeling work very tedious, but also may introduce errors in recording the digging point positions during the labeling process, resulting in large prediction deviations and thus unreliable excavation point recommendation. SUMMARY

[0005] Therefore, the present application provides a multi-sensor fusion-based excavation point recommendation method, device, equipment and medium, which can accurately and quickly recommend excavation points, better assist the excavator in excavation operations, thereby reducing the loading time and improving the loading efficiency.

[0006] In a first aspect, an embodiment of the present application provides a multi-sensor fusion-based excavation point recommendation method, comprising:

[0007] obtaining visual data of an excavator operation scene; wherein the visual data comprises scene image data and scene point cloud data;

[0008] determine a excavatable region range in the excavator working scene based on the scene image data by using a target region prediction model pre-trained, wherein the target region prediction model is trained by using training images as input data and the excavatable region range as output data;

[0009] determine candidate point cloud data from the scene point cloud data based on the excavatable region range;

[0010] determine a target recommended digging point in the excavator working scene from the candidate point cloud data according to a preset excavator limiting parameter.

[0011] In a second aspect, an excavator digging point recommendation device based on multi-sensor fusion is also provided, which comprises:

[0012] a data acquisition module configured to acquire visual data of an excavator working scene, wherein the visual data comprises scene image data and scene point cloud data;

[0013] a region prediction module configured to determine a excavatable region range in the excavator working scene based on the scene image data by using a target region prediction model pre-trained, wherein the target region prediction model is trained by using training images as input data and the excavatable region range as output data;

[0014] a candidate point cloud determination module configured to determine candidate point cloud data from the scene point cloud data based on the excavatable region range;

[0015] a digging point recommendation module configured to determine a target recommended digging point in the excavator working scene from the candidate point cloud data according to a preset excavator limiting parameter.

[0016] In a third aspect, an electronic device is also provided, which comprises a processor and a memory, wherein the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0017] In a fourth aspect, a computer readable storage medium is also provided, which stores computer executable instructions, and the computer executable instructions, when called and executed by a processor, cause the processor to implement the method according to any one of the first aspect.

[0018] The method, device, equipment and medium provided by the embodiment of the present application can first determine the excavatable region range in the excavator working scene based on the scene image data by using the target region prediction model obtained through pre-training, then determine the candidate point cloud data from the scene point cloud data based on the excavatable region range, and finally determine the target recommended digging point in the excavator working scene from the candidate point cloud data according to the preset excavator limiting parameter. The above method can ensure that the target recommended digging point determined finally is located in the excavatable region, avoid the reduction of loading efficiency caused by digging errors, and has lower labeling complexity compared with the scheme based on key point detection in the related art, and can avoid the problem that the prediction deviation is large due to the error introduced in the digging point position labeling process. In addition, the above method can determine the target recommended digging point from the candidate point cloud data according to the excavator limiting parameter, ensure that the target recommended digging point can be reached by the large arm posture and the small arm posture of the excavator, and thus can avoid the problem of low efficiency caused by reselecting the point due to the too close or too far distance between the excavator and the digging point. Therefore, the embodiment of the present application can accurately and quickly recommend the digging point, better assist the excavator to perform the digging work, and thus reduce the loading time and improve the loading efficiency.

[0019] Other features and advantages of the present application will be set forth in the descriptions below, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description, claims and drawings.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative labor.

[0022] Figure 1 A rule-based digging point prediction scheme diagram is provided for the embodiment of the present application.

[0023] Figure 2A flowchart of a multi-sensor fusion-based excavation point recommendation method provided by an embodiment of the present application is shown in the figure.

[0024] Figure 3 A schematic diagram of a laser support provided by an embodiment of the present application is shown in the figure.

[0025] Figure 4 A schematic diagram of a range of excavatable areas provided by an embodiment of the present application is shown in the figure.

[0026] Figure 5 A schematic diagram of an excavatable area label provided by an embodiment of the present application is shown in the figure.

[0027] Figure 6 A schematic diagram of the state of a shovel when the zero position is provided by an embodiment of the present application is shown in the figure.

[0028] Figure 7 A schematic diagram of a target recommended excavation point provided by an embodiment of the present application is shown in the figure.

[0029] Figure 8 A structural schematic diagram of a multi-sensor fusion-based excavation point recommendation device provided by an embodiment of the present application is shown in the figure.

[0030] Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the present application will be described in detail below with reference to the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] Planning the trajectory path of the loading operation requires an algorithm module to provide an optimal excavation point in each round of operation, so the embodiments of the present application play an important role in automatic loading tasks. This technology has broad application prospects and is expected to play an even more important role in the future field of excavator automation.

[0033] Currently, common excavation point prediction schemes are divided into rule-based schemes and key point detection-based schemes. Specifically:

[0034] (1) The rule-based scheme is a relatively traditional excavation point prediction method, and its core idea is to limit the excavation point within a ring-shaped zone through prior knowledge, such as Figure 1A rule-based excavation point prediction scheme is shown in the figure, and then a 2D point is randomly selected in the band as an excavation point in each round. In the specific implementation, the rule-based excavation point generation scheme is simple and easy to implement, but has certain limitations, because when the height of the 2D ring-shaped band corresponding to the real 3D scene changes, the accuracy and applicability of the ring-shaped band will be affected, resulting in inaccurate excavation points, causing the motion control planning to fail, thereby interrupting the regional loading process.

[0035] (2) The key point detection-based scheme is divided into a 2D scheme and a 3D scheme. The 2D scheme is based on deep learning to predict a 2D excavation point on an image, and the 3D scheme is based on deep learning to predict a 3D excavation point on a 3D mesh. In the specific implementation, the key point prediction-based scheme needs to label a large number of excavation (stone) sequences. The 2D key point prediction needs to record the position of the contact between the shovel and the ground at the excavation moment, and label the recorded position on the image in which the shovel does not appear. In addition, the 3D scheme needs to reconstruct the excavation sequence image into a mesh, label the excavation point position on the mesh, and filter the large arm and small arm of the excavator. However, both the 2D and 3D key point prediction schemes need to select a large number of excavation sequences and record the position of the contact between the shovel and the ground, and the labeling work is very tedious. Moreover, the recording of the excavation point position in the labeling process may introduce errors, resulting in a large prediction deviation.

[0036] Therefore, the present application provides a multi-sensor fusion-based excavation point recommendation method, device, equipment and medium, which can accurately and quickly recommend excavation points, better assist the excavator in excavation operation, thereby reducing the loading time and improving the loading efficiency.

[0037] In order to facilitate the understanding of the present application, first, a multi-sensor fusion-based excavation point recommendation method is introduced in detail, as shown in Figure 2 A flowchart of a multi-sensor fusion-based excavation point recommendation method is shown in the figure. The method mainly includes the following steps S202 to S208:

[0038] Step S202, acquiring visual data of the excavator working scene.

[0039] The visual data includes scene image data and scene point cloud data. In an embodiment, the excavator is configured with an image acquisition device (such as a camera) and a plurality of point cloud acquisition devices (such as lasers). The image acquisition device is used to acquire scene image data of the excavator working scene, and each point cloud acquisition device is used to acquire scene point cloud data of the excavator working scene.

[0040] At step S204, the target region prediction model obtained through pre-training is used to determine the excavatable region range in the excavator working scene based on the scene image data. The target region prediction model is obtained by training the training image as input data and the excavatable region range as output data.

[0041] In an embodiment, the target region prediction model can adopt an HRNet model, the input of the target region prediction model is the scene image data, and the output of the target region prediction model is the excavatable region range. In an embodiment, the excavatable region can be defined in advance, and the excavatable region label is labeled in the training image. The HRNet model is used as the backbone network, the training image and the excavatable region label are used to train the HRNet model, and the target region prediction model required is obtained. Then, the scene image data is input into the target region prediction model, and the target region prediction model is used to predict the excavatable region range.

[0042] At step S206, the candidate point cloud data is determined from the scene point cloud data based on the excavatable region range.

[0043] In an embodiment, the point cloud data outside the excavatable region range can be removed from the scene point cloud data, and the point cloud data inside the excavatable region range is retained. The retained point cloud data is subjected to voxel downsampling processing to obtain the candidate point cloud data.

[0044] At step S208, the target recommended digging point in the excavator working scene is determined from the candidate point cloud data according to the preset excavator limiting parameter.

[0045] In an embodiment, one candidate recommended digging point can be determined from the candidate point cloud data, and it is determined whether the candidate recommended digging point, the origin of the excavator boom coordinate system, and the origin of the excavator arm coordinate system satisfy the triangular set relationship when the excavator performs the digging operation. If the result of the determination is yes, the boom pose and the arm pose required for the excavator to reach are determined, so as to determine whether the boom pose and the arm pose satisfy the excavator limiting parameter. If the boom pose and the arm pose satisfy the excavator limiting parameter, the candidate recommended digging point is determined as the target recommended digging point. If the boom pose and / or the arm pose do not satisfy the excavator limiting parameter, a new candidate recommended digging point needs to be determined, and the above process is repeated until the target recommended digging point is determined.

[0046] The method for recommending a digging point based on multi-sensor fusion provided in the embodiments of the present application can determine the range of the excavatable area in the working scene of the excavator through a target area prediction model, can ensure that the finally determined target recommended digging point is located in the excavatable area, and can avoid the reduction of loading efficiency caused by excavation errors. Compared with the scheme based on key point detection in the related art, the labeling complexity of the embodiments of the present application is lower, and the problem that the prediction deviation is large due to the errors introduced in the process of labeling the position of the digging point can be avoided. In addition, the above method can determine the target recommended digging point from the alternative point cloud data according to the limiting parameters of the excavator, can ensure that the target recommended digging point can be reached by both the large arm posture and the small arm posture of the excavator, and can thus avoid the problem of low efficiency caused by re-selecting the point due to the too close or too far distance between the excavator and the digging point. Therefore, the embodiments of the present application can accurately and quickly recommend the digging point, can better assist the excavator to perform the excavation work, and can thus reduce the loading time and improve the loading efficiency.

[0047] For the convenience of understanding, a specific implementation of the method for recommending a digging point based on multi-sensor fusion is provided in the embodiments of the present application.

[0048] For the foregoing step S202, an implementation of acquiring visual data of the working scene of the excavator is provided in the embodiments of the present application. The scene image data collected by the image acquisition device for the working scene of the excavator can be acquired, and the scene point cloud data collected by each point cloud acquisition device for the working scene of the excavator can be acquired.

[0049] In the specific implementation, the parameters such as laser fov and laser density need to be considered comprehensively. For example, the DJI Aolun laser can be selected. Since the horizontal field of view of a single DJI Aolun laser is only 70°, it is difficult to cover the entire excavatable area in the main viewing angle. Therefore, multiple lasers need to be used to collect the scene point cloud data of the working scene of the excavator. For example, three lasers can be used to collect the scene point cloud data of the working scene of the excavator. In actual application, the bracket is used to install the three lasers on the excavator, such as the laser bracket shown in FIG. 3. Figure 3 FIG. 3 is a schematic view of a laser bracket, three lasers are installed on the left, middle and right respectively, and the horizontal field of view reaches 180° after splicing through mapping and other calibration algorithms. Figure 3 FIG. 3 is a schematic view of a laser bracket, three lasers are installed on the left, middle and right respectively, and the horizontal field of view reaches 180° after splicing through mapping and other calibration algorithms.

[0050] In an example, after the camera and the laser are installed on the excavator, the camera needs to be calibrated. Optionally, the chessboard calibration board is used, and the Zhang Zhengyou calibration algorithm is used to obtain the intrinsic parameters of the camera, including the principal point, distortion parameters, focal length, etc.

[0051] In one case, it is also necessary to calibrate the laser and the camera in advance. Optionally, the extrinsic parameters of the laser and the camera can be calibrated through an April tag calibration version, which includes rotation and translation matrices.

[0052] After calibration, the laser can be used to collect scene point cloud data, and the camera can be used to collect point cloud image data.

[0053] For the foregoing step S204, the embodiments of the present application provide an implementation of determining the excavatable region range in the excavator working scene based on the scene image data by using a target region prediction model obtained by pre-training. The scene image data is input into the target region prediction model, and the target region prediction model can output the corresponding excavatable region range for the scene image data.

[0054] In a specific implementation, during the loading operation, it is necessary to first predict the excavatable region. For this purpose, the target region prediction model is used to infer the main view image (i.e., the scene image data) to obtain the excavatable region range in the excavator working scene. For example, Figure 4 As shown in a schematic diagram of an excavatable region range, the closed polygon surrounded by black dots is the excavatable region range predicted by the target region prediction model.

[0055] In actual application, in order to accurately predict the excavatable region range by the target region prediction model, it is necessary to pre-train the target region prediction model. The embodiments of the present application provide an implementation of training the target region prediction model, as shown in the following steps A1 to A2:

[0056] Step A1, obtaining a training image set. The training image set includes a plurality of training images and excavatable region labels annotated for each training image. In actual application, a large amount of 2D image data (i.e., training images) of the excavator working scene can be collected for the cat excavator working scene, and excavatable region labels can be defined, such as Figure 5 As shown in a schematic diagram of an excavatable region label, the 2D image data is annotated according to the defined excavatable region label.

[0057] Step A2, training an initial region prediction model using the training image set until a preset training stop condition is met to obtain the target region prediction model; wherein the initial region prediction model includes an HRNet model, and the preset training stop condition includes a loss value threshold or a preset training number. In actual application, the HRNet model is selected as the backbone network (i.e., the initial region prediction model), and the backbone network is trained using the 2D image data and the annotated excavatable region labels to obtain the target region prediction model.

[0058] For the foregoing step S206, the embodiment of the present application provides an implementation of determining the candidate point cloud data from the scene point cloud data based on the excavatable region range, see the following steps B1 to B2:

[0059] Step B1, project the scene point cloud data to the plane where the excavatable region range is located, and determine the scene point cloud data located in the excavatable region range. In an implementation, according to the extrinsic parameter relationship of laser and camera calibration, all scene point cloud data whose projection points fall within the closed polygon shown in FIG. 8 are obtained, denoted as pc1, and the scene point cloud data pc1 will be used as the basis data for subsequent planning of the excavation point. Figure 4

[0060] Step B2, perform voxel downsampling processing on the scene point cloud data located in the excavatable region range to obtain the candidate point cloud data. In order to more efficiently process these point cloud data, the embodiment of the present application performs voxel downsampling on the scene point cloud data pc1 to obtain the candidate point cloud data pc2, so as to find the excavation point meeting the requirements faster. The voxel downsampling is to reduce the number of points while keeping the shape features of the point cloud basically unchanged, and basically retains the spatial structure information. In actual application, first, the point cloud space is gridded, also known as voxelized, each grid after gridding is called a voxel, and some points are contained in these divided extremely small grids, then an average or weighted average of these points is taken to obtain a point, which replaces all the points in the original grid.

[0061] For the foregoing step S208, the embodiment of the present application provides an implementation of determining the target recommended excavation point in the working scene of the excavator from the candidate point cloud data according to the preset excavator limiting parameter, see the following steps C1 to C4:

[0062] Step C1, sort the candidate point cloud data in the order of height from high to low, and determine the candidate recommended excavation point from the candidate point cloud data based on the sorting. Specifically, sort the candidate point cloud data in the order of height of each point in the point cloud data from high to low, and determine the candidate recommended excavation point from the candidate point cloud data based on the sorting. In an implementation, in order to further improve the excavation efficiency, the highest point in the excavatable region is selected each time, so the points in the candidate point cloud data pc2 are sorted in the order of z value from high to low to obtain pc3, and the point with the highest z value is selected as the candidate recommended excavation point P.

[0063] Step C2, determine the large arm pose and small arm pose corresponding to the candidate recommended excavation point based on the candidate recommended excavation point, the pre-constructed excavator large arm coordinate system and the excavator small arm coordinate system. See the following steps C2-1 to C2-3 for details:

[0064] ​Step C2-1, obtain the pre-constructed excavator boom coordinate system and the excavator arm coordinate system. In the excavating point selection process, the length of the excavator boom and arm and their joint limits need to be considered. See Figure 6 Figure 1 shows a state diagram of the zero position of the excavator, where arm_link is the arm coordinate system, book_link is the boom coordinate system, the angle between the z-axis of the boom coordinate system and the vertical direction is 1 radian, which is the zero position of the boom, and the angle between the z-axis of the arm coordinate system and the boom is 1.5 radians, which is the zero position of the arm. Assuming that the origin of the arm coordinate system is O1, the coordinates in the arm_base_link coordinate system are (x1, y1, z1), and the origin of the boom coordinate system is O2, the coordinates in the arm_base_link coordinate system are (x2, y2, z2).

[0065] Step C2-2, in the excavator boom coordinate system or the excavator arm coordinate system, determine whether the selected recommended excavating point, the origin of the excavator boom coordinate system, and the origin of the excavator arm coordinate system satisfy the triangular set relationship. The triangular set relationship is that the sum of the two sides is greater than the third side, and the difference between the two sides is less than the third side. In an embodiment, the selected recommended excavating point P is the highest point in the point cloud set pc3, and the coordinates in the arm_base_link coordinate system are (x, y, z). If the three points P, O1, and O2 can form a triangle, i.e., they satisfy the triangular set relationship that the sum of the two sides is greater than the third side and the difference between the two sides is less than the third side, then step C2-3 is executed; otherwise, the next highest point is selected as the new selected recommended excavating point P.

[0066] Step C2-3, in the case where the triangular set relationship is satisfied, determine the boom pose and arm pose corresponding to the selected recommended excavating point based on the triangular set relationship. In an embodiment, the lengths of the three sides and the angles between the three sides in the triangular set relationship are known, and the lengths of the boom and arm are known. Based on the known quantities, the boom pose and arm pose can be calculated.

[0067] Step C3, if the boom pose and arm pose corresponding to the selected recommended excavating point both satisfy the preset excavator limit parameters, then the selected recommended excavating point is determined as the target recommended excavating point in the excavator working scene. In an embodiment, after the boom pose and arm pose are calculated based on the triangular set relationship, it is determined whether the boom pose and arm pose are within the limits. If they are, it means that point P is a point that can be planned by the motion control module, and the point is returned as the target recommended excavating point.

[0068] Step C4, if the large arm pose and / or the small arm pose does not satisfy the preset limit position parameter of the excavator, a new alternative recommended digging point is determined from the alternative point cloud data based on the sorting until the large arm pose and the small arm pose corresponding to the new alternative recommended digging point both satisfy the preset limit position parameter of the excavator, and the new alternative recommended digging point is determined as the target recommended digging point in the working scene of the excavator. In an implementation, if the large and small arm poses are not within the limit position, it indicates that the point P is a point that cannot be planned by the motion control module, at which time the next highest point in the point cloud set pc3 will be continuously judged until the point satisfies the requirement. For example, refer to Figure 7 Fig. 2 shows a schematic diagram of a target recommended digging point, wherein the left drawing is the position of the target recommended digging point in the scene image data, and the right drawing is the position of the target recommended digging point in the scene point cloud data.

[0069] The excavating point selected by the excavating point recommendation method based on multi-sensor fusion provided by the embodiment of the present application can ensure that the excavating point is the highest point in the excavatable area, and the point is definitely a point that can be planned by the motion control, thereby ensuring the continuity and efficiency of loading. The embodiment of the present application can provide a high-accuracy excavating point in the loading process, thereby realizing high efficiency and high quality of loading operation. In actual application, the embodiment of the present application can continuously load n+ times, effectively reducing the labor cost, and ensuring the continuity and stability of the loading operation. The embodiment of the present application is a high-efficiency, reliable and intelligent loading scheme, which has a wide application prospect and market value.

[0070] In summary, the embodiment of the present application provides an excavating point recommendation method based on multi-sensor fusion, aiming to solve the problem of accurate excavation in automatic loading tasks, and aiming to improve the efficiency of automatic loading and the accuracy of excavating points. In order to achieve this goal, the embodiment of the present application proposes a simple and fast method, which can accurately obtain excavating point information. The method can help the excavator to excavate more accurately in the automatic loading task, thereby reducing the loading time, and thus greatly improving the loading efficiency. In addition, the method has the advantages of simple operation, fast acquisition of excavating point information, high accuracy, and effective avoidance of excavation errors and waste. In summary, the method proposed by the embodiment of the present application can bring higher efficiency and better effect to the automatic loading task.

[0071] On the basis of the foregoing embodiment, the embodiment of the present application provides an excavating point recommendation device based on multi-sensor fusion, referring to Figure 8 Fig. 3 shows a structural schematic diagram of an excavating point recommendation device based on multi-sensor fusion, which mainly includes the following parts:

[0072] The data acquisition module 802 is configured to acquire visual data of the excavator working scene; wherein the visual data comprises scene image data and scene point cloud data.

[0073] The region prediction module 804 is configured to determine a excavatable region range in the excavator working scene based on the scene image data by using a target region prediction model pre-trained, wherein the target region prediction model is trained by taking training image data as input data and taking the excavatable region range as output data.

[0074] The alternative point cloud determination module 806 is configured to determine alternative point cloud data from the scene point cloud data based on the excavatable region range.

[0075] The excavating point recommendation module 808 is configured to determine a target recommended excavating point in the excavator working scene from the alternative point cloud data according to a preset excavator limiting parameter.

[0076] The excavating point recommendation device based on multi-sensor fusion provided by the embodiment of the present application can determine the excavatable region range in the excavator working scene by using the target region prediction model, so as to ensure that the target recommended excavating point determined finally is located in the excavatable region, avoid the reduction of loading efficiency caused by excavating errors, and further avoid the problem of large prediction deviation caused by errors introduced in the excavating point position labeling process compared with the scheme based on key point detection in the related art. In addition, the above method can determine the target recommended excavating point from the alternative point cloud data according to the excavator limiting parameter, so as to ensure that the target recommended excavating point can be reached by both the large arm posture and the small arm posture of the excavator, thereby avoiding the problem of low efficiency caused by reselecting points due to too close or too far distance between the excavator and the target recommended excavating point. Therefore, the embodiment of the present application can accurately and quickly recommend the excavating point, better assist the excavator to perform the excavating operation, thereby reducing the loading time and improving the loading efficiency.

[0077] In an embodiment, the alternative point cloud determination module 806 is further configured to:

[0078] project the scene point cloud data to a plane where the excavatable region range is located, and determine the scene point cloud data located in the excavatable region range;

[0079] perform voxel down-sampling processing on the scene point cloud data located in the excavatable region range, and obtain the alternative point cloud data.

[0080] In an embodiment, the excavating point recommendation module 808 is further configured to:

[0081] sort the alternative point cloud data in descending order of height, and determine the alternative recommended excavating point from the alternative point cloud data based on the sorting;

[0082] Based on the alternative recommended digging point, the pre-constructed excavator boom coordinate system and the excavator arm coordinate system, the boom pose and the arm pose corresponding to the alternative recommended digging point are determined.

[0083] If the boom pose and the arm pose corresponding to the alternative recommended digging point both satisfy the preset excavator limiting parameter, the alternative recommended digging point is determined as the target recommended digging point in the excavator working scene.

[0084] In an embodiment, the digging point recommendation module 808 is further configured to:

[0085] obtain the pre-constructed excavator boom coordinate system and the excavator arm coordinate system;

[0086] determine whether the alternative recommended digging point, the origin of the excavator boom coordinate system and the origin of the excavator arm coordinate system satisfy a triangular set relationship under the excavator boom coordinate system or the excavator arm coordinate system;

[0087] If yes, the boom pose and the arm pose corresponding to the alternative recommended digging point are determined according to the triangular set relationship.

[0088] In an embodiment, the digging point recommendation module 808 is further configured to:

[0089] If the boom pose and / or the arm pose do not satisfy the preset excavator limiting parameter, a new alternative recommended digging point is determined from the alternative point cloud data based on the ranking until the boom pose and the arm pose corresponding to the new alternative recommended digging point both satisfy the preset excavator limiting parameter, and the new alternative recommended digging point is determined as the target recommended digging point in the excavator working scene.

[0090] In an embodiment, the data acquisition module 802 is further configured to:

[0091] acquire scene image data collected by the image acquisition device for the excavator working scene, and acquire scene point cloud data collected by each point cloud acquisition device for the excavator working scene.

[0092] In an embodiment, further comprising a model training module configured to:

[0093] acquire a training image set; wherein the training image set comprises a plurality of training images and a label of a excavatable region annotated for each training image;

[0094] train an initial region prediction model using the training image set until a preset training stop condition is satisfied to obtain a target region prediction model; wherein the initial region prediction model comprises an HRNet model, and the preset training stop condition comprises a loss value threshold or a preset training number.

[0095] The device provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiment, and for brief description, the part of the device embodiment not mentioned can be referred to the corresponding content in the foregoing method embodiment.

[0096] The embodiment of the present application provides an electronic device, specifically, the electronic device comprises a processor and a storage device; the storage device stores a computer program, and the computer program executes the following when being run by the processor:

[0097] A mining point recommendation method based on multi-sensor fusion, comprising:

[0098] Obtaining visual data of a mining machine working scene; wherein the visual data comprises scene image data and scene point cloud data;

[0099] Determining a mineable region range in the mining machine working scene based on the scene image data through a target region prediction model obtained by pre-training; the target region prediction model is obtained by training with training images as input data and the mineable region range as output data;

[0100] Determining candidate point cloud data from the scene point cloud data based on the mineable region range;

[0101] Determining a target recommended mining point in the mining machine working scene from the candidate point cloud data according to preset mining machine limiting parameters.

[0102] In an embodiment, the step of determining candidate point cloud data from the scene point cloud data based on the mineable region range comprises:

[0103] Projecting the scene point cloud data to a plane where the mineable region range is located to determine the scene point cloud data located in the mineable region range;

[0104] Performing voxel downsampling processing on the scene point cloud data located in the mineable region range to obtain candidate point cloud data.

[0105] In an embodiment, the step of determining a target recommended mining point in the mining machine working scene from the candidate point cloud data according to preset mining machine limiting parameters comprises:

[0106] Sorting the candidate point cloud data in order of height from high to low, and determining candidate recommended mining points from the candidate point cloud data based on the sorting;

[0107] Determining a large arm pose and a small arm pose corresponding to the candidate recommended mining points based on the candidate recommended mining points, a large arm coordinate system and a small arm coordinate system of the mining machine constructed in advance;

[0108] If the large arm pose and the small arm pose corresponding to the alternative recommended digging point both satisfy the preset excavator limiting parameters, the alternative recommended digging point is determined as the target recommended digging point in the excavator working scene.

[0109] In an embodiment, based on the alternative recommended digging point, a pre-constructed excavator large arm coordinate system and an excavator small arm coordinate system, the step of determining the large arm pose and the small arm pose corresponding to the alternative recommended digging point comprises:

[0110] obtaining a pre-constructed excavator large arm coordinate system and an excavator small arm coordinate system;

[0111] In the excavator large arm coordinate system or the excavator small arm coordinate system, it is judged whether a triangular set relationship is satisfied among the alternative recommended digging point, the origin of the excavator large arm coordinate system and the origin of the excavator small arm coordinate system;

[0112] If yes, the large arm pose and the small arm pose corresponding to the alternative recommended digging point are determined according to the triangular set relationship.

[0113] In an embodiment, the method further comprises:

[0114] If the large arm pose and / or the small arm pose do not satisfy the preset excavator limiting parameters, a new alternative recommended digging point is determined from the alternative point cloud data based on the sorting until the large arm pose and the small arm pose corresponding to the new alternative recommended digging point both satisfy the preset excavator limiting parameters, and the new alternative recommended digging point is determined as the target recommended digging point in the excavator working scene.

[0115] In an embodiment, the excavator is configured with an image acquisition device and a plurality of point cloud acquisition devices; the step of obtaining visual data of the excavator working scene comprises:

[0116] obtaining scene image data collected by the image acquisition device for the excavator working scene, and obtaining scene point cloud data collected by each point cloud acquisition device for the excavator working scene.

[0117] In an embodiment, the training step of the target region prediction model comprises:

[0118] obtaining a training image set; wherein the training image set comprises a plurality of training images and a diggable region label annotated for each training image;

[0119] The initial region prediction model is trained by using the training image set until a preset training stop condition is met, and a target region prediction model is obtained; wherein the initial region prediction model includes an HRNet model, and the preset training stop condition includes a loss value threshold or a preset training number.

[0120] The electronic device provided by the embodiment of the present application determines the excavatable region range in the excavator working scene through the target region prediction model, can ensure that the finally determined target recommended digging point is located in the excavatable region, avoids the reduction of loading efficiency caused by excavation errors, and has lower annotation complexity compared with the scheme based on key point detection in the related art, and can avoid the problem that the prediction deviation is large due to the error introduced in the digging point position annotation process. In addition, the above method can determine the target recommended digging point from the alternative point cloud data according to the excavator limiting parameter, ensure that the target recommended digging point can be reached by the large arm posture and the small arm posture of the excavator, and thus can avoid the problem of low efficiency caused by reselecting the point due to the too close or too far distance between the excavator and the target recommended digging point. Therefore, the embodiment of the present application can accurately and quickly recommend the digging point, better assist the excavator to perform the excavation work, and thus reduce the loading time and improve the loading efficiency.

[0121] Figure 9 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in the figure, and the electronic device 100 includes a processor 90, a memory 91, a bus 92 and a communication interface 93. The processor 90, the communication interface 93 and the memory 91 are connected through the bus 92. The processor 90 is used to execute the executable modules stored in the memory 91, such as a computer program.

[0122] The memory 91 may contain a high-speed random access memory (RAM, Random Access Memory) and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 93 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0123] The bus 92 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 9 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0124] The memory 91 is configured to store a program, and the processor 90 is configured to execute the program after receiving an execution instruction. The method performed by the device for defining a flow process according to any one of the embodiments of the present application can be applied to the processor 90 or implemented by the processor 90.

[0125] The processor 90 can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 90 or the instruction in the form of software. The processor 90 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 91, and the processor 90 reads the information in the memory 91, and combines the hardware to complete the steps of the above method.

[0126] The computer program product of the readable storage medium provided by the embodiments of the present application includes a computer readable storage medium storing a program code, and the program code includes instructions for executing the following steps:

[0127] A mining point recommendation method based on multi-sensor fusion includes:

[0128] Obtaining visual data of the excavator working scene; wherein the visual data includes scene image data and scene point cloud data;

[0129] determine a range of a excavatable region in the working scene of the excavator based on the scene image data by using a target region prediction model pre-trained, wherein the target region prediction model is trained by using a training image as input data and a range of a excavatable region as output data;

[0130] determine candidate point cloud data from the scene point cloud data based on the range of the excavatable region;

[0131] determine a target recommended digging point in the working scene of the excavator from the candidate point cloud data according to a preset excavator limiting parameter.

[0132] In an embodiment, the step of determining candidate point cloud data from the scene point cloud data based on the range of the excavatable region comprises:

[0133] project the scene point cloud data to a plane where the range of the excavatable region is located, and determine the scene point cloud data located in the range of the excavatable region;

[0134] perform voxel down-sampling processing on the scene point cloud data located in the range of the excavatable region to obtain candidate point cloud data.

[0135] In an embodiment, the step of determining a target recommended digging point in the working scene of the excavator from the candidate point cloud data according to a preset excavator limiting parameter comprises:

[0136] sort the candidate point cloud data according to height from high to low, and determine a candidate recommended digging point from the candidate point cloud data based on the sorting;

[0137] determine a large arm pose and a small arm pose corresponding to the candidate recommended digging point based on the candidate recommended digging point, a large arm coordinate system and a small arm coordinate system of the excavator pre-constructed;

[0138] if the large arm pose and the small arm pose corresponding to the candidate recommended digging point both satisfy a preset excavator limiting parameter, determine the candidate recommended digging point as the target recommended digging point in the working scene of the excavator.

[0139] In an embodiment, the step of determining a large arm pose and a small arm pose corresponding to the candidate recommended digging point based on the candidate recommended digging point, a large arm coordinate system and a small arm coordinate system of the excavator pre-constructed comprises:

[0140] obtain a large arm coordinate system and a small arm coordinate system of the excavator pre-constructed;

[0141] Determine whether the alternative recommended digging point, the origin of the excavator boom coordinate system, and the origin of the excavator arm coordinate system satisfy a triangular set relationship under the excavator boom coordinate system or the excavator arm coordinate system.

[0142] If yes, determine the boom pose and the arm pose corresponding to the alternative recommended digging point according to the triangular set relationship.

[0143] In an embodiment, the method further comprises:

[0144] If the boom pose and / or the arm pose do not satisfy the preset excavator limit parameters, determine a new alternative recommended digging point from the alternative point cloud data based on the sorting until the boom pose and the arm pose corresponding to the new alternative recommended digging point both satisfy the preset excavator limit parameters, and determine the new alternative recommended digging point as the target recommended digging point in the excavator working scene.

[0145] In an embodiment, the excavator is configured with an image acquisition device and a plurality of point cloud acquisition devices; the step of acquiring visual data of the excavator working scene comprises:

[0146] Acquiring scene image data collected by the image acquisition device for the excavator working scene, and acquiring scene point cloud data collected by each of the point cloud acquisition devices for the excavator working scene.

[0147] In an embodiment, the training step of the target region prediction model comprises:

[0148] Acquiring a training image set; wherein the training image set comprises a plurality of training images and a diggable region label annotated for each of the training images;

[0149] Training an initial region prediction model using the training image set until a preset training stop condition is satisfied to obtain a target region prediction model; wherein the initial region prediction model comprises an HRNet model, and the preset training stop condition comprises a loss value threshold or a preset training number.

[0150] The readable storage medium provided by the embodiment of the present application can determine the excavatable area range in the excavator operation scene through the target area prediction model, can ensure that the finally determined target recommended digging point is located in the excavatable area, avoids the reduction of loading efficiency caused by excavation errors, and compared with the scheme based on key point detection in the related art, the labeling complexity of the embodiment of the present application is lower, and the problem that the prediction deviation is large caused by the error introduced in the digging point position labeling process can be avoided. In addition, the above method can determine the target recommended digging point from the alternative point cloud data according to the excavator limiting parameter, ensure that the target recommended digging point can be reached by the large arm posture and the small arm posture of the excavator, and thus the problem of low efficiency caused by reselecting the point due to too close or too far distance can be avoided. Therefore, the embodiment of the present application can accurately and quickly recommend the digging point, better assist the excavator to perform the excavation operation, thereby reducing the loading time and improving the loading efficiency.

[0151] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some technical features. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for recommending a digging point based on multi-sensor fusion, characterized in that, The application is applied to an automatic loading task, and includes the following steps: Obtaining visual data of a working scene of a excavator; wherein the visual data includes scene image data and scene point cloud data; Determining a excavatable region range in the working scene of the excavator based on the scene image data by using a target region prediction model obtained through pre-training; the target region prediction model is obtained through training of training image as input data and the excavatable region range as output data; Determining candidate point cloud data from the scene point cloud data based on projection of the scene point cloud data to a plane where the excavatable region range is located and retaining point cloud data located in the excavatable region range; Sorting the candidate point cloud data in a descending order of heights of points in the point cloud data, and determining a highest target recommended digging point in the working scene of the excavator that can be reached by the excavator based on the sorting and a preset excavator limiting parameter; The step of sorting the candidate point cloud data in a descending order of heights of points in the point cloud data and determining a highest target recommended digging point in the working scene of the excavator that can be reached by the excavator based on the sorting and a preset excavator limiting parameter includes the following steps: Sorting the candidate point cloud data in a descending order of heights, and determining a candidate recommended digging point from the candidate point cloud data based on the sorting; Determining a large-arm pose and a small-arm pose corresponding to the candidate recommended digging point based on the candidate recommended digging point, a large-arm coordinate system and a small-arm coordinate system of the excavator which are constructed in advance; If the large-arm pose and the small-arm pose corresponding to the candidate recommended digging point both satisfy a preset excavator limiting parameter, the candidate recommended digging point is determined as the target recommended digging point in the working scene of the excavator that can be reached by the excavator; The step of determining a large-arm pose and a small-arm pose corresponding to the candidate recommended digging point based on the candidate recommended digging point, a large-arm coordinate system and a small-arm coordinate system of the excavator which are constructed in advance includes the following steps: Obtaining a large-arm coordinate system and a small-arm coordinate system of the excavator which are constructed in advance; Judging whether a triangular set relationship is satisfied among the candidate recommended digging point, an origin of the large-arm coordinate system and an origin of the small-arm coordinate system in the large-arm coordinate system or the small-arm coordinate system; the triangular set relationship is that a sum of two sides is greater than a third side and a difference between the two sides is less than the third side; If yes, the large-arm pose and the small-arm pose corresponding to the candidate recommended digging point are determined according to the triangular set relationship. 2.The multi-sensor fusion based excavation point recommendation method of claim 1, wherein, The step of determining candidate point cloud data from the scene point cloud data based on projection of the scene point cloud data to a plane where the excavatable region range is located and retaining point cloud data located in the excavatable region range includes the following steps: Projecting the scene point cloud data to the plane where the excavatable region range is located to determine the scene point cloud data located in the excavatable region range; Performing voxel down-sampling processing on the scene point cloud data located in the excavatable region range to obtain the candidate point cloud data. 3.The multi-sensor fusion based excavation point recommendation method of claim 1, wherein, The method further includes the following steps: If the boom pose and / or the arm pose does not satisfy the preset limit parameters of the excavator, a new alternative recommended digging point is determined from the alternative point cloud data based on the sorting until the boom pose and the arm pose corresponding to the new alternative recommended digging point both satisfy the preset limit parameters of the excavator, and the new alternative recommended digging point is determined as a target recommended digging point reachable by the excavator in the excavator working scene. 4.The multi-sensor fusion based excavation point recommendation method of claim 1, wherein, The excavator is configured with an image acquisition device and a plurality of point cloud acquisition devices; The step of obtaining visual data of the excavator working scene comprises: Obtaining scene image data collected by the image acquisition device for the excavator working scene, and obtaining scene point cloud data collected by each point cloud acquisition device for the excavator working scene. 5.The multi-sensor fusion based excavation point recommendation method of claim 1, wherein, The training step of the target region prediction model comprises: Obtaining a training image set; wherein the training image set comprises a plurality of training images and a excavatable region label labeled for each training image; Training an initial region prediction model using the training image set until a preset training stop condition is met to obtain a target region prediction model; wherein the initial region prediction model comprises an HRNet model, and the preset training stop condition comprises a loss value threshold or a preset training number. 6.A digging point recommendation device based on multi-sensor fusion, characterized by Applied to an automatic loading task, comprising: A data acquisition module for obtaining visual data of the excavator working scene; wherein the visual data comprises scene image data and scene point cloud data; A region prediction module for determining a excavatable region range in the excavator working scene based on the scene image data through a target region prediction model obtained by pre-training; the target region prediction model is obtained by training with training images as input data and excavatable region range as output data; An alternative point cloud determination module for determining alternative point cloud data from the scene point cloud data based on projecting the scene point cloud data to a plane where the excavatable region range is located and retaining point cloud data located in the excavatable region range; A digging point recommendation module for sorting the alternative point cloud data in an order of height from high to low of each point in the point cloud data, and determining a highest target recommended digging point reachable by the excavator in the excavator working scene from the alternative point cloud data based on the sorting according to a preset limit parameter of the excavator; The digging point recommendation module is further configured to: Sort the alternative point cloud data in an order of height from high to low, and determine alternative recommended digging points from the alternative point cloud data based on the sorting; Determine a boom pose and an arm pose corresponding to the alternative recommended digging points based on the alternative recommended digging points, a pre-constructed excavator boom coordinate system and an excavator arm coordinate system; If the boom pose and the arm pose corresponding to the alternative recommended digging points both satisfy the preset limit parameters of the excavator, the alternative recommended digging points are determined as target recommended digging points in the excavator working scene; The digging point recommendation module is further configured to: Obtain a pre-constructed excavator boom coordinate system and an excavator arm coordinate system; In the excavator boom coordinate system or the excavator arm coordinate system, it is judged whether the alternative recommended digging point, the origin of the excavator boom coordinate system, and the origin of the excavator arm coordinate system satisfy a triangle set relationship; the triangle set relationship is that the sum of two sides is greater than the third side, and the difference between the two sides is less than the third side; If yes, the boom pose and the arm pose corresponding to the alternative recommended digging point are determined according to the triangle set relationship.

7. An electronic device, comprising: The processor and the memory are included, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by the processor, cause the processor to implement the method in any one of claims 1 to 5.

Images