Method and system for assisting in driving an excavator, electronic device and storage medium

Abstract

The application discloses a kind of method and system for assisting driving excavator, electronic equipment and storage medium. Among them, the method comprises: obtaining the multidimensional point cloud information detected by laser radar configured on excavator;From the multidimensional point cloud information, the projection coordinate information of the excavator's bucket on the ground is determined;The projection coordinate information is converted into camera coordinate system, and the position information of the ground projection of the bucket in the camera coordinate system is obtained;Wherein, the camera coordinate system is the coordinate system based on display image, and the display image is the real-time image photographed by the camera configured on the excavator;According to the position information, the projection area of the bucket in the display image is determined.The application solves the technical problems that the construction machinery has low safety and low construction efficiency during construction.

Description

Technical Field

[0001] This invention relates to the field of driver assistance technology, and more specifically, to a method and system, electronic device and storage medium for driver assistance of an excavator. Background Technology

[0002] For remote operation of construction machinery such as excavators and electric shovels, the driver typically sees a two-dimensional image transmitted from a camera mounted on the excavator. This makes it difficult to judge the distances and heights of objects in the image, or to determine the landing point of the bucket. This situation easily leads to misoperation, injuries, or damage to the excavator. Current solutions rely heavily on driving experience and luck, placing excessive demands on the driver's experience, skills, and current working conditions. Therefore, construction safety cannot be effectively guaranteed during operation, resulting in low efficiency. Summary of the Invention

[0003] This invention provides a method and system, electronic device and storage medium for assisting in driving an excavator, to at least solve the technical problems of low safety and low construction efficiency of construction machinery during construction.

[0004] According to one aspect of the present invention, a method for assisting in the operation of an excavator is provided, comprising: acquiring multi-dimensional point cloud information detected by a lidar array configured on the excavator; determining, from the multi-dimensional point cloud information, the projected coordinates of the excavator's bucket on the ground; converting the projected coordinates to a camera coordinate system to obtain position information of the bucket's ground projection in the camera coordinate system; wherein the camera coordinate system is a coordinate system based on a displayed image, the displayed image being a real-time image captured by a camera configured on the excavator; and determining, based on the position information, the projection area of ​​the bucket in the displayed image.

[0005] According to another aspect of the present invention, an apparatus for assisting in driving an excavator is also provided, comprising: an acquisition unit for acquiring multi-dimensional point cloud information detected by a lidar array configured on the excavator; a first determination unit for determining, from the multi-dimensional point cloud information, the projection coordinates of the excavator's bucket on the ground; a conversion unit for converting the projection coordinates to a camera coordinate system to obtain the position information of the bucket's ground projection in the camera coordinate system; wherein the camera coordinate system is a coordinate system based on a displayed image, and the displayed image is a real-time image captured by a camera configured on the excavator; and a second determination unit for determining, based on the position information, the projection area of ​​the bucket in the displayed image.

[0006] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described method for assisting in driving an excavator through the computer program.

[0007] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described method for assisting in driving an excavator when it is run.

[0008] In this embodiment of the invention, a method is employed to acquire multi-dimensional point cloud information detected by a lidar array mounted on an excavator; determine the projected coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information; transform the projected coordinates into a camera coordinate system to obtain the position information of the bucket's ground projection in the camera coordinate system; wherein the camera coordinate system is a coordinate system based on a display image, and the display image is a real-time image captured by a camera mounted on the excavator; and determine the projection area of ​​the bucket in the display image based on the position information. In this method, because the projected coordinates of the bucket on the ground obtained by the lidar are transformed into the real-time image obtained by the camera, the accuracy of bucket operation for assisting excavator operation is improved, thereby solving the technical problems of low safety and low construction efficiency of construction machinery during construction. Attached Figure Description

[0009] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0010] Figure 1 This is a schematic diagram of an application environment for an optional method for assisting in driving an excavator according to an embodiment of the present invention;

[0011] Figure 2 This is a schematic diagram of an application environment for another optional method for assisting in driving an excavator according to an embodiment of the present invention;

[0012] Figure 3 This is a schematic flowchart of an optional method for assisting in driving an excavator according to an embodiment of the present invention;

[0013] Figure 4 This is a schematic diagram showing the width of the bucket and arm of an optional excavator according to an embodiment of the present invention;

[0014] Figure 5This is a schematic flowchart of another optional method for assisting in driving an excavator according to an embodiment of the present invention;

[0015] Figure 6 This is a schematic diagram of an optional radar coordinate system to camera coordinate system according to an embodiment of the present invention;

[0016] Figure 7 This is a schematic diagram of the bucket projection point of an optional excavator for assisting driving, according to an embodiment of the present invention;

[0017] Figure 8 This is a schematic diagram of the bucket projection of an optional excavator for assisting in driving, according to an embodiment of the present invention;

[0018] Figure 9 This is a schematic diagram of an optional device for assisting in driving an excavator according to an embodiment of the present invention;

[0019] Figure 10 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to one aspect of the present invention, a method for assisting in the driving of an excavator is provided. Optionally, as an alternative implementation, the above-described method for assisting in the driving of an excavator may be applied to, but is not limited to, [examples of other methods]. Figure 1The application environment shown includes: a terminal device 102 for human-machine interaction with the user, a network 104, and a server 106. The user 108 can interact with the terminal device 102, which runs an application for assisting in driving the excavator. The terminal device 102 includes a human-machine interaction screen 1022, a processor 1024, and a memory 1026. The human-machine interaction screen 1022 displays real-time environmental images of the excavator's operation, as well as the projection point information of the excavator bucket on the ground. The processor 1024 acquires multi-dimensional point cloud information detected by a lidar system configured on the excavator. The memory 1026 stores the multi-dimensional point cloud information acquired by the lidar system configured on the excavator, as well as real-time images captured by a camera on the excavator.

[0023] Furthermore, server 106 includes database 1062 and processing engine 1064. Database 1062 stores the multi-dimensional point cloud information detected by the LiDAR configured on the excavator, as well as real-time images captured by the camera on the excavator. Processing engine 1064 is used to: determine the projected coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information detected by the LiDAR configured on the excavator; convert the projected coordinates to a camera coordinate system to obtain the position information of the bucket's ground projection in the camera coordinate system; wherein the camera coordinate system is a coordinate system based on the displayed image, and the displayed image is a real-time image captured by the camera configured on the excavator; determine the projection area of ​​the bucket in the displayed image based on the position information; and send the projection area to terminal device 102.

[0024] In one or more embodiments, the method described above for assisting in driving an excavator can be applied to... Figure 2 The application environment shown. For example... Figure 2 As shown, user 202 and user device 204 can interact with each other. User device 204 includes memory 206 and processor 208. In this embodiment, user device 204 can, but is not limited to, referencing and executing the operations performed by terminal device 102 to obtain the projection area of ​​the excavator bucket in the displayed image.

[0025] Optionally, the terminal device 102 and user device 204 mentioned above include, but are not limited to, mobile phones, tablets, laptops, PCs, in-vehicle electronic devices, wearable devices, and other terminals. The network 104 mentioned above may include, but is not limited to, wireless networks or wired networks. The wireless network includes Wi-Fi and other networks that enable wireless communication. The wired network may include, but is not limited to, wide area networks (WANs), metropolitan area networks (MANs), and local area networks (LANs). The server 106 mentioned above may include, but is not limited to, any hardware device capable of computing. The server may be a single server, a server cluster consisting of multiple servers, or a cloud server. The above is merely an example, and no limitations are imposed in this embodiment.

[0026] In the related technologies, the bucket recognition and depth display scheme based on binocular cameras is as follows: the determination of the three-dimensional position of the bucket is based on the point cloud of the binocular camera, but the generation of the point cloud is easily affected by ambient light; the field of view of the binocular camera is relatively narrow, and the range of the point cloud of the bucket that can be acquired is limited, which is not suitable for machinery with a large range of bucket movement.

[0027] To address the aforementioned technical problems, as an optional implementation method, such as Figure 3 As shown, this embodiment of the invention provides a method for assisting in the operation of an excavator, comprising the following steps:

[0028] S302 acquires multi-dimensional point cloud information detected by the lidar configured on the excavator.

[0029] In this embodiment of the invention, the aforementioned lidar includes a single lidar or multiple lidar modules; by setting up lidar, it is possible to acquire in real time three-dimensional point cloud information, including the excavator bucket and the robotic arm connected to the bucket. The aforementioned multi-dimensional point cloud information includes not only coordinate information in the three-dimensional coordinate system but also dimensional information regarding the reflection intensity of the point cloud.

[0030] S304, determine the projection coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information.

[0031] Specifically, this includes, but is not limited to, filtering out the remaining point cloud information (excluding the bucket) from the above-mentioned three-dimensional point cloud information using a preset processing method, retaining only the point cloud information of the bucket, and then using the coordinates of the lowest point of the bucket in the vertical direction as the projection coordinate information of the excavator's bucket on the ground.

[0032] S306, the projection coordinate information is converted to the camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the displayed image, and the displayed image is a real-time image captured by a camera configured on the excavator.

[0033] Specifically, for example, the ground projection coordinates of the excavator bucket can be converted into real-time images captured by a camera by converting the world coordinate system to the camera coordinate system.

[0034] like Figure 6 As shown, Figure 6 The right side represents the lidar coordinate system, and the left side represents the camera coordinate system, which is the coordinate system of the real-time image captured by the camera mounted on the excavator. The ground projection point cloud of the bucket can be transformed from the lidar coordinate system to the camera coordinate system through rotation and translation. The transformation from the lidar coordinate system to the camera coordinate system is a transformation from three-dimensional space to three-dimensional space. Generally, this transformation requires one translation operation and one rotation operation. It can be understood as first translating the origin of the lidar coordinate system to the position in the camera coordinate system, and then performing a coordinate system rotation to align the coordinate axes. This is expressed by formula (1) and transformation formula (2) as follows:

[0035] X cam =R(XC) (1)

[0036]

[0037] Where R represents the rotation matrix, X represents the position of point X in the lidar coordinate system, and C represents the position of the camera origin in the lidar coordinate system. cam This indicates the position of point X in the camera coordinate system.

[0038] Then, the transformation from the camera coordinate system to the pixel plane is completed by using the projection matrix K. A point X in the camera coordinate system (a point in the real 3D world) corresponds to a point x in the image plane coordinate system. The transformation from the camera coordinate system to the image plane coordinate system is to change the position (X, Y, Z) of point X in the camera coordinate system to the coordinates of point x through transformation formulas (3) and (4). That is, the transformation from the camera coordinate system to the image coordinate system (camera intrinsic parameter transformation), as follows: Figure 6 As shown, the camera coordinate system is the coordinate system of the display plane 602, in which real-time images captured by the camera can be displayed in real time.

[0039] x = PX (3)

[0040] P = K[I|0] (4)

[0041] Where P is the projection matrix, K is the camera intrinsic parameter, and I is the unit rotation matrix.

[0042] Substituting formula (4) into formula (3), x = K[I|0], X cam =K[R|-RC]X;

[0043] The final projection matrix obtained is P = K[R|t], where t = -RC. The expression for K is as follows:

[0044]

[0045] Among them, f x f y C represents the focal length of the camera in the x and y directions of the pixel plane. x C y The coordinates of the camera's optical center in the pixel plane.

[0046] K is generally called the camera intrinsic parameter, which describes the camera's internal parameters, including focal length f, the position of the principal point p, and the size ratio of pixels to the real environment, etc., which are inherent properties of the camera; R and t are called camera extrinsic parameters. R here is a rotation matrix, which can be converted into a three-dimensional rotation vector, representing the rotation angles around the x, y, and z axes, respectively. t is currently a translation vector, representing the translation amount in the x, y, and z directions, respectively.

[0047] S308, determine the projection area of ​​the bucket in the displayed image based on the location information.

[0048] Specifically, based on the ground projection coordinates of the bucket and its position information in the real-time image, the projection area of ​​the bucket in the displayed image can be determined. This includes, but is not limited to, highlighting the projection area, adding a shadow, or adding text for identification. Alternatively, extended reality (XR) technology can be used to create a human-machine interactive environment that combines real and virtual elements, allowing the operator to monitor the bucket's projection position on the ground in real time.

[0049] In this embodiment of the invention, the invention can form a projection of the bucket on the ground in the real-time image transmitted back by the camera, thereby enabling the excavator driver to better judge the distance through the bucket projection, so as to achieve multiple technical objectives such as safe operation, precise operation and scientific operation. The excavator driver has a better user experience and greatly improves the experience of the driver or operator.

[0050] In one or more embodiments, determining the projected coordinates of the excavator's bucket on the ground from the multidimensional point cloud information includes:

[0051] The multi-dimensional point cloud information of the excavator's bucket is determined from the multi-dimensional point cloud information;

[0052] The point cloud set of the lowest point in the vertical direction in the multi-dimensional point cloud information of the bucket is used as the projection coordinate information of the excavator bucket on the ground.

[0053] After obtaining the 3D point cloud containing only the bucket, the point cloud is searched downwards along the z-direction based on the (x, y) range of the bucket point cloud in the 3D coordinate system. Within the (x, y) range, the part of the point cloud with the lowest z-value is the ground projection point cloud of the bucket.

[0054] In one or more embodiments, determining the multidimensional point cloud information of the excavator's bucket from the multidimensional point cloud information includes:

[0055] From the multidimensional point cloud information, a mixed multidimensional point cloud information including the excavator's bucket and the excavator arm connected to the bucket is determined;

[0056] Based on the relationship between the widths of the bucket and the excavator arm, the multidimensional point cloud information of the bucket is separated from the hybrid multidimensional point cloud information.

[0057] In embodiments of the present invention, such as Figure 4 As shown, for example, the bucket width is 2 meters and the excavator arm width is 0.5 meters. Based on the width ratio between the bucket and the excavator arm, the multidimensional point cloud information of the bucket can be separated from the mixed multidimensional point cloud information.

[0058] In one or more embodiments, in a construction scenario involving sand and dust, the multidimensional point cloud information further includes the reflection intensity information of the point cloud, and the method further includes:

[0059] Points in the multidimensional point cloud with a reflection intensity less than a preset threshold are identified as dust points; dust points in the multidimensional point cloud are removed to obtain the first reference point cloud information.

[0060] Specifically, in excavation scenarios, large amounts of dust often fill the air, interfering with the detection of the excavator bucket. To address this issue, intensity filtering is used to remove dust. The point cloud information acquired by the lidar is supplemented with reflection intensity dimension information; that is, the lidar point cloud contains four dimensions: (x, y, z, I). The last dimension, I, represents the reflection intensity of a point, and dust generally has low reflection intensity. Therefore, a threshold is set based on the dust reflection intensity. If the reflection intensity of a point is less than the threshold, that point is considered dust and filtered out from the original data, yielding the first reference point cloud information.

[0061] Determining the multidimensional point cloud information of the excavator's bucket from the multidimensional point cloud information includes: determining the multidimensional point cloud information of the excavator's bucket from the first reference point cloud information.

[0062] Specifically, in a construction scenario involving sand and dust, the multi-dimensional point cloud information of the excavator's bucket is determined by filtering the three-dimensional point cloud information obtained from the sand and dust.

[0063] In one or more embodiments, the method for assisting in driving an excavator further includes: downsampling the multidimensional point cloud through a voxel filter to obtain second reference point cloud information;

[0064] Determining the multidimensional point cloud information of the excavator's bucket from the multidimensional point cloud information includes: determining the multidimensional point cloud information of the excavator's bucket from the second reference point cloud information.

[0065] Specifically, to reduce the complexity of calculating the bucket projection point cloud and improve system response time, without affecting the overall shape of the point cloud, the multidimensional point cloud is downsampled using a voxel filter to reduce the number of points. The multidimensional point cloud information of the excavator's bucket is then determined from the downsampled second reference point cloud information, thus reducing the point cloud information processing time.

[0066] In one or more embodiments, the method for assisting in driving an excavator further includes:

[0067] The second reference point cloud information is processed by a preset ground filtering method to obtain the third reference point cloud information.

[0068] Here, the preset ground filtering methods include asymptotic morphological ground filtering methods and cloth filtering methods to filter out point cloud information containing ground information, thereby enabling further acquisition of point cloud information of the excavator bucket.

[0069] Each point in the third reference point cloud information is clustered based on Euclidean distance to obtain multiple point cloud clusters;

[0070] The point cloud clusters whose distance is within a preset threshold among the plurality of point cloud clusters are identified as the target point cloud clusters;

[0071] Determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information includes:

[0072] Based on the relationship between the width of the bucket and the excavator arm, the multidimensional point cloud information of the bucket is separated from the target point cloud cluster.

[0073] Specifically, the entire bucket identification is based on the LiDAR coordinate system. In this embodiment, the LiDAR coordinate system includes, but is not limited to, the world coordinate system, with the excavator origin being the origin of the LiDAR coordinate system (0, 0, 0). The distance from the center of each clustered point cloud to the origin is calculated, and the closest point cloud cluster is selected. If the distance from this point cloud cluster to the origin is within a preset range (the combined length of the excavator arm and bucket), the current point cloud cluster is determined to contain both the bucket and the excavator arm. Then, based on the relationship between the widths of the bucket and the excavator arm, the multi-dimensional point cloud information of the bucket is separated from the point cloud cluster containing both the bucket and the excavator arm.

[0074] In one or more embodiments, the method for assisting in driving an excavator further includes:

[0075] Using the RANSAC algorithm, vehicles in the third reference point cloud information are filtered to obtain the fourth reference point cloud information;

[0076] The step of clustering each point in the third reference point cloud information based on Euclidean distance to obtain multiple point cloud clusters includes: clustering each point in the fourth reference point cloud information based on Euclidean distance to obtain multiple point cloud clusters.

[0077] Specifically, the RANSAC algorithm is used to extract the plane with the largest area parallel to the Z-axis. The length, width, and number of point clouds of the plane are analyzed. If the length, width, and number of point clouds all exceed the threshold, the plane can be filtered out, that is, planes containing engineering vehicles can be removed.

[0078] In one or more embodiments, determining the projection area of ​​the bucket in the displayed image based on the location information includes:

[0079] The pixel region of the excavator in the displayed image is determined based on the location information;

[0080] The pixel region is marked on the display image to generate the projection of the excavator bucket onto the ground in the display image.

[0081] like Figure 7 and Figure 8 As shown, Figure 7 In the image, the pixel area of ​​the black circular region on the ground is used as the projection of the excavator bucket onto the ground in the image. Figure 8 In the image shown, the shadowed area inside the bucket of the construction vehicle below the excavator is the projection of the bucket onto the ground. At this time, the excavator can directly load and unload the excavated soil into the construction vehicle.

[0082] In one embodiment, the determined projection area is obtained based on pixel-level depth matching. Therefore, the technical solution of the present invention also has outstanding advantages such as high accuracy, high data processing precision, and strong reliability. Furthermore, it has broader applicability to scenarios involving remote operation of excavators and other large-scale mechanical equipment controlled based on video feeds. In some embodiments of the present invention, marking pixel areas on the real-time image acquired by the camera includes: marking pixel areas on the real-time image in the video using augmented reality.

[0083] Based on the above embodiments, such as Figure 5 As shown in one application embodiment, the bucket identification process in the above-described method for assisting in driving an excavator includes the following steps:

[0084] 1. Acquire multi-dimensional point cloud information collected by lidar.

[0085] 2. Utilizing intensity filtering to remove dust. Specifically, in excavation scenarios, a large amount of dust often fills the air, interfering with the detection of the excavator bucket. To address this issue, intensity filtering is used to remove dust. The point cloud information acquired by the lidar is supplemented with reflection intensity dimension information; that is, the lidar point cloud contains four dimensions: (x, y, z, I). The last dimension, I, represents the reflection intensity of a point, and dust generally has low reflection intensity. Therefore, a threshold is set based on the dust reflection intensity. If the reflection intensity of a point is less than the threshold, that point is considered dust and filtered out from the original data, yielding the first reference point cloud information.

[0086] 3. Based on distance segmentation, extract the point cloud containing the bucket and the excavator arm.

[0087] 4. Use plane fitting and surface fitting to eliminate ground and engineering vehicles; use voxel filtering to downsample the point cloud, reduce the number of points without affecting the overall shape of the point cloud, reduce the time of subsequent point cloud processing, and separate the bucket and excavator arm through point cloud processing methods such as region growing.

[0088] 5. The output only contains the point cloud of the bucket.

[0089] The embodiments of the present invention also have the following beneficial effects:

[0090] The embodiments of the present invention utilize lidar, which has a wider field of view and higher detection accuracy than binocular cameras, enabling it to detect buckets over a larger area with higher precision. By fusing lidar point clouds with images, the undulations of the ground around the bucket's landing point can be effectively displayed, providing precise assistance to the driver.

[0091] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0092] According to another aspect of the present invention, an apparatus for implementing the above-described method for assisting in driving an excavator is also provided. For example... Figure 9 As shown, the device includes:

[0093] Acquisition unit 902 is used to acquire multi-dimensional point cloud information detected by the lidar configured on the excavator;

[0094] The first determining unit 904 is used to determine the projection coordinate information of the excavator's bucket on the ground from the multi-dimensional point cloud information;

[0095] The conversion unit 906 is used to convert the projection coordinate information to the camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the display image, and the display image is a real-time image captured by a camera configured on the excavator;

[0096] The second determining unit 908 is used to determine the projection area of ​​the excavator in the displayed image based on the location information.

[0097] In this embodiment of the invention, the following method was adopted.

[0098] In one or more embodiments, the first determining unit 904 includes:

[0099] The first determining module is used to determine the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information;

[0100] The second determining module is used to take the set of point clouds of the lowest point in the vertical direction in the multi-dimensional point cloud information of the bucket as the projection coordinate information of the bucket of the excavator on the ground.

[0101] In one or more embodiments, the first determining module includes:

[0102] The first determining subunit is used to determine, from the multidimensional point cloud information, a mixed multidimensional point cloud information including the excavator's bucket and the excavator arm connected to the bucket;

[0103] The first separation subunit is used to separate the multidimensional point cloud information of the bucket from the mixed multidimensional point cloud information based on the relationship between the widths of the bucket and the excavator arm.

[0104] In one or more embodiments, in a construction scenario involving sand and dust, the multidimensional point cloud information further includes point cloud reflection intensity information, and the device for assisting in driving the excavator further includes:

[0105] The third determining unit is used to determine points in the multidimensional point cloud whose reflection intensity is less than a preset threshold as dust points.

[0106] A removal unit is used to remove dust points from the multidimensional point cloud to obtain first reference point cloud information;

[0107] The first determining module further includes:

[0108] The second determining subunit is used to determine the multi-dimensional point cloud information of the excavator's bucket from the first reference point cloud information.

[0109] In one or more embodiments, the device for assisting in driving an excavator further includes: a downsampling unit, used to downsample the multidimensional point cloud through a voxel filter to obtain second reference point cloud information;

[0110] The first determining module includes:

[0111] The third determining subunit is used to determine the multi-dimensional point cloud information of the excavator's bucket from the second reference point cloud information.

[0112] In one or more embodiments, the device for assisting in driving an excavator further includes:

[0113] The first filtering unit is used to process the second reference point cloud information through a preset ground filtering method to obtain the third reference point cloud information.

[0114] Clustering unit, used to cluster each point in the third reference point cloud information based on Euclidean distance to obtain multiple point cloud clusters;

[0115] The fourth determining unit is used to determine the point cloud clusters among the plurality of point cloud clusters whose distance is within a preset threshold as the target point cloud cluster;

[0116] The first determining module further includes:

[0117] The separation subunit is used to separate the multi-dimensional point cloud information of the bucket from the target point cloud cluster based on the relationship between the width of the bucket and the width of the excavator arm.

[0118] In one or more embodiments, the device for assisting in driving an excavator further includes:

[0119] The second filtering unit is used to filter the vehicles in the third reference point cloud information using the RANSAC algorithm to obtain the fourth reference point cloud information.

[0120] The clustering unit includes:

[0121] The clustering module is used to cluster each point in the fourth reference point cloud information based on Euclidean distance to obtain multiple point cloud clusters.

[0122] In one or more embodiments, the second determining unit 908 includes:

[0123] The third determining module is used to determine the pixel region of the excavator in the displayed image based on the location information;

[0124] A marking module is used to mark the pixel region on the display image to generate the projection of the excavator bucket onto the ground in the display image.

[0125] In one or more embodiments, the device includes a lidar and a camera configured on the excavator, as well as the aforementioned means for assisting in driving the excavator.

[0126] In one or more embodiments, a system for assisting in driving an excavator is also provided, including a lidar and a camera configured on the excavator, and further including a bucket identification and projection point search module based on lidar point clouds and a lidar and camera fusion module.

[0127] According to another aspect of the present invention, an electronic device for implementing the above-described method for assisting in driving an excavator is also provided, the electronic device being... Figure 10 The terminal device or server shown. This embodiment uses this electronic device as an example for illustration. Figure 10 As shown, the electronic device includes a memory 1002 and a processor 1004. The memory 1002 stores a computer program, and the processor 1004 is configured to execute the steps of any of the above method embodiments via the computer program.

[0128] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0129] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0130] S1, acquire multi-dimensional point cloud information detected by the lidar configured on the excavator;

[0131] S2, determine the projection coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information;

[0132] S3, the projection coordinate information is converted to the camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the displayed image, and the displayed image is a real-time image captured by a camera configured on the excavator;

[0133] S4, determine the projection area of ​​the bucket in the displayed image based on the location information.

[0134] Alternatively, as those skilled in the art will understand, Figure 10The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 10 The different configurations shown.

[0135] The memory 1002 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and apparatus for assisting in driving an excavator in this embodiment of the invention. The processor 1004 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002, thereby implementing the aforementioned method for assisting in driving an excavator. The memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1002 may further include memory remotely located relative to the processor 1004, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1002 may be used, but is not limited to, to store information such as a set of target intra-frame prediction modes. As an example, such as... Figure 10 As shown, the memory 1002 may include, but is not limited to, the acquisition unit 902, the first determination unit 904, the conversion unit 906, and the second determination unit 908 in the device for assisting in driving an excavator. Furthermore, it may include, but is not limited to, other module units in the device for assisting in driving an excavator, which will not be described in detail in this example.

[0136] Optionally, the transmission device 1006 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1006 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1006 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0137] In addition, the aforementioned electronic device also includes: a display 1008 for displaying the projection of the excavator's bucket on the ground; and a connection bus 1010 for connecting the various module components in the aforementioned electronic device.

[0138] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0139] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for assisting in driving an excavator described above, wherein the computer program is configured to perform the steps of any of the method embodiments described above when running.

[0140] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0141] S1, acquire multi-dimensional point cloud information detected by the lidar configured on the excavator;

[0142] S2, determine the projection coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information;

[0143] S3, the projection coordinate information is converted to the camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the displayed image, and the displayed image is a real-time image captured by a camera configured on the excavator;

[0144] S4, determine the projection area of ​​the bucket in the displayed image based on the location information.

[0145] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0146] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0148] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for assisting in the operation of an excavator, characterized in that, include: Acquire multi-dimensional point cloud information detected by the lidar configured on the excavator; Determining the projected coordinates of the excavator's bucket on the ground from the multi-dimensional point cloud information includes: determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information; and using the set of point clouds at the lowest point in the vertical direction in the multi-dimensional point cloud information of the bucket as the projected coordinates of the excavator's bucket on the ground. The projection coordinate information is converted into the camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the display image, and the display image is a real-time image captured by a camera configured on the excavator; The projection area of ​​the excavator in the displayed image is determined based on the location information; The step of determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information includes: From the multidimensional point cloud information, a mixed multidimensional point cloud information including the excavator's bucket and the excavator arm connected to the bucket is determined; Based on the relationship between the widths of the bucket and the excavator arm, the multidimensional point cloud information of the bucket is separated from the hybrid multidimensional point cloud information.

2. The method according to claim 1, characterized in that, In construction scenarios involving sand and dust, the multidimensional point cloud information also includes the reflection intensity information of the point cloud, and the method further includes: Points in the multidimensional point cloud with a reflection intensity less than a preset threshold are identified as dust points; By removing the dust points from the multidimensional point cloud, the first reference point cloud information is obtained; Determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information includes: The multi-dimensional point cloud information of the excavator's bucket is determined from the first reference point cloud information.

3. The method according to claim 1, characterized in that, The method further includes: performing a downsampling operation on the multidimensional point cloud through a voxel filter to obtain second reference point cloud information; Determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information includes: The multi-dimensional point cloud information of the excavator's bucket is determined from the second reference point cloud information.

4. The method according to claim 3, characterized in that, The method further includes: The second reference point cloud information is processed by a preset ground filtering method to obtain the third reference point cloud information; Each point in the third reference point cloud information is clustered based on Euclidean distance to obtain multiple point cloud clusters; The point cloud clusters whose distance is within a preset threshold among the plurality of point cloud clusters are identified as the target point cloud clusters; Determining the multi-dimensional point cloud information of the excavator's bucket from the multi-dimensional point cloud information includes: Based on the relationship between the width of the bucket and the excavator arm, the multidimensional point cloud information of the bucket is separated from the target point cloud cluster.

5. The method according to claim 4, characterized in that, The method further includes: Using the RANSAC algorithm, vehicles in the third reference point cloud information are filtered to obtain the fourth reference point cloud information; The step of clustering each point in the third reference point cloud information based on Euclidean distance to obtain multiple point cloud clusters includes: Each point in the fourth reference point cloud information is clustered based on Euclidean distance to obtain multiple point cloud clusters.

6. The method according to claim 1, characterized in that, Determining the projection area of ​​the bucket in the displayed image based on the location information includes: The pixel region of the excavator in the displayed image is determined based on the location information; The pixel region is marked on the display image to generate the projection of the excavator bucket onto the ground in the display image.

7. A device for assisting in the operation of an excavator, characterized in that, include: The acquisition unit is used to acquire multi-dimensional point cloud information detected by the lidar configured on the excavator; The first determining unit is used to determine the projection coordinate information of the excavator bucket on the ground from the multi-dimensional point cloud information, including: determining the multi-dimensional point cloud information of the excavator bucket from the multi-dimensional point cloud information; and taking the point cloud set of the lowest point in the vertical direction in the multi-dimensional point cloud information of the bucket as the projection coordinate information of the excavator bucket on the ground. A conversion unit is used to convert the projection coordinate information to a camera coordinate system to obtain the position information of the ground projection of the bucket in the camera coordinate system; wherein, the camera coordinate system is a coordinate system based on the displayed image, and the displayed image is a real-time image captured by a camera configured on the excavator; The second determining unit is used to determine the projection area of ​​the bucket in the displayed image based on the location information; Specifically, the first determining unit is used to determine, from the multidimensional point cloud information, a mixed multidimensional point cloud information including the excavator bucket and the excavator arm connected to the bucket; and to separate the multidimensional point cloud information of the bucket from the mixed multidimensional point cloud information based on the relationship between the widths of the bucket and the excavator arm.

8. A system for assisting in the operation of an excavator, characterized in that, It includes a lidar and camera mounted on the excavator, and a device for assisting in driving the excavator as described in claim 7.

Images