Operating machinery and methods, controllers, systems and media for testing its operating devices

Abstract

This application discloses a working machine and a method, controller, system, and medium for detecting its working device, relating to the field of active safety perception technology. The method includes: acquiring point cloud data, operating condition data, and image data associated with the working device connected to the mechanical equipment; performing foreground segmentation of the point cloud data using a minimum cut algorithm based on a pre-constructed energy function to obtain candidate clusters of point clouds, wherein the energy function is constructed based on the operating condition data, image data, and point cloud data; extracting at least two planes of the working device from the candidate clusters of point clouds, and cropping the candidate clusters of point clouds based on the extracted planes to obtain a target point cloud; and obtaining the detection result of the working device based on the target point cloud. This application constructs an innovative two-level segmentation framework through multimodal fusion of point cloud data and image data, combined with prior knowledge of the mechanical equipment's operating conditions, enabling more accurate detection results.

Description

Technical Field

[0001] This application relates to the field of active safety sensing technology, specifically to a working machine and a method, controller, system and medium for detecting its working device. Background Technology

[0002] Many types of work machinery, such as aerial work platforms and fire trucks, are equipped with working devices (e.g., the end-cap bucket). These devices are typically used by operators to perform designated tasks or transport materials. Real-time, accurate positional sensing of these devices is crucial for achieving core functions of work machinery, such as automated material placement, collision avoidance, and precise personnel transport. It also forms the basis for overall modeling of the work machinery and is significant for improving its collision detection accuracy. Furthermore, taking boom equipment as an example, due to its weight and load, the boom undergoes bending deformation, with the end-cap bucket being the point of greatest deformation. Therefore, accurate detection and positioning of the working devices are essential for the safe control of work machinery.

[0003] However, in actual operations, the operating distance of the machinery is usually long, and the large changes in the density of the imaging resolution of the end-sensor data during boom extension, contraction and rotation pose great challenges to the detection and positioning of the operating device, which is often located at the end of the boom.

[0004] In the detection and positioning of working devices, obtaining a target point cloud of the device is of paramount importance. Currently, the main target point cloud acquisition methods include the following: 1. Use a single lidar for target detection to obtain a target point cloud. This method is easy to implement, but when the working device is partially obstructed (such as by the boom itself), the point cloud features are severely lacking, which can easily lead to detection failure.

[0005] 2. Using multiple radars for synchronous detection and performing point cloud segmentation based on fixed-rule geometric segmentation methods (such as directly fitting cylinders or cuboids). This method has poor adaptability and insufficient robustness to highly unstructured work devices (such as work buckets consisting of a base plate and surrounding openwork railings) and complex and ever-changing construction site environments.

[0006] 3. Use LiDAR and image data to fuse detection to obtain target point cloud. This method requires a large amount of point cloud data for model training, which is costly and has low generalization. In addition, it relies solely on extrinsic parameters for target localization, which is not robust and is highly sensitive to background interference or complex environments. 4. Simultaneous coarse-fine segmentation using multi-radar point cloud information is used to obtain the target point cloud. However, coarse segmentation relies heavily on general clustering algorithms (such as Euclidean clustering) and does not utilize strong prior information, resulting in a high missegmentation rate.

[0007] Therefore, a new detection device solution is needed to obtain a more accurate target point cloud. Summary of the Invention

[0008] The purpose of this application is to provide a working machine and a method, apparatus, controller and medium for detecting its working device, so as to at least partially solve the above-mentioned technical problems.

[0009] To achieve the above objectives, a first aspect of this application provides a method for detecting a work device, comprising: acquiring point cloud data, working condition data, and image data associated with the work device connected to mechanical equipment; performing foreground segmentation on the point cloud data using a minimum cut algorithm based on a pre-constructed energy function to obtain a point cloud candidate cluster, wherein the energy function is constructed based on the working condition data, the image data, and the point cloud data; extracting at least two planes of the work device from the point cloud candidate cluster, and cropping the point cloud candidate cluster based on the spatial positional relationship between the extracted planes to obtain a target point cloud; and obtaining a work device detection result based on the target point cloud. In this embodiment, the foreground segmentation of the point cloud data using the minimum cut algorithm includes: determining the Region of Interest (ROI) for the working device based on the working condition data and the kinematic model of the mechanical equipment, and cropping the ROI point cloud from the point cloud data; constructing a graph based on the energy function and the ROI point cloud to obtain a flow network for solving graph cuts, wherein the flow network is configured such that each point in the ROI point cloud is a node, adjacent nodes are connected to form edges, and the weight of each edge is determined by the energy function, and source points representing the foreground and sink points representing the background are introduced simultaneously; and processing the constructed flow network using the minimum cut algorithm to segment the point cloud candidate clusters corresponding to the foreground.

[0010] In this embodiment of the application, determining the ROI of the working device includes: determining the theoretical center point position and spatial fluctuation range of the working device based on the working condition data and the kinematic model; and constructing a three-dimensional region centered on the theoretical center point position as the ROI, wherein the size of the three-dimensional region is equal to or greater than the size value of the actual size of the working device plus the spatial fluctuation range.

[0011] In this embodiment, the energy function is configured to have: a data term, which is configured based on the theoretical center point position of the working device and the two-dimensional pixel region information in the image data, wherein the theoretical center point position is determined based on the working condition data and the kinematic model; a smoothing term, which is configured based on the inter-point distance and normal difference between adjacent points in the point cloud data; and a fusion term, which is configured based on the visual detection confidence associated with the image data and the prediction probability function associated with the point cloud data.

[0012] In this embodiment of the application, the fusion term is denoted as ,in The weighting coefficients for the fusion term. Let f(x) represent the fusion term function, and:

[0013] in, Indicates the foreground moderating strength coefficient; Indicates the background adjustment intensity coefficient; This represents the visual detection confidence level associated with the image data; Representing the A point cloud, Representing the A label for a point cloud; This represents the two-dimensional pixel region information in the image data; Indicates the measurement of point cloud Whether it belongs to the prediction probability function of the foreground, and yes Back-projected regional information into three-dimensional space.

[0014] In this embodiment of the application, the at least two planes include: a primary plane serving as the base plate of the working device; and a secondary plane relating to the connection surface of the working device relative to the mechanical equipment.

[0015] In this embodiment, extracting the dominant plane includes: randomly selecting at least three non-collinear points from the point cloud candidate cluster as samples, estimating an initial plane model based on the samples; performing plane detection, including calculating the distance from each point in the point cloud candidate cluster to the initial plane model, and if the distance is less than a preset value, marking the corresponding point as an interior point, otherwise marking it as an exterior point; if the number of interior points obtained under the initial plane model is greater than the number of interior points under a predetermined optimal model, updating the initial plane model to iteratively perform the plane detection until a preset maximum number of iterations or the proportion of interior points exceeds a given minimum proportion of interior points; and fitting multiple first planes based on the final obtained set of interior points, and determining the dominant plane from the multiple first planes based on the structural features of the working device.

[0016] In this embodiment of the application, extracting the secondary plane includes: after removing the point cloud set corresponding to the dominant plane from the point cloud candidate cluster, continuing to perform the plane detection to refit multiple second planes, and determining the secondary plane from the multiple second planes according to the spatial positional relationship of the secondary plane relative to the dominant plane and the mechanical equipment.

[0017] In this embodiment of the application, the detection result of the working device includes the target point cloud and the bounding box and plane equation of the working device obtained based on the target point cloud.

[0018] In this embodiment of the application, the mechanical equipment is a boom, and the working device is a work bucket located at the end of the boom.

[0019] A second aspect of this application provides a controller for a detection work apparatus, comprising: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and to implement any of the methods described above when executing the instructions.

[0020] A third aspect of this application provides a system for detecting a working device at the end of a boom, comprising: two lidar sensors mounted on both sides of any boom section for acquiring point cloud data associated with the working device; a vision sensor mounted on a turntable at the root of the boom for acquiring image data associated with the working device; any of the above-mentioned controllers; and a boom control system communicating with the controllers, the boom control system providing the controllers with operating condition data associated with the working device.

[0021] The fourth aspect of this application provides a work machine that includes any of the systems described above.

[0022] A fifth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform any of the methods described above.

[0023] Through the above technical solution, the embodiments of this application abandon the single, publicly available scenic spot segmentation algorithm. By multimodal fusion of point cloud data and image data, and combined with the prior knowledge of the working conditions of mechanical equipment, point cloud foreground segmentation is first performed based on the energy function associated with the working conditions data, image data, and point cloud data. Then, point cloud precise cropping is performed based on the multi-plane spatial position relationship of the working device, thus forming an innovative two-level segmentation framework, and finally obtaining high-precision and high-reliability detection results of the working device.

[0024] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a flow chart of a point cloud processing method according to an embodiment of this application; Figure 2 This illustration schematically shows a foreground segmentation process according to an embodiment of this application; Figure 3 This illustration schematically shows a process diagram for extracting the dominant plane according to an embodiment of this application; Figure 4 The schematic diagram illustrates a flowchart of an example of a boom equipment end-cap detection and positioning method according to an embodiment of this application; Figure 5 The illustration shows a schematic diagram of a boom point cloud obtained by first-level segmentation according to an embodiment of this application; Figure 6 The illustration shows a schematic diagram of a target point cloud obtained by secondary segmentation according to an embodiment of this application; Figure 7 A schematic diagram of the structure of a controller according to another embodiment of this application is shown; and Figure 8 A schematic diagram of the structure of a system according to another embodiment of this application is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0028] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0029] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0030] Figure 1 The illustration shows a schematic flowchart of a point cloud processing method according to an embodiment of this application. Figure 1 As shown, the method may include the following steps S100-S400.

[0031] Step S100: For the working device connected to the mechanical equipment, acquire point cloud data, working condition data and image data associated with the working device.

[0032] The mechanical equipment is, for example, a boom, and the working device is, for example, a work bucket at the end of the boom. Point cloud data and image data associated with the working device are collected by lidar and vision sensors (such as cameras) installed at appropriate locations on the boom, respectively; the working condition data associated with the working device can be provided in real time by the boom control system, including but not limited to the extension length of each boom, the luffing angle, the slewing angle, etc.

[0033] Step S200: Based on the pre-constructed energy function, the minimum cut algorithm is used to segment the point cloud data foreground to obtain point cloud candidate clusters.

[0034] The energy function is constructed based on the working condition data, the image data, and the point cloud data. The energy function (also called the cost function) is defined to solve for the optimal label (foreground / background). A typical energy function includes a data term and a smoothing term, but these are usually constructed based only on point cloud geometric features and general semantic information. This embodiment differs in that the energy function construction includes not only the point cloud geometric features in the point cloud information but also working condition priors regarding the working condition data and visual semantics regarding the image data. This unifies point cloud geometry, working condition priors, and visual semantics into a single optimization framework, helping to improve the accuracy of point cloud segmentation.

[0035] In a preferred embodiment, such as Figure 2 As shown, the foreground segmentation of the point cloud data using the minimum cut algorithm may include the following steps S210-S230: Step S210: Based on the working condition data and the kinematic model of the mechanical equipment, determine the Region of Interest (ROI) for the working device, and crop the ROI point cloud from the point cloud data.

[0036] Preferably, determining the region of interest (ROI) for the working device may include: determining the theoretical center point location and spatial fluctuation range of the working device based on the working condition data and the kinematic model; and constructing a three-dimensional region centered on the theoretical center point location as the ROI, wherein the size of the three-dimensional region is equal to or greater than the actual size of the working device plus the spatial fluctuation range.

[0037] Step S220: Based on the energy function and the ROI point cloud, a graph is constructed to obtain a flow network for solving graph cuts.

[0038] The flow network is configured such that each point in the ROI point cloud is a node, adjacent nodes are connected to form edges, the weight of each edge is determined by the energy function, and source points representing the foreground and sink points representing the background are introduced simultaneously.

[0039] Preferably, the energy function is configured to have: a data term configured based on the theoretical center point position of the working device and two-dimensional pixel region information in the image data, wherein the theoretical center point position is determined based on the working condition data and the kinematic model; a smoothing term configured based on the inter-point distance and normal difference between adjacent points in the point cloud data; and a fusion term configured based on the visual detection confidence associated with the image data and the prediction probability function associated with the point cloud data.

[0040] The specific design of ROI and energy function will be further described below with examples, and will not be repeated here.

[0041] Step S230: The constructed flow network is processed using the minimum cut algorithm to segment the point cloud candidate clusters corresponding to the foreground.

[0042] Back Figure 1 Continue with step S300.

[0043] Step S300: Extract at least two planes of the working device from the point cloud candidate cluster, and trim the point cloud candidate cluster based on the spatial positional relationship between the extracted planes to obtain the target point cloud.

[0044] Preferably, the at least two planes include: a primary plane serving as the base plate of the working device; and a secondary plane relating to the connection surface of the working device relative to the mechanical equipment.

[0045] More preferably, such as Figure 3 As shown, extracting the dominant plane may include the following steps S310-S340: Step S310: Randomly select at least three non-collinear points from the point cloud candidate cluster as samples, and estimate the initial plane model based on the samples.

[0046] Step S320: Perform planar detection. Specifically, calculate the distance from each point in the candidate point cloud cluster to the initial planar model. If the distance is less than a preset value, mark the corresponding point as an interior point; otherwise, mark it as an exterior point.

[0047] Step S330: Update the initial planar model. Specifically, if the number of interior points obtained under the initial planar model is greater than the number of interior points under the predetermined optimal model, then update the initial planar model to iteratively perform the planar detection until the preset maximum number of iterations or the proportion of interior points exceeds the given minimum proportion of interior points.

[0048] Step S340: Based on the final obtained set of interior points, fit multiple first planes accordingly, and based on the structural features of the working device, determine the dominant plane from the multiple first planes.

[0049] Furthermore, after extracting the dominant plane, secondary planes can be further extracted, including: after removing the point cloud set corresponding to the dominant plane from the point cloud candidate cluster, continuing to perform the plane detection to refit multiple second planes, and determining the secondary plane from the multiple second planes according to the spatial positional relationship of the secondary plane relative to the dominant plane and the equipment.

[0050] The extraction of the primary and secondary planes will be further explained in detail below with the example of the boom end bucket, but will not be repeated here.

[0051] Step S200 above abandons the assumptions about the specific shape of the working device and proposes a foreground segmentation method that uses a minimum cut algorithm (energy-minimum graph cut model) and is based on an energy function relating to the working condition data, image data, and point cloud data, thereby performing first-level segmentation of the point cloud. Building on this, step S300 further proposes a second-level segmentation based on the multi-plane spatial positional relationship of the working device. This second-level segmentation can obtain an accurate target point cloud based on a geometric analysis method that verifies the multi-plane assumption, and based on this target point cloud, accurate positioning and pose estimation of the working device can be achieved. Therefore, returning to... Figure 1 Step S400 can then be further executed.

[0052] Step S400: Based on the target point cloud, obtain the detection results of the working device.

[0053] Preferably, the detection results of the working device include the target point cloud and the bounding box and planar equation of the working device obtained based on the target point cloud. The detection results of the working device will be further described below with reference to an example of the boom end bucket, and will not be repeated here.

[0054] In summary, compared with the prior art, the embodiments of this application abandon the single, publicly available site segmentation algorithm. By multimodal fusion of point cloud data and image data, and combined with the prior knowledge of the working conditions of mechanical equipment, point cloud foreground segmentation is first performed based on the energy function associated with the working conditions data, image data, and point cloud data. Then, point cloud precise cropping is performed based on the multi-plane spatial positional relationship of the working device, thus forming an innovative two-level segmentation framework. Finally, high-precision and high-reliability positioning detection and pose estimation of the working device are achieved.

[0055] It should be noted that the above steps S100-S400 can be executed by, for example, a controller installed on the operating console of the boom equipment. This controller takes the point cloud data, working condition data, and image data associated with the work bucket as input, and ultimately outputs the point cloud after secondary segmentation. However, in practice, in addition to this controller, multiple sensors are required for data acquisition. Therefore, the following section will still use the example of the work bucket at the boom end to specifically describe the process of work bucket detection and positioning using the method of this application embodiment.

[0056] like Figure 4As shown, this example provides a robust and high-precision method for detecting and locating the end effector bucket of a boom assembly. Based on the aforementioned point cloud processing approach, this method utilizes heterogeneous fusion of multiple LiDAR and visual sensors, deeply integrating the real-time kinematic model of the boom as a priori constraint to construct a two-level processing framework: "model-guided candidate region extraction—structure-based fine separation and verification." This method eliminates the need for any targeting markers on the bucket, enabling stable output of the bucket's 3D bounding box and its 6-DOF pose with the base plate as the reference plane in complex scenarios.

[0057] Specifically, such as Figure 4 As shown, the method may include the following steps S510-S540: Step S510, multi-sensor arrangement and data preprocessing, specifically includes the following steps S511-S516 (not shown in the figure).

[0058] Step S511: According to the installation diagram, install two lidars on both sides of the first boom section of the boom assembly. Each lidar has a 70-degree conical field of view. Simultaneously, install a vision camera on the boom root turntable, with the vision direction parallel to the boom extension / retraction direction. The camera coordinate system is fixed relative to the two lidar coordinate systems. The lidar coordinate system is defined as follows: the positive X-axis points outwards along the normal boom extension direction, the Z-axis is perpendicular to the boom and upwards, and the Y-axis is determined according to the right-hand rule.

[0059] Step S512: Obtain the transformation matrix from the left radar to the boom base coordinate system using the calibration algorithm. Transformation matrix from right radar to boom base coordinate system And the coordinate transformation matrix from the camera to the boom base coordinate system. This ensures spatial synchronization of multimodal data.

[0060] Step S513: Enable PTP (Precision Time Protocol) slave mode for the radar and camera, and simultaneously enable the PTP host server within the network to ensure spatial synchronization of multimodal data.

[0061] Step S514: Acquire real-time radar point cloud data , And real-time operating data provided by the boom control system (including the extension length of each boom and the luffing angle). Among these, radar point cloud data... , Further processing is required via step S515, where the operating data is... Figure 1 The operating condition data required for step S100.

[0062] Step S515, for point cloud , After filtering and downsampling, the calibrated extrinsic parameters are used to... , Unified conversion to boom base coordinate system: , Finally, the points are merged to obtain the global point cloud. ,Right now Figure 1 The point cloud data required for step S100.

[0063] Step S516: Simultaneously acquire camera images and use a lightweight deep learning model (such as the YOLO series) to detect the 2D bounding box of the work area, obtaining rough 2D pixel region information. Then, through the prior calibration matrix... And depth estimation, by back-projecting the 2D bounding box into 3D space, to obtain region information. . and All Figure 1 The image data required for step S100 is still denoted as [image data].

[0064] Back Figure 4 Continue with step S520.

[0065] Step S520, First-level segmentation: Extraction of multimodal candidate regions.

[0066] Specifically, this first-level segmentation application Figure 2 The foreground segmentation method presented abandons the assumptions about the specific shape of the bucket and proposes a foreground segmentation method based on an energy-minimum graph cut model. This model not only considers the geometric features of the point cloud itself (such as point spacing and normals), but also embeds the spatial probability distribution derived from the boom working conditions into the energy function in the form of a univariate potential energy term. At the same time, it uniformly models the visual interest region, achieving an effective combination of data-driven and model-driven approaches, thereby robustly separating candidate point cloud clusters for the bucket in complex point clouds.

[0067] Specifically, corresponding to Figure 2 The first-level segmentation method in this example includes the following steps S521-S524 (not shown in the figure): dynamic ROI generation, graph construction, energy function design, and minimum cut solution.

[0068] Step S521, Dynamic ROI Generation. Based on real-time operating data (such as real-time boom joint angles, including luffing, slewing, and telescopic movements) and the boom kinematic model, calculate the theoretical center point position of the end-cap bucket. and the potential range of spatial fluctuations. To build a system based on this. Centered on the work bucket, a cubic region larger than the actual size of the work bucket is designated as the ROI, and this is derived from the global point cloud data. Cut out dot clouds from the middle It is used to quickly eliminate background interference at a distance.

[0069] Step S522, graph construction. Each point in the graph is a node; connecting neighboring nodes forms an edge (n-links); two virtual terminal nodes are also introduced, namely the source node (representing the foreground) and the sink node (representing the background).

[0070] Step S523, Energy Function Design. Define the total energy function. Used to solve for optimal label allocation (Foreground / Background), Energy Function It consists of data items, smoothing items, and fusion items:

[0071] Data Items : Used to assess whether a single point belongs to the foreground or background at the cost. Among them, These are the weight coefficients for the data items; The data item function is constructed based on the boom working condition calculation model, where... Representative point cloud The first in One point, Representing the Each data item is labeled with a foreground and background. The data item can be based on the theoretical center point location of the working device. and the two-dimensional pixel region information in the image data The configuration is as described above, wherein the theoretical center point position is determined based on the working condition data and the kinematic model, and the two-dimensional pixel region information... This is a visual detection result. For example, the background cost is related to the point's position relative to the theoretical center point. The horizontal distance is negatively correlated, meaning the farther the distance, the lower the cost of belonging to the background; the foreground cost, on the other hand, is determined by information related to the two-dimensional pixel region. Adjustable parameter settings, and spatial correlation between point coordinates and visual detection results (i.e. The projection area is enhanced by association.

[0072] Smoothing Term This is used to encourage spatially close points with similar geometric features to have the same label, and is configured based on the inter-point distance and normal difference between adjacent points in the point cloud data. Represents the weighting coefficients of the smoothing term. Let N represent the smoothing function for adjacent points, where N represents the set of point clouds corresponding to adjacent point clouds. For example, if the points are close in distance and have similar normals, it indicates a large penalty when the two points have different labels, forcing them to have the same label; if the points are far apart and have large differences in normals, it indicates that the two points are allowed to have different labels.

[0073] This is used to achieve direct and adaptive modulation of 3D point cloud segmentation based on visual semantic information, and to impose bidirectional constraints on the energy function. The fusion term function is as follows:

[0074] in: Foreground adjustment intensity coefficient, which controls the influence of visual information on the foreground point; Background adjustment intensity coefficient, which controls the influence of visual information on background points; Visual detection confidence, which is associated with image data, reflects the reliability of the detection. : Indicates the measurement of point cloud Whether it belongs to the prediction probability function of the foreground, and yes Back-projected regional information into three-dimensional space.

[0075] It can be seen that the strength of the entire fusion term can be dynamically adjusted by the confidence level of visual detection. When the visual detection result is reliable, the guiding effect on 3D segmentation is enhanced; when the visual detection result is unreliable (such as false detection caused by extremely poor lighting), the impact is automatically weakened, and the system degenerates into a fusion mode mainly based on laser point cloud and working condition prior.

[0076] The energy function designed above unifies point cloud geometry, working condition priors, and visual semantics into a single optimization framework. Furthermore, through... Parameters enable dynamic adjustment of the reliability of visual information, using The function transforms a discrete detection box into a continuous spatial constraint, while simultaneously having a dual effect of encouraging the foreground and penalizing the background.

[0077] Step S524, minimum cut solution. By maximizing (or minimizing) the flow on the graph, the graph is partitioned into two sets connecting the source and sink vertices respectively. The set of vertices connecting the source vertices is thus initially partitioned into candidate clusters of working bucket point clouds. ,like Figure 5 The circled area shows the cloud formations around the boom.

[0078] Back Figure 4 Continue with step S530.

[0079] Step S530, Secondary segmentation: Structural verification and fine segmentation.

[0080] Because the end-capsule and boom are closely connected in the point cloud data, there is a lot of noise in the point cloud under harsh environmental conditions. In such cases, even the minimum cut algorithm cannot completely separate the point cloud of the end-capsule from the boom. Therefore, considering the unique structure of the end-capsule—a solid base plate plus openwork railings—a two-level segmentation based on the multi-plane assumption and verification mechanism proposed in step S300 can be further considered. This two-level segmentation can obtain an accurate target end-capsule point cloud based on a geometric analysis method that verifies the multi-plane assumption. Based on this target end-capsule point cloud, accurate positioning and pose estimation of the end-capsule can be achieved.

[0081] Specifically, the secondary segmentation approach in this example includes: first, robustly extracting the dominant plane from the candidate point cloud, and determining whether it is the bottom plate plane of the bucket based on its area, height, and spatial relationship with the boom end; then, searching for secondary planes that may be related to the boom connection surface; finally, using the spatial logic rule of "located on the side of the bottom plate plane and in front of the connection surface," precisely trimming the candidate point cloud to obtain a clean bucket point cloud. Specifically, this may include the following steps S531-S538 (not shown in the figure): Step S531: Randomly select three non-collinear points in the point cloud data. These three points will serve as samples and will be used to estimate an initial planar model. Let the coordinates of the three points be... , .

[0082] Step S532: Calculate two vectors based on the three selected points. , Then, calculate the normal vector of the plane based on the two vectors. Finally, utilize any point Calculate the plane equation .

[0083] Step S533, Interior point detection. For each point... Calculate its distance to the current plane. ,if Then the point is considered If an element is an interior point, it is an exterior point; otherwise, it is an exterior point. Record the total number of interior points. Where d is the plane constant term, sqrt represents the square root function, and t represents the set distance threshold.

[0084] Step S534, update the optimal model. If the number of interior points in the current model... If the number of interior points is greater than the previously found optimal model, then update the optimal model and its corresponding set of interior points. .

[0085] Step S535: Repeat steps S432 to S435 until the maximum number of iterations is reached. Or when the proportion of interior points in a certain iteration exceeds the given minimum proportion of interior points. until.

[0086] Step S536, from Several dominant planes are extracted. Based on the structural characteristics of the work bucket, the following rule is set: the dominant plane with the largest number of point clouds, the smallest fitting residual, and the angle between the plane normal vector and the horizontal plane is less than a threshold is the bottom plate plane of the work bucket. And record its point cloud set A and unit normal vector. .

[0087] Step S537, connect the area plane detection. After removing point set A from the plane, perform dominant plane probing again. (Search for...) A plane with a certain included angle (usually large) and located close to the theoretical hinge point at the end of the boom is tentatively designated as a possible connection surface. Its point cloud collection record .

[0088] Step S538: Perform spatial logic trimming. The effective point cloud of the work bucket should meet the following requirements: located on the plane of the base plate. normal vector Pointing to one side, and for the connecting surface It should be located on the side furthest from the boom body. Definition point to plane Directed distance function The final work will be a whirlwind of challenges. It consists of points that satisfy the following conditions:

[0089]

[0090] in It is a small tolerance threshold.

[0091] Thus, through the aforementioned steps S531-S538, under-divided points located below the base plate or sandwiched between the boom and the bucket are filtered out, resulting in the following: Figure 6 The working dot cloud is shown.

[0092] Back Figure 4 Continue with step S540.

[0093] Step S540: Output the detection results.

[0094] Specifically, with the base plate plane Establish a local coordinate system for the working bucket based on the reference. In the local coordinate system, the final point set... Calculate the minimum volume orientation bounding box. This bounding box is regular in the local coordinate system, and through coordinate transformation, it can be transformed to the boom base coordinate system. At this point, the bounding box directly reflects the attitude of the bucket in the base coordinate system. In addition, by calculating the rotation of the bounding box in the local coordinate system, the yaw angle, pitch angle, roll angle, etc. of the bucket in the vehicle coordinate system can be directly calculated.

[0095] Based on this, the final detection results of the working bucket can be output, including, for example, the point cloud after secondary segmentation. 3D directional bounding box and base plate plane equation.

[0096] This example demonstrates that the method used in this application embodiment for detecting the boom end bucket has at least the following advantages: 1. By using real-time joint angles (amplitude, slewing, telescopic) and other working condition data of the boom as strong spatial priors, the search area (ROI) of the end-cap bucket is dynamically generated, greatly eliminating irrelevant background interference.

[0097] 2. A two-level segmentation mechanism based on model guidance and data-driven approach is introduced. The first-level segmentation uses the working conditions as prior information and uses graph optimization to quickly segment the foreground; the second-level segmentation obtains the accurate target working bucket point cloud based on the working bucket point cloud characteristics and a geometric analysis method based on multi-plane hypothesis verification.

[0098] 3. In the first-level segmentation stage, a unified energy minimization model is constructed that integrates laser point cloud geometric features, boom working condition priors, and visual semantic information, and the energy function is defined accordingly. Therefore, at the optimization decision level, image semantics, point cloud geometry, and working condition priors are modeled in a unified manner, realizing bidirectional, adaptive, and probabilistic coupling of 2D semantics and 3D geometry in the optimization process.

[0099] Figure 7 A schematic diagram illustrating the structure of a controller for a detection apparatus according to another embodiment of this application is shown. Figure 7 As shown, the controller includes: a memory configured to store instructions; and a processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for detecting the working device as described above. The controller can be integrated, for example, into a control system of mechanical equipment associated with the working device, or it can be a stand-alone controller.

[0100] Figure 8 A schematic diagram of a system for detecting a working device at the end of a boom according to another embodiment of this application is shown. Figure 8As shown, the system may include: two lidars installed on both sides of any section of the boom for acquiring point cloud data associated with the working device; a vision sensor installed on the root turntable of the boom for acquiring image data associated with the working device; the controller described in the above embodiment; and a boom control system communicating with the controller, the boom control system providing the controller with operating condition data associated with the working device.

[0101] In this way, the system can not only achieve deep integration of multi-view lidar and working condition model, but also fundamentally solve the self-occlusion problem under single radar view by symmetrically deploying dual radars at the base of the boom.

[0102] This application also provides a working machine that may include the system for detecting the working device at the end of the boom as described in the above embodiments. The working machine is boom equipment, including but not limited to truck cranes, excavators, loaders, mining dump trucks, concrete pump trucks, pavers, bulldozers, tunnel boring machines, port machinery, and fire trucks.

[0103] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the method of the detection apparatus described in the above embodiments.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0110] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting a working device, characterized in that, include: For the working device connected to the mechanical equipment, acquire point cloud data, working condition data and image data associated with the working device; Based on a pre-constructed energy function, a minimum cut algorithm is used to segment the point cloud data for the foreground to obtain point cloud candidate clusters, wherein the energy function is constructed based on the working condition data, the image data, and the point cloud data; At least two planes of the working device are extracted from the candidate point cloud clusters, and the candidate point cloud clusters are cropped based on the spatial positional relationship between the extracted planes to obtain the target point cloud; as well as Based on the target point cloud, the detection results of the working device are obtained.

2. The method according to claim 1, characterized in that, The step of using the minimum cut algorithm to perform foreground segmentation on the point cloud data includes: Based on the working condition data and the kinematic model of the mechanical equipment, the region of interest (ROI) for the working device is determined, and the ROI point cloud is cropped from the point cloud data. Based on the energy function and the ROI point cloud, a graph is constructed to obtain a flow network for solving graph cuts. This flow network is configured such that each point in the ROI point cloud is a node, adjacent nodes are connected to form edges, and the weight of each edge is determined by the energy function. Simultaneously, source nodes representing the foreground and sink nodes representing the background are introduced. The minimum cut algorithm is used to process the constructed flow network to segment the point cloud candidate clusters corresponding to the foreground.

3. The method according to claim 2, characterized in that, Determining the ROI of the operating device includes: Based on the operating data and the kinematic model, the theoretical center point location and spatial fluctuation range of the working device are determined; and A three-dimensional region centered on the theoretical center point is constructed as the ROI, and the size of the three-dimensional region is equal to or greater than the size of the actual size of the working device plus the spatial fluctuation range.

4. The method according to claim 2, characterized in that, The energy function is configured to have: The data item is configured based on the theoretical center point position of the working device and the two-dimensional pixel region information in the image data, wherein the theoretical center point position is determined based on the working condition data and the kinematic model. A smoothing term is configured based on the inter-point distance and normal difference between adjacent points in the point cloud data; as well as The fusion term is configured based on the visual detection confidence associated with the image data and the prediction probability function associated with the point cloud data.

5. The method according to claim 4, characterized in that, The fusion term is denoted as ,in The weighting coefficients for the fusion term. Let represent the fusion term function, and: in, Indicates the foreground moderating strength coefficient; Indicates the background adjustment intensity coefficient; This represents the visual detection confidence level associated with the image data; Representing the A point cloud, Representing the A label for a point cloud; This represents the two-dimensional pixel region information in the image data; Indicates the measurement of point cloud Whether it belongs to the prediction probability function of the foreground, and yes Back-projected regional information into three-dimensional space.

6. The method according to claim 1, characterized in that, The at least two planes include: a primary plane serving as the base plate of the working device; and a secondary plane relating to the connection surface of the working device relative to the mechanical equipment.

7. The method according to claim 6, characterized in that, Extracting the dominant plane includes: At least three non-collinear points are randomly selected from the candidate point cloud cluster as samples, and an initial planar model is estimated based on the samples. Performing planar detection includes: calculating the distance from each point in the candidate cluster of the point cloud to the initial planar model; if the distance is less than a preset value, the corresponding point is marked as an interior point; otherwise, it is marked as an exterior point. If the number of interior points obtained under the initial planar model is greater than the number of interior points under the predetermined optimal model, then the initial planar model is updated to iteratively perform the planar probing until the preset maximum number of iterations or the proportion of interior points exceeds the given minimum proportion of interior points; and Based on the final set of interior points, multiple first planes are fitted, and based on the structural features of the working device, the dominant plane is determined from the multiple first planes.

8. The method according to claim 7, characterized in that, Extracting the secondary plane includes: After removing the point cloud set corresponding to the dominant plane from the point cloud candidate cluster, the plane detection is continued to refit multiple second planes, and the secondary plane is determined from the multiple second planes according to the spatial positional relationship of the secondary plane relative to the dominant plane and the mechanical equipment.

9. The method according to claim 1, characterized in that, The detection results of the working device include the target point cloud and the bounding box and plane equation of the working device obtained based on the target point cloud.

10. The method according to any one of claims 1 to 9, characterized in that, The mechanical equipment is a boom, and the working device is a working bucket located at the end of the boom.

11. A controller for a detection operation device, characterized in that, include: The memory is configured to store instructions; as well as A processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method according to any one of claims 1 to 10.

12. A system for detecting a working device at the end of a boom, characterized in that, include: Two lidar sensors installed on either side of any section of the boom are used to collect point cloud data associated with the working device; A vision sensor mounted on the root turntable of the boom is used to acquire image data associated with the working device; The controller as claimed in claim 11; as well as A boom control system that communicates with the controller, the boom control system being used to provide the controller with operating condition data related to the working device.

13. A type of operating machinery, characterized in that, Includes the system described in claim 12.

14. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method according to any one of claims 1 to 10.

Images