Obstacle matching determination method, cleaning device, storage medium and program product

Abstract

Embodiments of the present application provide an obstacle matching judgment method, a cleaning device, a storage medium and a program product, which belong to the technical field of cleaning devices. The method comprises: obtaining point cloud cluster data related to an obstacle and a pre-established obstacle list, the point cloud cluster data comprising segmented point cloud clusters; obtaining at least one candidate pair of each point cloud cluster and an obstacle in the obstacle list based on the point cloud cluster data and the obstacle list; extracting a multi-dimensional feature vector of each candidate pair, wherein the multi-dimensional feature vector comprises a semantic category feature, a spatial position relationship feature, a color feature and a point cloud structure difference feature of the candidate pair; matching the candidate pairs based on a pre-trained machine learning model and the multi-dimensional feature vector of each candidate pair to obtain a matching probability of each candidate pair; and judging whether there is a successfully matched candidate pair based on the matching probability. The method is used to improve the recognition accuracy of the robot for the obstacle.

Description

Technical Field

[0001] This invention relates to the field of cleaning equipment technology, and more specifically to an obstacle matching and judgment method, cleaning equipment, storage medium, and program product. Background Technology

[0002] Household robotic vacuum cleaners are typically equipped with multiple sensors to detect obstacles in the surrounding environment. During the robot's continuous movement, it needs to determine whether a cluster of point clouds in the image acquired by the current sensors belongs to a type of obstacle that has been recorded. The result of this determination directly affects the robot's accuracy in identifying obstacles and its ability to continuously track them.

[0003] Point cloud clusters contain rich information; however, existing obstacle detection methods for robotic vacuum cleaners only utilize a portion of this data. For example, when identifying obstacles, they only use three features: the position, distance, and category relationship between the point cloud cluster and existing obstacles, resulting in low feature utilization. Furthermore, existing methods use a fixed distance threshold when comparing and matching point cloud clusters with existing obstacles, such as a uniform distance threshold. This leads to situations where the distance is underestimated for large obstacles and overestimated for small obstacles, easily causing misidentification.

[0004] Therefore, existing methods suffer from at least the drawbacks of low feature utilization and difficulty in generalizing fixed thresholds, resulting in low accuracy of robot obstacle recognition. Summary of the Invention

[0005] The purpose of this invention is to provide an obstacle matching and judgment method, cleaning equipment, storage medium, and program product to improve the accuracy of robot obstacle recognition.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide an obstacle matching and judgment method, the method comprising: acquiring point cloud cluster data related to obstacles and a pre-established obstacle list, the point cloud cluster data including segmented point cloud clusters; obtaining at least one candidate pair between each point cloud cluster and an obstacle in the obstacle list based on the point cloud cluster data and the obstacle list; extracting a multidimensional feature vector for each candidate pair, wherein the multidimensional feature vector includes semantic category features, spatial position relationship features, color features, and point cloud structure difference features of the candidate pair; matching the candidate pairs based on a pre-trained machine learning model and the multidimensional feature vector of each candidate pair to obtain a matching probability for each candidate pair; and determining whether there is a successfully matched candidate pair based on the matching probability.

[0007] Optionally, the machine learning model is a gradient boosting decision tree model, and the training method of the gradient boosting decision tree model includes: obtaining sample candidate pairs for training the model, and labeling each sample candidate pair with a binary classification label, the binary classification label including positive samples and negative samples; extracting multi-dimensional feature vectors of the sample candidate pairs; setting an initial value for the cumulative predicted value, the initial value being the log odds value of the proportion of positive samples in the sample candidate pair; and iteratively training the gradient boosting decision tree model using the gradient boosting method based on the multi-dimensional feature vectors of the sample candidate pairs and the cumulative predicted value, to obtain the trained gradient boosting decision tree model.

[0008] Optionally, when iteratively training the gradient boosting decision tree model using the gradient boosting method, the following processing is performed for each training round: using the binary cross-entropy as the loss function, the negative gradient between the current model prediction value and the binary label is calculated, and the negative gradient is used as the residual; using the residual as the fitting target, a new decision tree is trained; at each node of the new decision tree, candidate features and their corresponding candidate split points are traversed, and splitting is performed based on the splitting method that maximizes the decrease in the loss function, until a preset stopping condition is met; and the newly trained decision tree is multiplied by the learning rate and added to the gradient boosting decision tree model obtained in the previous training round, and the cumulative prediction value is updated to obtain the gradient boosting decision tree model after this training round, wherein the candidate features include all features of the multidimensional feature vector, and the preset stopping condition includes the maximum tree depth or the minimum number of leaf node samples.

[0009] Optionally, the training method of the gradient boosting decision tree model further includes performing the following processing before starting iterative training: obtaining the number of feature vectors of the multidimensional feature vector; comparing the number of feature vectors with a preset number threshold and selecting the smaller value; and setting the number of decision trees in the gradient boosting decision tree model to the smaller value.

[0010] Optionally, if the machine learning model is a gradient boosting decision tree model, then matching the candidate pairs based on the pre-trained machine learning model and the multi-dimensional feature vector of each candidate pair to obtain the matching probability of each candidate pair includes performing the following processing on each candidate pair: inputting the multi-dimensional feature vector of the candidate pair into the pre-trained gradient boosting decision tree model, the gradient boosting decision tree model including multiple decision trees; for each decision tree: starting from the root node, judging based on the splitting condition of the feature vector corresponding to each node of the decision tree, and judging based on the judgment result to enter the next node, until reaching the leaf node of the decision tree to obtain the score value output by the decision tree; and summing the scores of each decision tree to obtain the matching probability of each candidate pair.

[0011] Optionally, determining whether a successfully matched candidate pair exists based on the matching probability includes: converting the matching probability of each candidate pair to obtain a fusion probability value, such that the fusion probability value is within the (0,1) interval; sorting all candidate pairs based on the fusion probability value, and taking the candidate pair corresponding to the maximum fusion probability value as the matching candidate pair; and comparing the fusion probability value of the matching candidate pair with a preset probability threshold. If the fusion probability value of the matching candidate pair is greater than the preset probability threshold, it is determined that a successfully matched candidate pair exists, and the matching candidate pair is taken as the successfully matched candidate pair.

[0012] Optionally, the method further includes: if the fusion probability value of the matching candidate pair is less than or equal to the preset probability threshold, it is determined that there is no successfully matched candidate pair, and an obstacle corresponding to the point cloud cluster is created in the obstacle list.

[0013] Optionally, the spatial positional relationship features include at least one of the following features: the three-dimensional centroid distance, horizontal projection center distance, height difference, size ratio, two-dimensional bounding box intersection-union ratio, and whether the center point of the point cloud cluster is located within the area occupied by the obstacle in the candidate pair.

[0014] Optionally, the color feature includes a statistical distance metric between the point cloud cluster in the candidate pair and the obstacle, the statistical distance metric including at least one of the following features: histogram correlation, chi-square distance, histogram intersection, and Bach distance between the color histograms of the point cloud cluster and the obstacle.

[0015] Optionally, the point cloud structure difference feature includes the difference between the point cloud cluster in the candidate pair and the point cloud structure feature of the obstacle, and the point cloud structure feature includes at least one of the following features: point cloud spatial dispersion, point cloud density, average height, curvature, and plane roughness.

[0016] Optionally, the semantic category features include the semantic categories of the point cloud cluster and the obstacle in the candidate pair, wherein the semantic category of the point cloud cluster is obtained by: performing coordinate transformation on the three-dimensional points in the point cloud cluster; projecting the coordinate-transformed three-dimensional points onto the pixel plane of the RGB image to obtain the two-dimensional pixel coordinates corresponding to the three-dimensional points; and performing target detection on the RGB image corresponding to the point cloud cluster based on a pre-trained visual model, and associating the target detection result with the two-dimensional pixel coordinates to obtain the semantic category of the point cloud cluster.

[0017] Secondly, embodiments of the present invention provide a cleaning device, the cleaning device comprising: a cleaning device body; and a control module, the control module being configured to execute the method described herein, enabling the cleaning device body to perform obstacle matching judgment.

[0018] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing instructions that, when executed by a processor, implement the method described thereon.

[0019] Fourthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described.

[0020] By employing the aforementioned technical solution, the rich information from point cloud clusters and existing obstacles is fully utilized during obstacle matching and judgment, improving feature utilization and effectively enhancing the accuracy of obstacle matching and judgment. After obtaining multi-dimensional feature vectors, a pre-trained machine learning model is used to match candidate pairs and output matching probabilities, enabling adaptation to obstacles of different sizes and categories, thus enhancing the generalization ability of matching judgment. The method provided by this invention overcomes the shortcomings of low feature utilization and difficulty in generalization with fixed thresholds in existing technologies. By fully utilizing the multi-dimensional features between point cloud clusters and existing obstacles, it achieves more accurate, robust, and continuously improving matching judgment, significantly improving the obstacle recognition accuracy of robotic vacuum cleaners.

[0021] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the obstacle matching and judgment method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the decision tree structure provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of the control module of the cleaning equipment provided in an embodiment of the present invention.

[0023] Explanation of reference numerals in the attached figures 201 Processor 202 Memory 203 Communication Interface 204 Bus Detailed Implementation The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0024] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0025] Figure 1 This is a schematic diagram of the obstacle matching and judgment method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps S100 to S500.

[0026] Step S100: Obtain point cloud cluster data related to obstacles and a pre-established list of obstacles, wherein the point cloud cluster data includes segmented point cloud clusters.

[0027] During robot movement, onboard sensors, such as binocular depth sensors, acquire depth images. These depth images are then converted into point clouds representing the robot's perspective. These point clouds are segmented and extracted to obtain point cloud clusters, each representing an obstacle to be matched and judged. For each frame of a depth image, the segmented point cloud clusters constitute the point cloud cluster data for that frame. The robot's perspective can be understood as the viewpoint observed from the robot's center, which is assumed to be the ground.

[0028] The obstacle list is a pre-built list that stores information about each obstacle object, including its category, location, and size. The robot's system maintains this obstacle list during its movement.

[0029] Step S200: Based on the point cloud cluster data and the obstacle list, obtain at least one candidate pair between each point cloud cluster and an obstacle in the obstacle list.

[0030] Specifically, if the point cloud cluster data includes N point cloud clusters and the obstacle list stores M obstacles, then after matching the point cloud clusters and obstacles one-to-one, N×M candidate pairs are obtained.

[0031] Step S300: Extract the multidimensional feature vector of each candidate pair, wherein the multidimensional feature vector includes the semantic category features, spatial position relationship features, color features, and point cloud structure difference features of the candidate pair.

[0032] Point cloud clusters possess rich information such as point cloud quantity, 3D bounding box, sensor source, normal vector, and color histogram. Existing obstacles possess attributes such as detection confidence, cumulative perception count, grid occupancy, and laser observability. However, in existing technologies, this information is not involved in the fusion decision; only three features—position, distance, and category relationship—are used. Compared to existing technologies, this invention further extracts color features and point cloud structure features to obtain a multi-dimensional feature vector A, where A = [semantic category matching, spatial position relationship, color appearance similarity, point cloud structure]. It should be noted that this refers to feature vector A containing the listed four categories of features, not that feature vector A has a dimension of four. Each category of features in the multi-dimensional feature vector can contain multiple features. By systematically extracting multi-dimensional features from the existing attributes of point cloud clusters and existing obstacles, the available information in the perception system can be fully utilized to improve the accuracy of obstacle matching.

[0033] Color features can describe the visual similarity between point cloud clusters and existing obstacles, providing a third-dimensional information for fusion determination that is independent of spatial location and semantic category. Two point cloud clusters of the same category and close proximity but with significant color differences may correspond to different physical entities. For example, a white shoe and a black shoe, although of the same category and close proximity, belong to different physical entities. By introducing color features, misjudgments caused by relying solely on category and spatial location are avoided.

[0034] Point cloud structural features can describe the structural characteristics of a point cloud cluster itself. In actual robot operation, the point cloud clusters acquired in each frame typically only cover a local area of ​​an obstacle, such as a corner of a weighing scale or a leg of a chair. This invention incorporates point cloud structural features into the matching judgment, allowing for judgment based on the obstacle's structure rather than relying on the characteristics of the obstacle's complete shape, further avoiding misjudgments.

[0035] Step S400: Based on the pre-trained machine learning model and the multidimensional feature vector of each candidate pair, match the candidate pairs to obtain the matching probability of each candidate pair.

[0036] Existing technologies generally employ a weighted scoring strategy based on manually generated rules for fusion determination. Taking a typical implementation as an example, scores are accumulated for each candidate pair: based on the 3D centroid distance between the two candidates, if the Euclidean distance is less than a threshold, the score is increased by 1.0; based on the category of the point cloud cluster data detected by AI and the category stored in the obstacle list, if the categories match, the score is increased by 1.0; based on the positional relationship between the point cloud cluster and the obstacle, if the center of the point cloud cluster is within the area occupied by the obstacle, the score is increased by 0.5. Finally, if the total score exceeds a fixed threshold, such as 1.5 points, fusion is determined—the point cloud cluster and obstacle are successfully matched. Furthermore, for a few special types, such as pets and furniture legs, hard-coded distance coverage and category mutual exclusion logic are provided.

[0037] This invention employs a machine learning model to comprehensively evaluate the multi-dimensional feature vectors of each candidate pair and output a matching probability. Compared to existing technologies, the machine learning model can automatically learn the contribution weight of each feature dimension to the matching result, transforming the matching judgment from manual rule-driven to data-driven, thus avoiding mismatches caused by the poor generalization ability of fixed thresholds for obstacles of different sizes. The machine learning model can be selected according to the actual situation, such as a graph neural network or a gradient boosting decision tree model, which can output a matching probability based on the input feature vector.

[0038] Step S500: Based on the matching probability, determine whether there are any successfully matched candidate pairs. A successful match means that this point cloud cluster represents all or at least a part of the obstacle. In specific implementation, the existence of successfully matched candidate pairs can be determined by sorting and / or setting a threshold.

[0039] Through the above steps, when performing obstacle matching judgment, multi-dimensional feature vectors of each candidate pair are extracted, making full use of the rich information of point cloud clusters and existing obstacles, improving feature utilization, and effectively enhancing the accuracy of obstacle matching judgment. After obtaining the multi-dimensional feature vectors, a pre-trained machine learning model is used to match candidate pairs and output matching probabilities, replacing the hard-coded rule judgment based on fixed thresholds in existing technologies. This method can adapt to obstacles of different sizes and categories, avoiding the misidentification problem caused by a uniform distance threshold being too small for large obstacles and too large for small obstacles, thus enhancing the generalization ability of matching judgment. The method provided by this invention overcomes the shortcomings of low feature utilization and difficulty in generalization of fixed thresholds in existing technologies. By fully utilizing the multi-dimensional features between point cloud clusters and existing obstacles, it achieves more accurate, robust, and continuously improving matching judgment, significantly improving the obstacle recognition accuracy of sweeping robots.

[0040] Furthermore, the machine learning model is a gradient boosting decision tree model. The training method of the gradient boosting decision tree model includes: obtaining sample candidate pairs for training the model and labeling each sample candidate pair with a binary classification label, the binary classification label including positive samples and negative samples; extracting multi-dimensional feature vectors of the sample candidate pairs; setting an initial value for the cumulative predicted value, the initial value being the log-odds value of the proportion of positive samples in the sample candidate pair; and iteratively training the gradient boosting decision tree model using the gradient boosting method based on the multi-dimensional feature vectors of the sample candidate pairs and the cumulative predicted value, to obtain the trained gradient boosting decision tree model.

[0041] In practice, a robotic vacuum cleaner can collect point cloud clusters during operation, and then label the collected point cloud clusters to obtain labeled candidate pairs of samples. Positive samples are labeled with 1, indicating that the point cloud cluster in the candidate pair corresponds to the same physical entity as the obstacle; negative samples are labeled with 0, indicating that the point cloud cluster in the candidate pair does not correspond to the same physical entity as the obstacle.

[0042] After obtaining the initial values ​​of the multidimensional feature vector and the cumulative predicted value, the multidimensional feature vector is used as input and the initial value is used as the global benchmark to train the gradient boosting decision tree model, and finally the trained gradient boosting decision tree model is obtained.

[0043] This invention employs a gradient boosting decision tree model to calculate matching probabilities, balancing lightweight deployment requirements with high matching accuracy. The gradient boosting decision tree model is a cascaded decision architecture, simple in structure and fast in computation. Inference based on this model requires no additional NPU hardware, controlling computational overhead while maintaining accuracy. Furthermore, the splitting structure of the gradient boosting decision tree model naturally enables feature interaction; each tree automatically combines features during training, eliminating the need for manual design of combination rules. This simplicity and convenience significantly reduces the engineering complexity of system debugging.

[0044] Furthermore, when iteratively training the gradient boosting decision tree model using the gradient boosting method, the following processing is performed for each training round: Using binary cross-entropy as the loss function, the negative gradient between the current model prediction value and the binary label is calculated, and this negative gradient is used as the residual; using the residual as the fitting target, a new decision tree is trained; at each node of the new decision tree, candidate features and their corresponding candidate split points are traversed, and splitting is performed based on the splitting method that maximizes the decrease in the loss function, until a preset stopping condition is met; and the newly trained decision tree is multiplied by the learning rate and added to the gradient boosting decision tree model obtained in the previous training round, and the cumulative prediction value is updated to obtain the gradient boosting decision tree model after this training round. The candidate features include all features of the multidimensional feature vector, and the preset stopping condition includes the maximum tree depth or the minimum number of leaf node samples.

[0045] After multiple iterations, the final model is composed of the sum of the outputs of multiple decision trees. The trained model parameters are exported as a lightweight numerical format and deployed to a robot embedded platform, enabling online inference without relying on third-party inference frameworks or NPU hardware. The model parameters include the tree structure of each decision tree, the split feature index, the split threshold, and the leaf node score.

[0046] Furthermore, the training method for the gradient boosting decision tree model further includes performing the following processing before starting iterative training: obtaining the number of feature vectors of the multidimensional feature vector; comparing the number of feature vectors with a preset number threshold and selecting the smaller value; and setting the number of decision trees in the gradient boosting decision tree model to the smaller value.

[0047] The method proposed in this invention sets the number of decision trees based on the number of feature vectors when training a gradient boosting decision tree model. For example, a preset threshold of 10 is set, and the number of feature vectors is P. If P is less than or equal to 10, P decision trees are generated; if P is greater than 10, 10 decision trees are generated. By setting the number of decision trees to the smaller of the number of feature vectors and the preset threshold, the decrease in model inference accuracy caused by an excessive number of decision trees can be prevented.

[0048] Furthermore, if the machine learning model is a gradient boosting decision tree model, then matching the candidate pairs based on the pre-trained machine learning model and the multidimensional feature vector of each candidate pair to obtain the matching probability of each candidate pair includes the following steps S401 to S402.

[0049] Step S401, perform the following processing on each candidate pair: The multidimensional feature vectors of the candidate pairs are input into a pre-trained gradient boosting decision tree model, which includes multiple decision trees. For each decision tree: Starting from the root node, a judgment is made based on the splitting condition of the feature vector corresponding to each node of the decision tree, and the judgment is made based on the judgment result to enter the next node for judgment, until the leaf node of the decision tree is reached, and the score value output by the decision tree is obtained.

[0050] Figure 2 This is a schematic diagram of the decision tree structure provided in an embodiment of the present invention. Figure 2 Taking Tree 1 as an example, the process starts from the root node "Centroid distance < 0.12m?". If this condition is met, it proceeds to the next node "Are the subclasses consistent?"; otherwise, it proceeds to the next node "Centroid distance < 0.30m?", until it reaches the leaf node of Tree 1, obtaining the leaf node score, which is the score value of Tree 1. It should be noted that... Figure 2 Only two of the 10 decision trees are shown.

[0051] Step S402: Sum the scores of each decision tree to obtain the matching probability of each candidate pair.

[0052] by Figure 2 For example, for a given candidate pair, after obtaining the scores of the leaf nodes from tree 1 to tree 10, these scores are summed to obtain the matching probability of the candidate pair. This process is performed on each candidate pair to obtain the matching probability of each candidate pair.

[0053] This invention models the question of "whether a point cloud cluster in the current frame should match an existing obstacle in the list of existing obstacles" as a binary classification problem. As seen in steps S401 to S402, each decision tree has multiple branching judgment nodes. Each branch represents a binary classification judgment and screening for the corresponding feature. After branching, the next feature is judged. Through layer-by-layer judgment, the final matching probability of the candidate pair is obtained, which improves the judgment accuracy.

[0054] Furthermore, based on the matching probability, determining whether there is a successfully matched candidate pair includes the following steps S501 to S503.

[0055] Step S501: The matching probability of each candidate pair is transformed to obtain a fusion probability value, so that the fusion probability value is located in the (0,1) interval.

[0056] The matching probability of candidate pairs output by the gradient boosting decision tree model may be greater than 1. By performing a transformation process, a fusion probability value within the (0,1) interval is obtained, thus unifying the numerical scale. This facilitates subsequent comparison and judgment based on a preset probability threshold.

[0057] The conversion process can employ the Sigmoid transform, or other methods can be selected based on the actual situation; this invention does not limit this to any particular method.

[0058] Step S502: Based on the fusion probability value, sort all the candidate pairs and take the candidate pair corresponding to the maximum fusion probability value as the matching candidate pair.

[0059] Step S503: Compare the fusion probability value of the matching candidate pair with a preset probability threshold. If the fusion probability value of the matching candidate pair is greater than the preset probability threshold, it is determined that there is a successfully matched candidate pair, and the matching candidate pair is used as the successfully matched candidate pair.

[0060] In this embodiment, the preset probability threshold can be 0.4, 0.5, 0.6, 0.65, 0.8, 0.9, etc., and the present invention does not limit it.

[0061] by Figure 2 For example, the matching probability S of each candidate pair is S = Tree 1 score + Tree 2 score + Tree 3 score + ... + Tree 10 score, and the fusion probability P' = Sigmoid(S) = 1 / (1+e (-S) ).

[0062] For a given matching candidate pair, the cumulative score after passing through 10 trees is S = 2.35; the fusion probability value is P' = 1 / (1 + e^(-1 / 2)). -2.35 =0.913. Taking a preset probability threshold of 0.5 as an example, if the calculated fusion probability value P'>0.5, then the match is considered successful.

[0063] In some embodiments of the present invention, if a match is successful, the information of the point cloud clusters in the matching candidate pair is fused into the existing obstacle to update its state. This "association-fusion" method improves the positional accuracy of the obstacle and enhances the stability of continuous tracking.

[0064] Furthermore, the method also includes, in step S503, if the fusion probability value of the matching candidate pair is less than or equal to the preset probability threshold, it is determined that there is no successfully matched candidate pair, and an obstacle corresponding to the point cloud cluster is created in the obstacle list.

[0065] For a given matching candidate pair, the cumulative score after passing through 10 trees is S = -1.82; the fusion probability value is P' = 1 / (1+e^(-1.82)). 1.82The calculated fusion probability value P' is 0.139. Taking a preset probability threshold of 0.5 as an example, if the calculated fusion probability value P' < 0.5, then the match is considered to have failed. This method not only improves the accuracy of obstacle detection but also allows the obstacle list to be continuously maintained and dynamically updated.

[0066] It should be noted that each type of feature in the multidimensional feature vector may contain multiple features. Specifically, the spatial positional relationship features include at least one of the following features: the three-dimensional centroid distance between the point cloud cluster and the obstacle in the candidate pair, the horizontal plane projection center distance, the height difference, the size ratio, the two-dimensional bounding box intersection-union ratio, and whether the center point of the point cloud cluster is located within the area occupied by the obstacle.

[0067] Among them, the size ratio is the aspect ratio of the horizontal projection of the point cloud cluster and the obstacle, and the two-dimensional bounding box intersection-union ratio is the degree of overlap between the point cloud cluster and the obstacle in the horizontal projection.

[0068] Spatial relationship features are the most information-rich set of features in the fusion judgment. Unlike existing technologies that only use a single distance threshold, this invention comprehensively depicts the positional relationship between pairs from multiple spatial dimensions. This enables the model to observe adjacent independent targets that are close in space but do not overlap, as well as the same obstacle that is slightly far away but highly overlapped in space from different angles. This significantly reduces the probability of false matching and missed matching, and improves the accuracy of matching candidate pairs.

[0069] Furthermore, the color feature includes a statistical distance metric between the point cloud clusters in the candidate pair and the obstacles. This statistical distance metric includes at least one of the following features: histogram correlation, chi-square distance, histogram intersection, and Bach distance between the color histograms of the point cloud clusters and the obstacles. The statistical distance metric is a mathematical expression, but its physical reality is represented by actual color. By introducing color features, misjudgments caused by relying solely on category and spatial location can be avoided, improving the accuracy of obstacle matching.

[0070] Furthermore, the point cloud structure difference features include the difference between the point cloud clusters in the candidate pair and the point cloud structure features of the obstacle, and the point cloud structure features include at least one of the following features: point cloud spatial dispersion, point cloud density, average height, curvature, and plane roughness.

[0071] Among these features, point cloud spatial dispersion describes the standard deviation of the distance from a point to its centroid; point cloud density describes the ratio of the number of points to the bounding box volume; and plane roughness characterizes the roughness of a plane. For example, the point cloud on a towel surface is very rough, while the point cloud on a box surface is relatively smooth and flat. Furthermore, regarding curvature features, a point cloud cluster of an obstacle may have many faces, each with a corresponding curvature. By using features such as point cloud spatial dispersion, point cloud density, average height, curvature, and plane roughness, the local structure of the obstacle can be revealed without relying on the features of the obstacle's complete shape, thus helping to improve the accuracy of obstacle matching.

[0072] Furthermore, the semantic category features include the semantic categories of the point cloud cluster and the obstacle in the candidate pair. The semantic category of the point cloud cluster is obtained by: performing coordinate transformation on the three-dimensional points in the point cloud cluster; projecting the coordinate-transformed three-dimensional points onto the pixel plane of the RGB image to obtain the two-dimensional pixel coordinates corresponding to the three-dimensional points; and performing target detection on the RGB image corresponding to the point cloud cluster based on a pre-trained visual model, and associating the target detection result with the two-dimensional pixel coordinates to obtain the semantic category of the point cloud cluster—that is, the obstacle type.

[0073] In practice, depth images and RGB images are acquired simultaneously during robot movement. After obtaining a point cloud cluster from the depth image, the coordinates of the 3D points in the point cloud cluster are transformed, converting the point cloud cluster from the depth sensor coordinate system to the camera coordinate system. After the coordinate transformation, the 3D points are projected onto the pixel plane of the RGB image using camera intrinsic parameters, obtaining the 2D pixel coordinates corresponding to each 3D point. Simultaneously, a pre-trained AI vision model is used to perform object detection on the same frame of the RGB image corresponding to the point cloud cluster, outputting several 2D detection boxes with semantic categories. Finally, the object detection results are associated with the 2D pixel coordinates, that is, the projected pixels are assigned to each detection box—points falling into a detection box are assigned to the semantic category corresponding to that detection box, while points not falling into any detection box remain in a semantic-free state.

[0074] It should be noted that, generally, one point cloud cluster corresponds to one obstacle, and after target detection, one semantic category is obtained. However, in special cases, such as a shoe next to a trash can, the segmented and extracted point cloud cluster will yield two detection results after target detection. Therefore, in some embodiments of the present invention, if the number of semantic categories obtained by target detection on an RGB image based on a pre-trained visual model is greater than one, the point cloud cluster is segmented based on the target detection results to obtain two sub-point cloud clusters, each corresponding to a semantic category. After obtaining the sub-point cloud clusters, they are matched with obstacles in the obstacle list, and subsequent steps are performed.

[0075] The obstacle matching and judgment method provided in this invention identifies obstacles by extracting multi-dimensional features and calculating the fusion probability value of each candidate pair based on these features. This fully utilizes the available information that the robot can perceive, improving the accuracy of obstacle matching and judgment. When calculating the matching probability, a gradient boosting decision tree model with a cascaded judgment architecture is used for judgment and reasoning, which can reduce the computational overhead of control while ensuring accuracy. This invention can achieve fast and accurate obstacle matching and judgment under the limited computing resources of a robotic vacuum cleaner, and has good engineering practical value.

[0076] This invention also provides a cleaning device, which includes: a cleaning device body; and a control module, the control module being used to execute the method of any of the above embodiments of this invention, enabling the cleaning device body to perform obstacle matching judgment.

[0077] The cleaning equipment itself is equipped with multiple sensors, including LiDAR, binocular or structured light cameras, and RGB cameras. LiDAR is used to acquire the two-dimensional laser point cloud contours of obstacles; the binocular or structured light cameras are used to acquire three-dimensional depth point clouds; and the RGB cameras are used to acquire RGB images, which are then provided to the AI ​​vision model to detect the semantic category and two-dimensional bounding box of obstacles, and combined with RGB camera ranging to estimate the three-dimensional position of the obstacles.

[0078] like Figure 3 As shown, the control module of the cleaning equipment provided in the embodiments of the invention may further include a processor 201, a memory 202, a communication interface 203 and a bus 204, wherein the processor 201, the memory 202 and the communication interface 203 are electrically connected through the bus 204.

[0079] The memory 202 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 204 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0080] Processor 201 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 201 or by instructions in software form. The processor 201 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 202. The processor 201 reads the information in memory 202 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0081] This invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the obstacle matching judgment method described above.

[0082] The computer-readable storage medium provided in this embodiment of the invention stores data and computer-executable instructions for the obstacle matching and judgment method described above. By implementing the above method, the computer-readable storage medium provided in this embodiment of the invention extracts multi-dimensional features from point cloud cluster data, inputs them into a gradient boosting decision tree model for reasoning to make full use of feature data of different dimensions in the point cloud cluster data, and through the cascaded reasoning and judgment of the gradient boosting decision tree model, it can more accurately and stably realize the association judgment of obstacles while controlling the computational cost of the model.

[0083] This invention also provides a computer program product that, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: Step S100: Obtain point cloud cluster data related to obstacles and a pre-established list of obstacles, wherein the point cloud cluster data includes segmented point cloud clusters.

[0084] Step S200: Based on the point cloud cluster data and the obstacle list, obtain at least one candidate pair between each point cloud cluster and an obstacle in the obstacle list.

[0085] Step S300: Extract the multidimensional feature vector of each candidate pair, wherein the multidimensional feature vector includes the semantic category features, spatial position relationship features, color features, and point cloud structure difference features of the candidate pair.

[0086] Step S400: Based on the pre-trained machine learning model and the multidimensional feature vector of each candidate pair, match the candidate pairs to obtain the matching probability of each candidate pair.

[0087] Step S500: Based on the matching probability, determine whether there are any candidate pairs that are successfully matched.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0093] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An obstacle matching and judgment method, characterized in that, The method includes: Acquire point cloud cluster data related to obstacles and a pre-established list of obstacles, wherein the point cloud cluster data includes segmented point cloud clusters; Based on the point cloud cluster data and the obstacle list, at least one candidate pair between each point cloud cluster and an obstacle in the obstacle list is obtained; Extract a multidimensional feature vector for each candidate pair, wherein the multidimensional feature vector includes the semantic category features, spatial position relationship features, color features, and point cloud structure difference features of the candidate pair; Based on a pre-trained machine learning model and the multi-dimensional feature vector of each candidate pair, the candidate pairs are matched to obtain the matching probability of each candidate pair; and Based on the matching probability, it is determined whether there are any candidate pairs that are successfully matched.

2. The method according to claim 1, characterized in that, The machine learning model is a gradient boosting decision tree model, and the training method of the gradient boosting decision tree model includes: Obtain candidate pairs of samples for training the model, and label each candidate pair with a binary classification label, which includes positive and negative samples; Extract the multidimensional feature vectors of the candidate sample pairs; Set an initial value for the cumulative predicted value, wherein the initial value is the log odds value of the proportion of positive samples in the candidate sample pairs; and Based on the multidimensional feature vectors of the candidate sample pairs and the cumulative predicted values, the gradient boosting method is used to iteratively train the gradient boosting decision tree model to obtain the trained gradient boosting decision tree model.

3. The method according to claim 2, characterized in that, When iteratively training the gradient boosting decision tree model using the gradient boosting method, the following processing is performed for each training round: Using the binary cross-entropy as the loss function, calculate the negative gradient between the current model prediction value and the binary label, and use this negative gradient as the residual. Using the residual as the fitting target, a new decision tree is trained. At each node of the new decision tree, candidate features and their corresponding candidate split points are traversed, and splitting is performed based on the splitting method that maximizes the reduction of the loss function until the preset stopping condition is met. as well as The newly trained decision tree is multiplied by the learning rate and then added to the gradient boosting decision tree model obtained from the previous training round. The cumulative prediction value is then updated to obtain the gradient boosting decision tree model after this round of training. The candidate features include all features of the multidimensional feature vector, and the preset stopping condition includes the maximum tree depth or the minimum number of leaf node samples.

4. The method according to claim 2, characterized in that, The training method for the gradient boosting decision tree model also includes performing the following processing before starting iterative training: Obtain the number of feature vectors in the multidimensional feature vector; The number of feature vectors is compared with a preset threshold, and the smaller value is selected; and Set the number of decision trees in the gradient boosting decision tree model to the smaller value.

5. The method according to claim 1, characterized in that, If the machine learning model is a gradient boosting decision tree model, then based on the pre-trained machine learning model and the multidimensional feature vector of each candidate pair, the candidate pairs are matched to obtain the matching probability of each candidate pair, including performing the following processing on each candidate pair: The multidimensional feature vectors of the candidate pairs are input into a pre-trained gradient boosting decision tree model, which includes multiple decision trees. For each decision tree: Starting from the root node, a judgment is made based on the splitting condition of the feature vector corresponding to each node of the decision tree, and the judgment is made based on the judgment result to enter the next node for judgment, until the leaf node of the decision tree is reached, and the score value output by the decision tree is obtained. as well as The scores of each decision tree are summed to obtain the matching probability of each candidate pair.

6. The method according to claim 1, characterized in that, Based on the matching probability, determining whether there are successfully matched candidate pairs includes: The matching probability of each candidate pair is transformed to obtain a fusion probability value, such that the fusion probability value is located in the interval (0,1). Based on the fusion probability values, all candidate pairs are sorted, and the candidate pair corresponding to the highest fusion probability value is selected as the matching candidate pair; and The fusion probability value of the matching candidate pair is compared with a preset probability threshold. If the fusion probability value of the matching candidate pair is greater than the preset probability threshold, it is determined that there is a successfully matched candidate pair, and the matching candidate pair is used as the successfully matched candidate pair.

7. The method according to claim 6, characterized in that, The method further includes: If the fusion probability value of the matching candidate pair is less than or equal to the preset probability threshold, it is determined that there is no successfully matched candidate pair, and an obstacle corresponding to the point cloud cluster is created in the obstacle list.

8. The method according to claim 1, characterized in that, The spatial location relationship feature includes at least one of the following features: The candidate pair includes the three-dimensional centroid distance between the point cloud cluster and the obstacle, the horizontal projection center distance, the height difference, the size ratio, the two-dimensional bounding box intersection-union ratio, and whether the center point of the point cloud cluster is located within the area occupied by the obstacle.

9. The method according to claim 1, characterized in that, The color feature includes a statistical distance metric between the point cloud clusters in the candidate pair and the obstacle, and the statistical distance metric includes at least one of the following features: Histogram correlation, chi-square distance, histogram intersection, and Bach distance between the point cloud cluster and the color histogram of the obstacle.

10. The method according to claim 1, characterized in that, The point cloud structure difference feature includes the difference between the point cloud cluster in the candidate pair and the point cloud structure feature of the obstacle, and the point cloud structure feature includes at least one of the following features: Point cloud spatial dispersion, point cloud density, average height, curvature, and plane roughness.

11. The method according to claim 1, characterized in that, The semantic category features include the semantic categories of the point cloud clusters and the obstacles in the candidate pairs, wherein the semantic categories of the point cloud clusters are obtained in the following ways: Perform coordinate transformation on the three-dimensional points in the point cloud cluster; The coordinate-transformed 3D points are projected onto the pixel plane of the RGB image to obtain the corresponding 2D pixel coordinates of the 3D points; and The point cloud cluster is subjected to target detection based on a pre-trained visual model, and the target detection result is associated with the two-dimensional pixel coordinates to obtain the semantic category of the point cloud cluster.

12. A cleaning device, characterized in that, The cleaning equipment includes: Cleaning equipment body; A control module, the control module being configured to execute the method as described in any one of claims 1 to 11, enabling the cleaning device body to perform obstacle matching judgment.

13. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.

Images