Machine Vision-Based Control Method for Large-Particle Debris Cleaning and Sweeping Robots

Abstract

This application relates to the field of sweeping robot control, and discloses a machine vision-based method for controlling the cleaning of large-particle debris and a sweeping robot. The method includes: acquiring an image of a cleaning area containing large-particle debris through machine vision; determining the debris outline region and debris type label in the cleaning area image through image recognition; determining the region with the largest and smallest horizontal diameter of the debris outline region through image recognition; obtaining the optimal gripping height of a tendon-driven manipulator corresponding to the region with the largest and smallest horizontal diameter of the outline and the debris type label through a cleaning database; and controlling the tendon-driven manipulator to grip and clean the large-particle debris corresponding to the cleaning area image according to the optimal gripping height. The tendon-driven manipulator can move vertically along the height direction of the sweeping robot. This application enables precise positioning and stable gripping of large-particle debris.

Description

Technical Field

[0001] This application relates to the field of sweeping robot control technology, and more specifically, to a machine vision-based control method for cleaning large-particle debris and a sweeping robot. Background Technology

[0002] Robotic vacuum cleaners, as intelligent cleaning devices, are widely used in various scenarios such as homes and offices. Current robotic vacuum cleaners typically integrate core components such as an autonomous movement module, a cleaning execution module, and a navigation and positioning module. The autonomous movement module utilizes a wheel-driven structure combined with positioning devices such as gyroscopes, lidar, or visual sensors to achieve full-coverage movement of the cleaning area. The cleaning execution module often employs a combination of rotating roller brushes for sweeping, side brushes for gathering debris, and a vacuum cleaner for collection, effectively cleaning common debris such as dust and crumbs from the floor.

[0003] While existing robotic vacuum cleaners are highly practical for general garbage cleaning, they still have significant technical limitations when handling specific types of debris on the ground, particularly in accurately gripping and holding it. In real-world cleaning scenarios, when large or irregularly shaped debris is present, current cleaning mechanisms can only handle it through a combination of roller brushes and vacuuming, failing to achieve precise positioning and stable gripping of such debris. This deficiency not only results in some debris not being effectively collected, but also leads to incomplete cleaning. Summary of the Invention

[0004] The purpose of this application is to provide a machine vision-based method for controlling the cleaning of large-particle debris and a sweeping robot, which solves the technical problem of being unable to accurately locate and stably hold large-particle debris, and achieves the technical effect of accurately locating and stably holding large-particle debris.

[0005] This application provides a machine vision-based method for controlling the cleaning of large-particle debris. The method includes: acquiring an image of a cleaning area containing large-particle debris using machine vision; determining the debris outline region and debris type label in the cleaning area image using image recognition; determining the region with the largest and smallest horizontal diameter corresponding to the debris outline region using image recognition; wherein the region with the largest and smallest horizontal diameter includes the region size and the region height relative to the ground; obtaining the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest and smallest horizontal diameter and the debris type label using a cleaning database; controlling the tendon-driven manipulator to grip and clean the large-particle debris corresponding to the cleaning area image according to the optimal gripping height; wherein the tendon-driven manipulator includes two tendon-driven mechanical grippers disposed on the outer wall of the sweeping robot, and the tendon-driven manipulator can move up and down along the height direction of the sweeping robot under the drive of the vertical drive component.

[0006] In one possible implementation, determining the region with the largest and smallest horizontal diameter of the waste outline region corresponding to the waste outline region through image recognition includes: when the waste type label is a flexible waste label, obtaining the closest distance between the sweeping robot and large-particle waste corresponding to the waste outline region as a first distance; controlling the sweeping robot to move along the current orientation to increase the closest distance between the sweeping robot and large-particle waste to a second distance, and obtaining a reference region image corresponding to the second distance through machine vision; determining multiple reference region feature points corresponding to the reference region image through image recognition; determining multiple waste outline sub-regions of the waste outline region; within each waste outline sub-region, determining the number of multiple matching feature points that match the positions of the multiple reference region feature points within the waste outline sub-region as the number of matching feature points; determining the ratio of the number of matching feature points corresponding to each waste outline sub-region to the number of reference region feature points as the transparency of each waste outline sub-region; among the multiple waste outline sub-regions, determining multiple opaque waste outline sub-regions whose transparency is less than a preset waste outline sub-region transparency; and determining the region with the largest and smallest horizontal diameter of the total region corresponding to the multiple opaque waste outline sub-regions through image recognition.

[0007] In another possible implementation, the method further includes: determining the number of adjacent edges of each opaque garbage outline sub-region and its adjacent opaque garbage outline sub-regions as the adjacency factor corresponding to the opaque garbage outline sub-region; determining multiple opaque garbage outline sub-regions with an adjacency factor greater than 1 as multiple adjacent opaque garbage outline sub-regions; and determining the region with the largest horizontal diameter and the region with the smallest horizontal diameter of the total region corresponding to the multiple adjacent opaque garbage outline sub-regions through image recognition.

[0008] In another possible implementation, the maximum and minimum horizontal diameter regions of the waste outline region are determined by image recognition, including: when the waste type label is a non-flexible waste label, the maximum and minimum horizontal diameter regions of the waste outline region are determined by image recognition.

[0009] In another possible implementation, the method further includes: determining a first proportion of the garbage outline region corresponding to the first distance in the image of the cleaning area by image recognition; and determining a second distance based on the flexible garbage label, the first distance, and the first proportion.

[0010] In another possible implementation, the method further includes: determining a garbage contour region in a reference region image; determining a second proportion of the garbage contour region corresponding to the second distance in the reference region image through image recognition; determining the ratio of the second proportion to the first proportion as a reference region adjustment proportion; when the reference region adjustment proportion is greater than or equal to a preset reference region adjustment proportion, multiplying the second distance by the preset distance adjustment proportion to increase the second distance; controlling the sweeping robot to move along the current orientation to increase the closest distance between the sweeping robot and large garbage particles to the increased second distance; and determining the region with the largest horizontal diameter and the region with the smallest horizontal diameter of the contour corresponding to the increased second distance; wherein the preset distance adjustment proportion is greater than 1.

[0011] In another possible implementation, the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the waste type label is obtained by cleaning the database. This includes: determining the initial gripping height based on the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the waste type label; obtaining multiple historical successful gripping records corresponding to the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the waste type label by cleaning the database; obtaining multiple historical successful gripping heights from the multiple historical successful gripping records, determining the average height of the multiple historical successful gripping heights as the average historical successful gripping height; and determining the median height between the average historical successful gripping height and the initial gripping height as the optimal gripping height.

[0012] In another possible implementation, by cleaning the database to obtain the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the waste type label, the method further includes: by cleaning the database to obtain multiple historical failed gripping records corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the waste type label; obtaining multiple historical failed gripping heights from the multiple historical failed gripping records, and determining the average height of the multiple historical failed gripping heights as the average historical failed gripping height; determining the median height between the average historical successful gripping height and the initial gripping height as the corrected gripping height; when the corrected gripping height is between the average historical successful gripping height and the average historical failed gripping height, determining the height difference between the corrected gripping height and the average historical failed gripping height as the corrected height difference; when the corrected height difference is less than the preset corrected height difference, adjusting the corrected gripping height toward the position corresponding to the average historical successful gripping height as the optimal gripping height.

[0013] In another possible implementation, the initial clamping height is determined based on the region with the largest horizontal diameter of the outline, the region with the smallest horizontal diameter of the outline, and the waste type label. This includes: obtaining the preset clamping area corresponding to the waste type label; when the preset clamping area is the region with the largest horizontal diameter of the outline, determining the initial clamping height as the height corresponding to the region with the largest horizontal diameter of the outline; when the preset clamping area is the region with the smallest horizontal diameter of the outline, determining the initial clamping height as the height corresponding to the region with the smallest horizontal diameter of the outline.

[0014] This application also provides a sweeping robot, including a unit for implementing the above-described machine vision-based large particle debris sweeping control method.

[0015] The beneficial effects of the embodiments in this application compared with the prior art are: This application provides a machine vision-based method for controlling the cleaning of large-particle waste. The method includes: acquiring an image of a cleaning area containing large-particle waste through machine vision; determining the waste outline region and waste type label in the cleaning area image through image recognition; determining the region with the largest and smallest horizontal diameter corresponding to the waste outline region through image recognition; obtaining the optimal gripping height of a tendon-driven manipulator corresponding to the region with the largest and smallest horizontal diameter and the waste type label through a cleaning database; and controlling the tendon-driven manipulator to grip and clean the large-particle waste corresponding to the cleaning area image according to the optimal gripping height. The method in this application acquires a cleaning area image through machine vision, obtains the waste outline region and waste type label through image recognition, determines the region with the largest and smallest horizontal diameter and its related dimensions and height, and matches the optimal gripping height of the tendon-driven manipulator with the cleaning database to control its gripping and cleaning. This process achieves precise identification and positioning, improves the targeting and accuracy of large-particle waste cleaning, adapts to waste of different sizes and heights, and enhances cleaning adaptability and flexibility. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the first machine vision-based large particle debris cleaning control method provided in this application embodiment; Figure 2 A schematic diagram illustrating the workflow of the first machine vision-based large particle debris cleaning control method provided in this application embodiment; Figure 3 A top view of the sweeping robot to which the first machine vision-based large particle debris sweeping control method provided in the embodiments of this application is applied; Figure 4 A schematic diagram of the left-side structure of the sweeping robot to which the first machine vision-based large particle debris sweeping control method provided in the embodiments of this application is applied; Figure 5 A schematic diagram of the cleaning area image in the first machine vision-based large particle debris cleaning control method provided in the embodiments of this application; Figure 6 A flowchart illustrating the second machine vision-based large-particle debris cleaning control method provided in this application embodiment; Figure 7 A schematic diagram illustrating the workflow of the second machine vision-based large particle debris cleaning control method provided in this application embodiment; Figure 8 A flowchart illustrating the third machine vision-based large-particle debris cleaning control method provided in this application embodiment; Figure 9A flowchart illustrating the fourth machine vision-based large-particle debris cleaning control method provided in this application embodiment; Figure 10 A flowchart illustrating the fifth machine vision-based large particle debris cleaning control method provided in this application embodiment; Figure 11 A schematic diagram of the workflow of the fifth machine vision-based large particle debris cleaning control method provided in the embodiments of this application; Figure 12 A flowchart illustrating the sixth machine vision-based large particle debris cleaning control method provided in this application embodiment; Figure 13 A schematic diagram of the workflow of the sixth machine vision-based large particle debris cleaning control method provided in the embodiments of this application; Figure 14 This is a schematic diagram of the logical structure of a sweeping robot provided in an embodiment of this application. Detailed Implementation

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0020] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] Existing robotic vacuum cleaners are not good at accurately gripping trash on the ground. When there are large pieces or irregularly shaped trash on the ground, they cannot accurately locate and stably grip such trash.

[0023] Based on the above reasons, this application provides a machine vision-based method for controlling the cleaning of large-particle waste. The method includes: acquiring an image of a cleaning area containing large-particle waste using machine vision; determining the waste outline region and waste type label in the cleaning area image through image recognition; determining the region with the largest and smallest horizontal diameter corresponding to the waste outline region through image recognition; obtaining the optimal gripping height of a tendon-driven manipulator corresponding to the region with the largest and smallest horizontal diameter and the waste type label from a cleaning database; and controlling the tendon-driven manipulator to grip and clean the large-particle waste corresponding to the cleaning area image at the optimal gripping height. The method in this application acquires a cleaning area image using machine vision, obtains the waste outline region and waste type label through image recognition, determines the region with the largest and smallest horizontal diameter and its related dimensions and height, and matches the optimal gripping height of the tendon-driven manipulator with the cleaning database to control its gripping and cleaning. This process achieves precise identification and positioning, improves the targeting and accuracy of large-particle waste cleaning, adapts to waste of different sizes and heights, and enhances cleaning adaptability and flexibility.

[0024] In some scenarios, the machine vision-based large-particle debris cleaning control method of this application embodiment can be applied to the control of a sweeping robot, which can be applied to the sweeping robot in the process of cleaning large-particle debris, thereby improving the daily cleaning effect of the sweeping robot.

[0025] The following describes in detail, with specific examples, a machine vision-based method for controlling the cleaning of large particulate waste provided in the embodiments of this application.

[0026] Figure 1 A flowchart illustrating the first machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 1 As shown in the embodiment of this application, a large particle debris cleaning control method based on machine vision is provided, including S110 to S120. S110 to S120 will be described in detail below.

[0027] S110. Acquire an image of the cleaned area containing large particles of waste using machine vision. Determine the waste outline region and waste type label in the cleaned area image through image recognition. Determine the region with the largest and smallest horizontal diameter of the waste outline region through image recognition. The region with the largest and smallest horizontal diameter includes the region size and the region's height relative to the ground.

[0028] Figure 2 A schematic diagram illustrating the workflow of the first machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 2 As shown, in this implementation, the vision device on the robot vacuum cleaner can be used to collect images of the cleaning area along the path, ensuring that the images cover large particles of debris on the ground, providing basic data for subsequent processing.

[0029] It should be noted that machine vision can be achieved through a horizontally mounted camera on the robot vacuum. The camera is installed horizontally at the front or side of the robot, with the lens facing the area above the ground, covering a certain range of the ground in front of the robot. For example, when the robot vacuum is cleaning in a bedroom, the front horizontal camera will capture images of the ground within 1.2 meters in front of it. When there are toy blocks present, the camera will capture an image of the area containing the blocks.

[0030] In this implementation, the image can first be preprocessed by converting it to grayscale and reducing noise, and then the edge detection algorithm can be used to extract the contour boundary of the garbage to form a closed contour region. At the same time, a pre-trained classification model is used to match the garbage features within the contour and output the category label to achieve localization and identification.

[0031] It should be noted that the image recognition process can be implemented by combining image recognition algorithms. For example, the Canny algorithm can be used to detect edges to form contours, and then the image blocks within the contours can be input into a convolutional neural network model to identify features such as shape and texture and output labels. For example, when there is a plastic bottle in the image, the Canny algorithm extracts the edges of the bottle to form a contour, the model identifies cylindrical and transparent features, and outputs the label "plastic bottle".

[0032] In this implementation, the distance between points along the horizontal direction within the outline of the waste can be measured to find the region with the largest distance between the two points as the largest region and the region with the smallest distance between the two points as the smallest region. At the same time, the size of the two regions and their height relative to the ground can be calculated using ground reference points (such as the contact points of the robot wheels).

[0033] It should be noted that the maximum and minimum areas of waste vary depending on the type of waste. For example, the maximum area of ​​a plastic bottle is in the middle of the bottle body, and the minimum area is at the mouth; the maximum area of ​​an aluminum can is in the middle of the can body, and the minimum area is at the mouth. For example, a 20cm tall plastic bottle has a maximum area size of 8cm and a height of 10cm; and a minimum area size of 3cm and a height of 18cm.

[0034] S120. Using the cleaning database, obtain the optimal gripping height of the tendon-driven robotic arm corresponding to the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the type of debris label. Control the tendon-driven robotic arm to grip and clean large debris particles corresponding to the cleaning area image according to the optimal gripping height. The tendon-driven robotic arm includes two tendon-driven mechanical grippers located on the outer wall of the robot vacuum. Driven by the vertical drive component, the tendon-driven robotic arm can move vertically along the height direction of the robot vacuum.

[0035] In this implementation, a cleaning database can be pre-established to store the optimal clamping height corresponding to different types of waste and different contour areas; after obtaining the label and maximum and minimum area data of the current waste, a matching query is performed in the database to obtain the optimal height.

[0036] It should be noted that the database can use waste type as a primary index, maximum area size and height as secondary indexes, and minimum area as a tertiary index, matching results sequentially. For example, when the label is "plastic bottle," the maximum area is 8cm × 10cm, and the minimum area is 3cm × 18cm, the database matches the optimal clamping height as 10cm.

[0037] In this implementation, the vertical drive component can be controlled to move the robotic arm to the target position along the robot's height direction according to the optimal gripping height; then, the two mechanical grippers can be controlled to move towards each other to grip the waste; finally, the waste can be put into the collection bin by the robot moving or the robotic arm rotating.

[0038] It should be noted that the robotic arm includes two tendon-driven grippers located on the outer wall, which can move up and down as a whole via the vertical assembly. For example, at the optimal height of 10cm, the electric push rod propels the robotic arm from 0cm to 10cm, the grippers retract from a 15cm spacing to 8cm to grip the middle of the plastic bottle, and then it moves to the waste collection area to release.

[0039] For example, the optimal height for a plastic bottle is 10cm, for a soda can it is 8cm, and for a spherical toy it is 5cm.

[0040] Figure 3 This is a top view of the sweeping robot used in the first machine vision-based large particle debris sweeping control method provided in this application embodiment. Figure 4A schematic diagram of the left-side structure of the sweeping robot used in the first machine vision-based large-particle debris sweeping control method provided in this application embodiment, as shown below. Figure 3 and Figure 4 As shown, the sweeping robot 1 has a tendon-driven manipulator 2, which includes two tendon-driven mechanical grippers 21 located on the outer side wall of the sweeping robot. The tendon-driven manipulator 21 can move up and down along the height direction of the sweeping robot 1 under the drive of the vertical drive component 22, and thus can grasp different large particles of garbage by adjusting the gripping height.

[0041] Figure 5 A schematic diagram of the cleaning area image in the first machine vision-based large particle debris cleaning control method provided in the embodiments of this application, as shown below. Figure 5 As shown in Figure a, the region with the largest horizontal diameter of the contour is the region corresponding to Dmax, and the region with the smallest horizontal diameter of the contour is the region corresponding to Dmin.

[0042] This implementation method acquires images of the cleaning area through machine vision, obtains the outline area of ​​the waste and the waste type label through image recognition, and then determines the largest and smallest areas of the horizontal diameter of the outline, as well as related dimensions and height. Combined with the cleaning database, it matches the optimal gripping height of the tendon-driven robotic arm and controls its gripping and cleaning. The process achieves precise identification and positioning, improving the targeting and accuracy of cleaning large particles of waste. This structure and movement method allow the robotic arm to adapt to waste of different sizes and heights, enhancing the adaptability and flexibility of the cleaning process.

[0043] This method first collects and identifies garbage information using machine vision, then quickly matches the optimal clamping height based on the cleaning database, and finally controls the tendons to drive the robotic arm to perform cleaning. This reduces ineffective operations, ensures stable clamping by the robotic arm, lowers the probability of garbage falling, and improves cleaning efficiency and reliability.

[0044] Figure 6 A flowchart illustrating the second machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 6 As shown, in some implementations, in the above-mentioned S110, the region with the largest horizontal diameter and the region with the smallest horizontal diameter of the contour corresponding to the garbage contour region are determined by image recognition, including S111 to S113. S111 to S113 will be explained in detail below.

[0045] S111. When the waste type label is a flexible waste label, obtain the closest distance between the sweeping robot and the large waste particles corresponding to the waste outline area, as the first distance. Control the sweeping robot to move along the current orientation to increase the closest distance between the sweeping robot and the large waste particles to a second distance, and obtain the reference area image corresponding to the second distance through machine vision.

[0046] Figure 7 A schematic diagram of the workflow of the second machine vision-based large particle debris cleaning control method provided in the embodiments of this application is shown below. Figure 7 As shown in this implementation, when the waste type label is a flexible waste label, the closest distance between the sweeping robot and the large-particle waste can be obtained as the first distance.

[0047] It should be noted that flexible waste tags are used to identify large, easily deformable, and soft-material waste, such as plastic bags, woven bags, and food packaging films. The first distance can be determined using a depth camera on the robot vacuum cleaner. The depth camera emits infrared light towards the outline of the waste, receives the reflected light from the surface, and calculates the depth value of each pixel. The minimum value among all depth values ​​is the closest distance (first distance) between the robot vacuum cleaner and the large waste.

[0048] For example, such as Figure 5 As shown in Figure a, the image corresponding to the first distance between the robot vacuum and large debris particles is the cleaning area image.

[0049] In this implementation, the robot vacuum can be controlled to move along its current orientation, increasing the closest distance to large debris particles from a first distance to a second distance. The current orientation is the direction the robot vacuum faces when acquiring the first distance. For example, when the robot vacuum is facing the debris, it moves in the opposite direction (backwards) to the front, increasing the closest distance from 0.5 meters (first distance) to 1 meter (second distance). After moving to the second distance, the horizontal camera on the robot vacuum can capture an image of the area containing the debris; this image serves as the reference area image corresponding to the second distance.

[0050] For example, such as Figure 5 As shown in Figure b, the image corresponding to the second distance between the robot vacuum and the large debris is the reference area image. Due to the increased distance between the robot vacuum and the large debris, the perspective relationship is affected, making the area corresponding to the large debris in the reference area image smaller than the area corresponding to the large debris in the cleaning area image.

[0051] S112. Through image recognition, determine multiple reference region feature points corresponding to the reference region image. Determine multiple garbage contour sub-regions within the garbage contour region. Within each garbage contour sub-region, determine the number of matching feature points that match the positions of the multiple reference region feature points within the garbage contour sub-region, which is taken as the number of matching feature points. Determine the ratio of the number of matching feature points corresponding to each garbage contour sub-region to the number of reference region feature points, which is taken as the transparency of each garbage contour sub-region.

[0052] In this implementation, multiple reference region feature points can be determined using image recognition algorithms. Specifically, the SIFT (Scale Invariant Feature Transform) algorithm can be used to extract unique points such as corner points, edge points, and texture feature points from the reference region image as reference region feature points. For example, the corners of garbage edges and abrupt texture changes in the reference region image can be used as reference region feature points.

[0053] In this implementation, the garbage outline region can be divided into multiple garbage outline sub-regions. Specifically, a grid segmentation method can be used to divide the garbage outline region into uniform grids according to a preset size (such as 10 pixels × 10 pixels). Each grid is a garbage outline sub-region, refining the local features of the garbage outline to facilitate subsequent analysis.

[0054] In this implementation, the number of matching feature points that match the feature point positions of the reference area can be counted within each garbage outline sub-region. Specifically, the KNN (K nearest neighbor) matching algorithm can be used: extract feature points within the garbage outline sub-region, calculate the Euclidean distance between each sub-region feature point and all reference area feature points, and if the distance is less than a preset threshold (such as 2 pixels), it is considered a matching feature point. The number of matching feature points within each sub-region is then counted.

[0055] In this implementation, the ratio of the number of matching feature points in each garbage contour sub-region to the number of feature points in the reference region can be calculated as the transparency of that sub-region. For example, if the total number of feature points in the reference region is 100 and the number of matching feature points in a certain sub-region is 85, then the transparency of that sub-region is 0.85. The larger the ratio, the more similar the image features of the sub-region and the reference region are, that is, the higher the transparency of the sub-region.

[0056] S113. In multiple garbage outline sub-regions, identify multiple opaque garbage outline sub-regions whose transparency is less than the preset transparency of the garbage outline sub-regions. Use image recognition to determine the region with the largest and smallest horizontal diameter of the total region corresponding to the multiple opaque garbage outline sub-regions.

[0057] In this implementation, opaque waste outline sub-regions with transparency greater than a preset waste outline sub-region transparency can be selected. The preset transparency can be set according to the characteristics of flexible waste (e.g., 0.7). If the transparency of a sub-region is 0.22 (less than 0.7), then that sub-region is an opaque waste outline sub-region. These sub-regions usually correspond to the solid parts of flexible waste (e.g., the polyethylene layer of a plastic bag), while sub-regions with higher transparency may correspond to the transparent parts or severely deformed parts of the waste.

[0058] In this implementation, multiple opaque waste outline sub-regions can be merged into a total region. The region with the largest and smallest horizontal diameter of the outline is determined by image recognition. After merging adjacent opaque sub-regions to form a continuous region, the region with the longest horizontal diameter (such as the opening of a plastic bag) and the region with the shortest horizontal diameter (such as the bottom of a plastic bag) are identified. The dimensions of these regions (such as the largest region with a horizontal diameter of 30 cm and the smallest region with a horizontal diameter of 10 cm) and their height relative to the ground (such as the largest region with a height of 20 cm and the smallest region with a height of 5 cm) are recorded, providing reliable data support for matching the optimal clamping height in the future.

[0059] This implementation method, when the waste type label is a flexible waste label, first obtains the first distance between the sweeping robot and the large-particle waste, then controls the sweeping robot to move to the second distance and obtains a reference area image. After determining the feature points of the reference area and the waste outline sub-regions, the ratio of the number of matching feature points in each sub-region to the number of feature points in the reference area is calculated as the transparency. After filtering out the opaque waste outline sub-regions, the maximum and minimum horizontal diameter regions of the total region's outline are identified. Based on the key region data of the opaque region of the flexible waste, reliable support is provided for the optimal clamping height. The opaque region is determined for areas with low transparency, improving the rationality of the clamping height determination and reducing the probability of clamping errors. This method compensates for the shortcomings of existing technologies in handling flexible waste and improves the cleaning adaptability to different types of large-particle waste.

[0060] Figure 8 A flowchart illustrating the third machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 8 As shown, in some implementations, the above method also includes S114 to S115, which will be explained in detail below.

[0061] S114. Determine the number of adjacent edges of each opaque garbage outline sub-region and its neighboring opaque garbage outline sub-regions, and use this number as the adjacency factor of the opaque garbage outline sub-region.

[0062] In this implementation, the set of boundary pixel coordinates of each opaque garbage outline sub-region can be obtained first, and then other opaque sub-regions can be traversed to determine whether there are continuous overlapping pixel segments between the boundary of the current sub-region and the boundary of other sub-regions. For each adjacent sub-region with an overlapping boundary, the number of adjacent edges of the current sub-region is incremented by 1. The final count is the adjacency factor of the sub-region.

[0063] For example, the right boundary pixels of sub-region A are (x=20, y=10) to (x=20, y=20), and the left boundary pixels of sub-region B are (x=20, y=15) to (x=20, y=25). The two have overlapping boundaries at x=20 and y=15 to y=20, which is counted as one adjacent edge. If sub-region A also overlaps with the upper boundary of sub-region C, the number of its adjacent edges is 2, and the adjacency factor is 2.

[0064] S115. Identify multiple opaque garbage contour sub-regions with an adjacency factor greater than 1, as multiple adjacent opaque garbage contour sub-regions. Use image recognition to determine the region with the largest and smallest horizontal contour diameter of the total region corresponding to these multiple adjacent opaque garbage contour sub-regions.

[0065] In this implementation, the adjacency factor of all opaque garbage outline sub-regions can be traversed, and sub-regions with values ​​greater than 1 can be filtered out. These sub-regions are the adjacent opaque garbage outline sub-regions. When filtering, it is only necessary to compare the adjacency factor with 1. Those with values ​​greater than 1 are included in the target set.

[0066] For example, in an opaque sub-region of a certain flexible waste, sub-region X has an adjacency factor of 3, sub-region Y has an adjacency factor of 2, and sub-region Z has an adjacency factor of 1. After screening, sub-regions X and Y are selected as adjacent opaque sub-regions, while Z is excluded because its adjacency factor is equal to 1.

[0067] In this implementation, multiple adjacent opaque sub-regions can be merged into a continuous total region. Then, an edge detection algorithm based on image recognition is used to find the regions with the largest and smallest horizontal diameters of the total region. At the same time, depth information is used to obtain the height of these two regions relative to the ground, providing control parameters for subsequent clamping height calculation.

[0068] This implementation method determines the number of adjacent edges of each opaque waste outline sub-region and its neighbors as the adjacency factor. Multiple opaque waste outline sub-regions with adjacency factors greater than 1 are selected. Image recognition is then used to determine the region with the largest and smallest horizontal diameter of the total outline. Sub-regions with adjacency factors greater than 1 are not isolated regions, thus eliminating isolated opaque noise sub-regions and improving the accuracy of identifying key opaque regions. This also makes the clamping height more closely match the shape of the opaque region of flexible waste, improving the rationality of the clamping height, reducing clamping offset, and enhancing clamping reliability.

[0069] In some implementations, in the above-mentioned S110, determining the maximum and minimum horizontal diameter regions of the waste outline region through image recognition includes: when the waste type label is a non-flexible waste label, determining the maximum and minimum horizontal diameter regions of the waste outline region through image recognition.

[0070] In this implementation, for cases where the waste type label is non-flexible waste, the geometric features of the waste outline region can be directly extracted through image recognition technology to determine the region with the largest and smallest horizontal diameter of the outline. Since non-flexible waste has a stable shape and its outline will not be significantly deformed by external factors, there is no need to perform the distance adjustment, feature point matching and other processes corresponding to flexible waste. Accurate key area data can be obtained directly by identification.

[0071] It should be noted that non-flexible waste tags correspond to waste types whose shapes are not easily deformed. The outline features of this type of waste can be clearly and stably presented in the image. For example, specific types of non-flexible waste tags can include plastic bottles, metal cans, ceramic fragments, cardboard boxes, etc. These wastes have a sturdy structure, and their outline shape will not change significantly due to the shooting distance of the robot vacuum cleaner or its own placement.

[0072] This implementation eliminates the need for processes like distance adjustment, reference area feature point matching, transparency calculation of waste outline sub-regions, and adjacency factor filtering when the waste type label is non-flexible. Instead, it directly determines the maximum and minimum horizontal diameter regions of the waste outline region through image recognition. This simplifies the recognition process for non-flexible waste, avoids redundant steps, reduces the computational load on the robot vacuum cleaner, shortens the overall time for key area recognition, and effectively improves recognition efficiency. It also provides precise data support for determining the clamping height, improving the accuracy of the clamping height, reducing clamping deviation, and enhancing clamping reliability.

[0073] Figure 9 A flowchart illustrating the fourth machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 9 As shown, in some implementations, the above method also includes S210 to S220, which will be described in detail below.

[0074] S210. Determine the first proportion of the garbage outline area corresponding to the first distance in the image of the cleaning area through image recognition.

[0075] In this implementation, the number of pixels in the garbage outline region at the first distance can be extracted by image recognition, and then the total number of pixels in the cleaned area image can be counted. The number of pixels in the garbage outline region is divided by the total number of pixels in the cleaned area image to obtain the first ratio. This ratio can reflect the proportion of garbage in the current cleaned area image and provide a quantitative basis for determining the second distance.

[0076] For example, when the initial distance between the robot vacuum and the soft debris is 0.5 meters, the number of pixels in the debris outline area detected by the image recognition tool is 2000, and the total number of pixels in the cleaning area image is 10000. Then the first ratio is 2000 divided by 10000, which is 0.2.

[0077] S220. Determine the second distance based on the flexible waste label, the first distance, and the first ratio.

[0078] In this implementation, the distance between the robot vacuum cleaner and large particles of waste can be adjusted to a second distance by combining the type of waste corresponding to the flexible waste label, the current first distance, and the first proportion of the waste outline area in the cleaning area image. By adjusting, the imaging proportion of flexible waste in the subsequently acquired reference area image is more suitable for feature point extraction and matching, thereby improving the accuracy of subsequent processes.

[0079] For example, for the "plastic bag" type in flexible waste labels, a pre-established empirical value table records the second distance corresponding to different first distances and first ratios. When the first distance is 0.5 meters and the first ratio is 0.2, the corresponding second distance in the empirical value table is 1 meter; when the first distance is 0.6 meters and the first ratio is 0.15, the corresponding second distance is 1.2 meters. By querying the empirical value table, the appropriate second distance can be quickly determined.

[0080] This implementation method first obtains the first distance between the robot vacuum cleaner and large-particle waste when the waste type label is a flexible waste label. Then, it determines the first proportion of the waste outline area corresponding to the first distance in the cleaning area image through image recognition. Finally, it determines the second distance based on the flexible waste label, the first distance, and the first proportion. The first proportion reflects the proportion of waste in the image. Combining the label and the first distance can dynamically adapt to the actual situation of flexible waste. By referring to the appropriate proportion of flexible waste imaging in the reference area image, the subsequent feature point extraction is more accurate, reducing subsequent process errors and improving the recognition accuracy of key parameters of the opaque waste outline sub-region and the corresponding total region.

[0081] Figure 10 A flowchart illustrating the fifth machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 10 As shown, in some implementations, the above method also includes S230 to S240, which will be described in detail below.

[0082] S230. Determine the garbage contour region in the reference region image, and determine the second proportion of the garbage contour region corresponding to the second distance in the reference region image through image recognition.

[0083] Figure 11A schematic diagram of the workflow of the fifth machine vision-based large particle debris cleaning control method provided in the embodiments of this application is shown below. Figure 11 As shown, in this implementation, the garbage contour region in the reference region image can be extracted first using an image segmentation algorithm. Then, the ratio of the number of pixels in this region to the total number of pixels in the reference region image is calculated to obtain the second ratio. The second ratio reflects the imaging proportion of flexible garbage in the reference region image and is a core indicator for judging whether the image is suitable for feature point extraction.

[0084] It should be noted that the calculation of the second ratio is based on pixel statistics logic. For example, if the reference area image is 1280×720 pixels (total pixels 921600), and the number of pixels in the garbage outline area is 184320, then the second ratio is 20%.

[0085] S240. Determine the ratio of the second ratio to the first ratio as the reference area adjustment ratio. When the reference area adjustment ratio is greater than or equal to the preset reference area adjustment ratio, multiply the second distance by the preset distance adjustment ratio to increase the second distance. Control the sweeping robot to move along the current orientation to increase the closest distance between the sweeping robot and large debris particles to the increased second distance, and determine the region with the largest horizontal diameter and the region with the smallest horizontal diameter corresponding to the increased second distance. The preset distance adjustment ratio is greater than 1.

[0086] In this implementation, the second ratio can be divided by the first ratio (the proportion of the garbage outline area corresponding to the first distance in the image of the cleaning area) to obtain the reference area adjustment ratio. The reference area adjustment ratio can quantify the degree of change in the proportion of garbage after the second distance is adjusted, and provide a basis for judgment for subsequent adjustments.

[0087] It should be noted that, for example, if the first ratio is 30% (the garbage occupies 30% of the cleaned area image at the first distance) and the second ratio is 20% (the garbage occupies 20% of the reference area image at the second distance), then the reference area adjustment ratio is 20% / 30%≈0.67, which means that the garbage proportion has been reduced by about 33%.

[0088] For example, if the first ratio is 25% and the second ratio is 18%, then the reference area adjustment ratio is 18% / 25%=0.72.

[0089] In this implementation, a preset reference area adjustment ratio (e.g., 0.8) can be set in advance. If the reference area adjustment ratio is greater than or equal to the preset reference area adjustment ratio, it indicates that the proportion of garbage is still too large and the imaging is easily affected by the background. At this time, the second distance is multiplied by the preset distance adjustment ratio (e.g., 1.2) to obtain the increased second distance, so as to further reduce the proportion of garbage area in the reference area image.

[0090] It should be noted that the preset reference area adjustment ratio can be an empirical threshold based on the clarity of flexible waste imaging, and the preset distance adjustment ratio is a coefficient to ensure that the distance adjustment is effective (the preset distance adjustment ratio must be greater than 1).

[0091] For example, if the preset reference area adjustment ratio is 0.8, the current reference area adjustment ratio is 0.85 (≥0.8), the initial second distance is 1.2 meters, and the preset distance adjustment ratio is 1.2, then the increased second distance is 1.2 × 1.2 = 1.44 meters.

[0092] In this implementation, the robot vacuum cleaner can be controlled to move along the current orientation using its LiDAR or visual SLAM navigation system until the closest distance to large debris particles reaches the increased second distance. After that, the maximum and minimum horizontal diameter regions of the debris contour area can be extracted again through image recognition to provide accurate parameters for matching the optimal clamping height.

[0093] It should be noted that the robot can check the distance in real time during the movement. For example, it can use a depth camera to measure the distance to the garbage 10 times per second, and stop moving when the distance reaches 1.44 meters.

[0094] This implementation method determines an initial second distance when the waste type label is a flexible waste label. The robot moves to this distance to acquire a reference area image. Image recognition is used to determine the second proportion of the waste outline region corresponding to the second distance in the reference area image. The ratio of the second proportion to the first proportion is calculated as the reference area adjustment proportion. If this proportion is less than a preset value, the robot is multiplied by the preset distance adjustment proportion to increase the second distance. The robot is then controlled to move to the increased second distance. The distance is dynamically adjusted through proportional feedback to avoid imaging problems caused by an improper initial second distance, thus improving the accuracy of the second distance setting. Precise adjustment of the second distance ensures that the flexible waste in the reference area image is clear and has a reasonable proportion, reducing feature point extraction errors and improving the accuracy of identifying opaque waste outline sub-regions. This, in turn, improves the accuracy of subsequent key area parameter identification. It also provides a reliable basis for the clamping height, reduces the probability of clamping deviation, improves clamping stability, and enhances the reliability of the overall cleaning process.

[0095] Figure 12 A flowchart illustrating the sixth machine vision-based large-particle debris cleaning control method provided in this application embodiment is shown below. Figure 12 As shown, in some implementations, in the above-mentioned S120, the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the garbage type label is obtained by cleaning the database, including S121 to S122. S121 to S122 will be explained in detail below.

[0096] S121. Determine the initial clamping height based on the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the waste type label. Obtain multiple historical successful clamping records corresponding to the region with the largest horizontal diameter, the region with the smallest horizontal diameter, and the waste type label from the cleaning database. Obtain multiple historical successful clamping heights from these records and determine the average height of these heights as the average historical successful clamping height.

[0097] Figure 13 A schematic diagram of the workflow of the sixth machine vision-based large particle debris cleaning control method provided in the embodiments of this application is shown below. Figure 13 As shown, in this implementation, the initial clamping height can be calculated by combining the height of the area with the largest horizontal diameter of the outline (corresponding to the wider part of the waste) and the height of the area with the smallest horizontal diameter of the outline (corresponding to the narrower part of the waste), and adapting to the physical characteristics of the waste type.

[0098] For example, if the maximum area height of the horizontal diameter of the outline is 30cm and the minimum area height is 10cm, and the type of waste is "hard plastic bottle" (non-flexible), the initial clamping height can be determined to be 20cm; if the type of waste is "plush toy" (flexible), the initial clamping height can be adjusted to 15cm.

[0099] In this implementation, the cleaning database stores historical data of successful gripping by the sweeping robot. Each record contains four sets of related parameters: the maximum height of the horizontal diameter area of ​​the outline, the minimum height of the horizontal diameter area of ​​the outline, the garbage type label, and the successful gripping height. The three sets of parameters of the current garbage can be used as query conditions to match multiple historical successful records of the same type and outline range from the database.

[0100] In this implementation, the "successful clamping height" field of each record can be extracted from the matched historical records, and then the average value of these heights can be calculated by the arithmetic mean method. The average value is a summary of effective experience in similar past scenarios and can reflect the stable clamping position of this type of garbage.

[0101] For example, if 5 historical records are retrieved, with successful clamping heights of 18cm, 20cm, 22cm, 19cm, and 21cm respectively, the average historical successful clamping height can be calculated to be 20cm.

[0102] S122. Determine the median height between the average historical successful clamping height and the initial clamping height as the optimal clamping height.

[0103] In this implementation, the median height between the average historical successful clamping height and the initial clamping height can be determined. The median height is the arithmetic mean of the two, which can take into account both the real-time contour characteristics of the current garbage (initial height) and past successful experience (average historical height), avoiding the bias of a single data source.

[0104] For example, if the initial clamping height is 20cm and the average historical successful clamping height is 20cm, the intermediate height is 20cm; if the initial clamping height is 15cm and the average historical successful clamping height is 25cm, the intermediate height is 20cm.

[0105] This implementation method retrieves multiple historical successful gripping records from the cleaning database, extracts the heights of multiple historical successful gripping records, and determines the average height as the average historical successful gripping height. The midpoint between this average height and the initial gripping height is taken as the optimal gripping height. By combining the initial height determined by real-time parameters and the average height of historical successful data, the bias of relying on a single factor is avoided, and the accuracy of the optimal gripping height is improved. The accurate optimal gripping height allows the robotic arm to conform to the height characteristics of large particles of waste when gripping, reducing problems such as gripping slippage or unstable gripping caused by improper height, and improving the gripping success rate.

[0106] In this implementation, the initial clamping height is based on the key parameters of the current waste outline, supported by real-time data; the average historical successful clamping height comes from multiple historical successful records and has been verified to be reliable in practice. The combination of the two determines the optimal clamping height, taking into account both real-time adaptability and historical experience, thereby enhancing the clamping reliability of the method under different waste scenarios.

[0107] In some implementations, S120 above, by cleaning the database, obtains the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the garbage type label, and also includes S123 to S124. S123 to S124 will be explained in detail below.

[0108] S123. By cleaning the database, obtain multiple historical failed clamping records corresponding to the region with the largest horizontal diameter of the outline, the region with the smallest horizontal diameter of the outline, and the waste type label. Obtain multiple historical failed clamping heights from the multiple historical failed clamping records, and determine the average height of the multiple historical failed clamping heights as the average historical failed clamping height.

[0109] In this implementation, the maximum horizontal diameter area of ​​the current waste outline, the minimum horizontal diameter area of ​​the current waste outline, and multiple historical failed clamping records matching the waste type label can be obtained from the cleaning database. These records are previous cases of unsuccessful clamping of similar waste, which can clearly identify the clamping height range that is prone to failure.

[0110] In this implementation, the corresponding historical failure clamping height is extracted from multiple historical failure clamping records, and the average value of these heights is calculated to obtain the average historical failure clamping height. This value is a statistical summary of the historical failure heights, providing a basis for subsequent failure range avoidance.

[0111] For example, if the current waste is a hard plastic bottle with a maximum horizontal diameter of 10cm and a minimum of 5cm, and the waste type label is "hard plastic bottle", five historical failure records are retrieved from the database, with failure heights of 8cm, 9cm, 7cm, 8cm, and 9cm respectively. The average historical failure clamping height is calculated to be 8.2cm.

[0112] S124. Determine the midpoint between the average historical successful clamping height and the initial clamping height as the corrected clamping height. When the corrected clamping height is between the average historical successful clamping height and the average historical failed clamping height, determine the height difference between the corrected clamping height and the average historical failed clamping height as the corrected height difference. When the corrected height difference is less than the preset corrected height difference, adjust the corrected clamping height towards the position corresponding to the average historical successful clamping height to obtain the preset height as the optimal clamping height.

[0113] In this implementation, the median value between the average historical successful clamping height and the initial clamping height can be used as the corrected clamping height. This step combines real-time garbage parameters and historical successful experience to make the height more in line with actual clamping needs.

[0114] For example, if the average historical successful clamping height is 10cm and the initial clamping height is 12cm, the midpoint of the two, 11cm, is the corrected clamping height.

[0115] In this implementation, when the corrected clamping height is between the average historical successful clamping height and the average historical failed clamping height, the difference between the corrected clamping height and the average historical failed clamping height can be calculated to obtain the corrected height difference. This difference can determine whether the corrected clamping height is close to the height range prone to failure.

[0116] In this implementation, when the corrected height difference is less than the preset corrected height difference, the corrected clamping height can be adjusted to the preset height in the direction of the average historical successful clamping height. The adjusted height is the optimal clamping height, which can effectively avoid historical failure heights and improve the reliability of clamping.

[0117] For example, if the preset correction height difference is 1cm, and the average historical successful clamping height is 11.5cm, the corrected clamping height is 10.5cm, and the average historical failed clamping height is 10cm, the correction height difference is 0.5cm, which is less than the preset correction height difference. The corrected clamping height is then adjusted by 0.5cm in the direction of the average historical successful clamping height of 10cm to obtain the optimal clamping height of 11cm.

[0118] This implementation method obtains multiple historical failed clamping records corresponding to the region with the largest horizontal diameter of the outline, the region with the smallest horizontal diameter of the outline, and the waste type label. Multiple historical failed clamping heights are extracted, and the average height is determined as the average historical failed clamping height. The median height between the average historical successful clamping height and the initial clamping height is used as the corrected clamping height. The height difference between the corrected clamping height and the average historical failed clamping height is calculated. If the difference is less than a preset value, the corrected clamping height is adjusted towards the average historical successful clamping height to obtain the optimal clamping height. By avoiding historical failed height ranges, the accuracy of the optimal clamping height is improved. The optimal clamping height is double-verified and corrected using historical success and failure data, which better matches the height characteristics of large-particle waste, effectively avoiding heights prone to failure, reducing problems such as clamping detachment and displacement, and significantly improving the clamping success rate.

[0119] In some implementations, in S121 above, the initial clamping height is determined based on the region with the largest horizontal diameter of the outline, the region with the smallest horizontal diameter of the outline, and the waste type label, including S121a to S121b. S121a to S121b will be explained in detail below.

[0120] S121a, Obtain the preset clamping area corresponding to the garbage type label.

[0121] In this implementation, the preset clamping area corresponding to the current waste type label can be obtained by querying the pre-built waste type-clamping area mapping relationship. This mapping relationship is established based on the physical characteristics of different wastes (such as hardness and morphological stability) and successful data from multiple clamping experiments. For example, the preset clamping area for spherical waste is usually configured as the area with the largest horizontal diameter of the outline; the preset clamping area for vertically placed dumbbell-shaped waste is usually configured as the area with the smallest horizontal diameter of the outline. This operation can directly associate the waste type with the appropriate clamping area, providing a clear basis for determining the initial clamping height.

[0122] It should be noted that the waste type label here can be linked to the shape of the waste.

[0123] S121b: When the preset clamping area is the area with the largest horizontal diameter of the contour, the initial clamping height is determined to be the height corresponding to the area with the largest horizontal diameter of the contour. When the preset clamping area is the area with the smallest horizontal diameter of the contour, the initial clamping height is determined to be the height corresponding to the area with the smallest horizontal diameter of the contour.

[0124] In this implementation, when the preset clamping area is the area with the largest horizontal diameter of the outline, the height corresponding to this area can be directly determined as the initial clamping height. The area with the largest horizontal diameter of the outline is the part with the widest lateral dimension of the waste and is also the most stable position of the structure. The largest diameter area of ​​spherical waste is usually distributed in the middle or near the center of gravity. Clamping at this height can maintain clamping stability by relying on the structural strength of the waste itself and reduce the risk of slippage.

[0125] For example, if the maximum diameter area of ​​the spherical waste is located in the middle, 18cm from the ground, the initial clamping height is set to 18cm to ensure that the clamping point is on the solid part of the spherical waste.

[0126] In this implementation, when the preset clamping area is the area with the smallest horizontal diameter of the outline, the height corresponding to this area can be determined as the initial clamping height. The area with the smallest horizontal diameter of the outline is the part with the narrowest horizontal dimension of the waste. For vertically placed dumbbell-shaped types of waste, the waste accumulation in this area is more concentrated and the density is higher, which can provide greater friction when clamping and avoid the waste from scattering due to the loose clamping area.

[0127] For example, the smallest diameter area of ​​the dumbbell-shaped type of waste is located in the middle, 6cm from the ground. At this time, the initial clamping height is set to 6cm, which makes it easier to grasp the compact part of the dumbbell-shaped type of waste and lays a reliable foundation for subsequent adjustment of the optimal clamping height.

[0128] This implementation obtains the preset clamping area corresponding to the waste type label. If the preset clamping area is the area with the largest horizontal diameter, the height corresponding to the area with the largest horizontal diameter is determined as the initial clamping height; if the preset clamping area is the area with the smallest horizontal diameter, the height corresponding to the area with the smallest horizontal diameter is determined as the initial clamping height. By associating the preset clamping area with the type of large-particle waste, the ambiguity of the initial clamping height setting is avoided, and the accuracy of the initial clamping height is improved. Different waste types are adapted to different preset clamping areas, making the initial clamping height more in line with the actual shape characteristics of the waste, laying the foundation for obtaining a more adaptable optimal clamping height in the future, and improving the adaptability of clamping.

[0129] This application also provides a sweeping robot, including a unit for implementing the above-described machine vision-based large particle debris sweeping control method.

[0130] Figure 14 This is a schematic diagram of the logical structure of a sweeping robot provided in an embodiment of this application, as shown below. Figure 14 As shown, the robotic vacuum cleaner 4 in this embodiment includes a processing unit 41, a storage unit 42, and a transceiver unit 43. The processing unit 41 is used to process data, the storage unit 42 is used to store data, and the transceiver unit 43 is used to send and receive data. The processing unit 41, the storage unit 42, and the transceiver unit 43 cooperate with each other to implement the above-described method. The beneficial effects of this embodiment have been explained in the above-described method and will not be repeated here.

[0131] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0134] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A machine vision-based control method for sweeping large-particle litter, characterized by, The method includes: The system acquires images of the cleaned area containing large particles of waste through machine vision; it identifies the waste outline regions and waste type labels in the cleaned area images through image recognition; it also identifies the regions with the largest and smallest horizontal diameters of the waste outline regions through image recognition; wherein, the regions with the largest and smallest horizontal diameters of the outline regions include the region size and the region height relative to the ground. By using a cleaning database, the optimal gripping height of the tendon-driven robotic arm corresponding to the region with the largest horizontal diameter and the region with the smallest horizontal diameter, as well as the type of debris, is obtained. The tendon-driven robotic arm is then controlled to grip and clean large particles of debris corresponding to the cleaning area image according to the optimal gripping height. The tendon-driven robotic arm includes two tendon-driven mechanical grippers located on the outer wall of the robot vacuum cleaner. The tendon-driven robotic arm can move up and down along the height direction of the robot vacuum cleaner under the drive of the vertical drive component. Obtain the optimal gripping height of the tendon-driven manipulator corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the waste type label, including: Based on the maximum horizontal diameter area of ​​the outline, the minimum horizontal diameter area of ​​the outline, and the waste type label, determine the initial clamping height; through the cleaning database, obtain multiple historical successful clamping records corresponding to the maximum horizontal diameter area of ​​the outline, the minimum horizontal diameter area of ​​the outline, and the waste type label; obtain multiple historical successful clamping heights from multiple historical successful clamping records, and determine the average height of multiple historical successful clamping heights as the average historical successful clamping height; Determine the median height between the average historical successful clamping height and the initial clamping height as the optimal clamping height; By cleaning the database, the optimal gripping height of the tendon-driven robotic arm corresponding to the region with the largest horizontal diameter of the contour, the region with the smallest horizontal diameter of the contour, and the waste type label is obtained. This also includes: By cleaning the database, we obtain multiple historical failed clamping records corresponding to the region with the largest horizontal diameter of the outline, the region with the smallest horizontal diameter of the outline, and the waste type label; we obtain multiple historical failed clamping heights from multiple historical failed clamping records, and determine the average height of multiple historical failed clamping heights as the average historical failed clamping height. Determine the midpoint between the average historical successful clamping height and the initial clamping height as the corrected clamping height. When the corrected clamping height is between the average historical successful clamping height and the average historical failed clamping height, determine the height difference between the corrected clamping height and the average historical failed clamping height as the corrected height difference. When the corrected height difference is less than the preset corrected height difference, adjust the corrected clamping height towards the position corresponding to the average historical successful clamping height to obtain the preset height as the optimal clamping height.

2. The method according to claim 1, characterized in that, Image recognition is used to determine the regions with the largest and smallest horizontal diameters corresponding to the garbage contour regions, including: When the waste type label is a flexible waste label, the nearest distance between the robot vacuum cleaner and the large waste particles corresponding to the waste outline area is obtained as the first distance; the robot vacuum cleaner is controlled to move along the current orientation to increase the nearest distance between the robot vacuum cleaner and the large waste particles to the second distance, and the reference area image corresponding to the second distance is obtained through machine vision; Image recognition is used to determine multiple reference region feature points corresponding to the reference region image; multiple garbage contour sub-regions are determined to be in the garbage contour region; within the garbage contour sub-region, the number of matching feature points that match the positions of multiple reference region feature points within the garbage contour sub-region is determined as the number of matching feature points; the ratio of the number of matching feature points corresponding to each garbage contour sub-region to the number of reference region feature points is determined as the transparency of each garbage contour sub-region. In multiple garbage outline sub-regions, identify multiple opaque garbage outline sub-regions whose transparency is less than the preset transparency of garbage outline sub-regions; use image recognition to identify the region with the largest horizontal diameter and the region with the smallest horizontal diameter of the total region corresponding to the multiple opaque garbage outline sub-regions.

3. The method according to claim 2, characterized in that, The method further includes: Determine the number of adjacent edges between each opaque garbage outline sub-region and its neighboring opaque garbage outline sub-regions, and use this as the adjacency factor for the opaque garbage outline sub-region. Multiple opaque garbage contour sub-regions with an adjacency factor greater than 1 are identified as multiple adjacent opaque garbage contour sub-regions; the region with the largest horizontal diameter and the region with the smallest horizontal diameter of the total region corresponding to the multiple adjacent opaque garbage contour sub-regions are determined by image recognition.

4. The method according to claim 3, characterized in that, Image recognition is used to determine the regions with the largest and smallest horizontal diameters corresponding to the garbage contour regions, including: When the waste type label is a non-flexible waste label, the maximum and minimum horizontal diameter regions of the waste outline area are determined through image recognition.

5. The method according to claim 4, characterized in that, The method further includes: The proportion of the garbage outline region corresponding to the first distance in the image of the cleaning area is determined by image recognition. The second distance is determined based on the flexible waste label, the first distance, and the first proportion.

6. The method according to claim 5, characterized in that, The method further includes: Determine the garbage contour region in the reference region image, and determine the second proportion of the garbage contour region corresponding to the second distance in the reference region image through image recognition; Determine the ratio of the second ratio to the first ratio as the reference area adjustment ratio; when the reference area adjustment ratio is greater than or equal to the preset reference area adjustment ratio, multiply the second distance by the preset distance adjustment ratio to increase the second distance; control the sweeping robot to move along the current orientation to increase the closest distance between the sweeping robot and large particles of debris to the increased second distance, and determine the maximum and minimum horizontal diameter regions of the contour corresponding to the increased second distance; wherein, the preset distance adjustment ratio is greater than 1.

7. The method according to claim 6, characterized in that, Based on the area with the largest horizontal diameter of the outline, the area with the smallest horizontal diameter of the outline, and the waste type label, determine the initial clamping height, including: Obtain the preset clamping area corresponding to the waste type label; When the preset clamping area is the area with the largest horizontal diameter of the contour, the initial clamping height is determined to be the height corresponding to the area with the largest horizontal diameter of the contour; when the preset clamping area is the area with the smallest horizontal diameter of the contour, the initial clamping height is determined to be the height corresponding to the area with the smallest horizontal diameter of the contour.

8. A robotic vacuum cleaner, characterized in that, Includes units for implementing the method of any one of claims 1 to 7.

Images