Object recognition method and device, mobile equipment and readable storage medium

Abstract

The invention provides an object recognition method and device, mobile equipment and a readable storage medium, and relates to the field of smart home. The object recognition method is applied to the movable equipment, the movable equipment comprises a mechanical arm used for clamping an object, and the object recognition method comprises the steps that object information of a target object in a to-be-cleaned environment is obtained, object recognition is conducted according to the object information, and an object recognition result is obtained; and according to the object identification result, determining the object category of the target object, and judging the clamping condition of the target object relative to the movable equipment. According to the embodiment of the invention, not only is the object judgment based on the confidence output by the object recognition model, but also the clamping condition of the target object can be further judged while or after the object recognition is carried out based on the object recognition model, so that the object recognition accuracy is improved, and the user experience is improved. The success rate of object clamping of the mechanical arm of the movable equipment is increased, and the risk of damaging the object or a clamping jaw of the mechanical arm is reduced.

Description

Technical Field

[0001] The present application relates to the field of smart homes, and in particular to an object recognition method, device, removable device, and readable storage medium. Background Art

[0002] In related technologies, mobile devices are equipped with sensors to collect information about gripping points in the environment. Based on this information, object recognition models are then used to identify objects. However, model predictions can also be inaccurate. For example, when the model is not highly accurate or the data is unseen, simply making a simple judgment based on the confidence level of the model output can lead to incorrect predictions, reducing the accuracy of object recognition. This can lead to gripping failures when the mobile device's robotic arm is trying to grip an object, or even damage to the object or the robotic arm's gripper. Summary of the Invention

[0003] In view of this, the present application provides an object recognition method, apparatus, removable device and readable storage medium to achieve accurate object recognition.

[0004] In a first aspect, an embodiment of the present application provides an object recognition method, which is applied to a movable device, wherein the movable device includes a robotic arm for gripping an object, and the method includes:

[0005] Acquiring object information of a target object in the environment to be cleaned, and performing object recognition based on the object information to obtain an object recognition result;

[0006] According to the object recognition result, the object category of the target object is determined, and the gripping condition of the target object relative to the movable device is judged.

[0007] In a second aspect, an embodiment of the present application provides an object recognition device, which is applied to a movable device, wherein the movable device includes a robotic arm for gripping an object, and the device includes:

[0008] An information acquisition module, used to acquire object information of a target object in the environment to be cleaned;

[0009] An object recognition module is used to perform object recognition based on the object information and obtain an object recognition result;

[0010] The gripping judgment module is used to determine the object category of the target object according to the object recognition result, and to judge whether the target object can be gripped relative to the movable device.

[0011] In a third aspect, an embodiment of the present application provides a removable device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0013] In an embodiment of the present application, object information of a target object in the environment to be cleaned is obtained, the object information is input into a pre-built object recognition model, and an object recognition result is output, which includes the recognized object category and the confidence level corresponding to each object category. The category of the recognized target object is determined based on the object recognition result, and the gripping condition of the target object relative to the movable device is further judged. In an embodiment of the present application, the object is not only judged based on the confidence level output by the object recognition model, but at the same time or after the object recognition is performed based on the object recognition model, the gripping condition of the target object is further judged, thereby improving the accuracy of object recognition, improving the success rate of the robotic arm of the movable device in gripping objects, and reducing the risk of damaging the object or the robotic arm's gripper.

[0014] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A schematic diagram of the structure of a mobile device according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing a clamping process of a movable device according to an embodiment of the present application;

[0018] Figure 3 A schematic diagram showing a flow chart of an object recognition method according to an embodiment of the present application is shown;

[0019] Figure 4 The figure shows a structural block diagram of an object recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0021] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0022] The object recognition method, apparatus, removable device, and readable storage medium provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other unless there is a conflict.

[0023] Mobile devices exist in indoor spaces and can be intelligent cleaning devices, including sweepers, mops, sweeper-mop machines, household robots, etc. Figure 1 As shown, the movable device includes a chassis 101 , running wheels 102 , universal wheels 103 and a mechanical arm 104 installed on the chassis 101 .

[0024] In one embodiment, the chassis 101 can achieve linear motion along the x-axis and y-axis directions, linear motion along the z-axis, and rotation along the z-axis via the running wheels 102 and the universal wheels 103. The linear motion of the chassis 101 along the z-axis, i.e., the height adjustment of the chassis 101 from the ground, can be achieved via the running wheels 102 and / or the universal wheels 103. The running wheels 102 and the universal wheels 103 can be raised or lowered independently or together, thereby adjusting the height of the chassis 101 from the ground.

[0025] The end of the movable device's robotic arm 104 is equipped with a gripper and a camera. The robotic arm 104 includes multiple rotating shafts and connecting rods connecting the rotating shafts. The number of rotating shafts determines the flexibility of the robotic arm 104. The robotic arm 104 can be a multi-degree-of-freedom robotic arm, such as a three-degree-of-freedom, four-degree-of-freedom, or five-degree-of-freedom robotic arm. The specific number of degrees of freedom is not specifically limited in this application.

[0026] The rotation shaft M1 can drive the robot arm 104 to rotate left or right as a whole ( Figure 1 The rotation direction of the rotating shaft M2 is the same as that of the rotating shaft M3 and the rotating shaft M4. The rotating shaft M1 and the rotating shaft M2 are mainly responsible for the robot arm 104 to exit and return to the warehouse. After the robot arm 104 exits the warehouse, the rotating shaft M1 and the rotating shaft M2 maintain a fixed angle. A warehouse body for recovering the robot arm 104 can be provided on the chassis 101. The robot arm 104 can be folded and stored in the warehouse body. This state is the robot arm 104 returning to the warehouse. When the robot arm is unfolded, it can be extended from the warehouse body. This state is the robot arm 104 exiting the warehouse. The rotating shaft M3 and the rotating shaft M4 determine the position of the gripper of the robot arm 104 in space. The rotating shaft M5 is the spin joint of the gripper, which determines the posture of the gripper.

[0027] In one embodiment, the movable device also includes a camera installed on the chassis 101 and / or a camera installed on the robotic arm 104, and a 3D ToF (3D Time of Flight, time-of-flight 3D imaging) installed on the chassis 101. The three-dimensional contour information of objects in the environment and object information such as color and texture are obtained through 3D ToF and the camera to realize object recognition. After the object is identified, the object can be clamped and moved with the assistance of the robotic arm 104.

[0028] Specifically, object recognition models, using 3D Time of Flight (ToF) and camera information, combined with AI algorithms such as deep learning, identify object categories. In object recognition models, confidence generally refers to the model's confidence in its predictions about an object. The model assigns a probability value to each object category, indicating how likely the model believes the input data belongs to that object category. Confidence can be a value between 0 and 1. If the confidence is close to 1, the model believes the box contains the target object. If the confidence is close to 0, the model believes the box likely does not contain the target object. For example, in an image classification task, an object recognition model may need to classify an input image as "slippers," "cat," or "socks." For a specific input image, the object recognition model might output the following probabilities: 0.70 for slippers, 0.25 for cats, and 0.05 for socks. In this example, the model has the highest confidence in the classification of "slippers," at 70%.

[0029] After identifying the object, it is accurately grasped. Figure 2 The figure shows the process of the end-of-arm gripper gripping a shoe. Point A on the shoe is the initial gripping point, and point B is the final gripping point. The arrow indicates the gripping direction of the end-of-arm gripper, and the gripping angle is the angle α between point A and point B and the horizontal. The gripping process can include the following steps:

[0030] (1) According to the shape of the part of the shoe to be grasped, determine the positions of the initial grasping point A and the final grasping point B and the initial grasping angle α. For example, the initial grasping point of the shoe is in the middle of the upper edge of the front side; (2) Dynamically plan the trajectory of the end gripper of the robot arm so that point C between the two fingers of the end gripper first reaches point A and then moves to point B; (3) Close the gripper, grasp the object, and then lift the end to the load-carrying moving position.

[0031] In the embodiment of the present application, the functions of the mobile device include:

[0032] (1) Auxiliary floor cleaning function

[0033] When traditional cleaning robots perform floor cleaning tasks, they often encounter many obstacles along their route. Cleaning robots with obstacle avoidance functions will actively avoid obstacles and maintain a certain distance while moving to reduce collisions and scrapes. However, this can result in missed cleaning of the ground below and around obstacles, resulting in a low cleaning coverage rate. Compared to traditional cleaning robots that can only avoid obstacles, resulting in large areas of missed cleaning, the mobile device of the present application can, with the assistance of a robotic arm, perform actions such as gripping and carrying obstacles to clean the ground below and around the obstacle, thereby improving the floor cleaning coverage rate and increasing user satisfaction with the floor cleaning function.

[0034] (2) Automatic sorting function

[0035] Objects may be randomly placed or dropped on the floor, making the floor at home messy. Therefore, in addition to cleaning the floor, the robot also needs to be able to automatically organize the floor. The mobile device of this application can automatically identify obstacles on the floor and mark their locations during floor cleaning or cruising. After cleaning, it can automatically sort and organize the obstacles on the floor, such as slippers placed at the door, toys placed in the children's room, and paper balls left in the trash can.

[0036] In addition, the mobile device of the present application may also have other functions for handling household chores, such as: cleaning skirting boards, real-time home monitoring, and pet companionship.

[0037] The embodiment of the present application provides an object recognition method, which is applied to a movable device, wherein the movable device includes a robotic arm, which can be used to grasp an object, such as Figure 3 As shown, the method includes:

[0038] Step 301 : Obtain object information of a target object in the environment to be cleaned, and perform object recognition based on the object information to obtain an object recognition result.

[0039] In this step, the three-dimensional contour information and object information such as color and texture of objects in the environment to be cleaned can be obtained through sensors such as 3D ToF and cameras of mobile devices, and the object information is input into a pre-built object recognition model to output the object recognition results, which include the recognized object categories and the confidence levels corresponding to each object category.

[0040] Step 302 : Determine the object category of the target object based on the object recognition result, and judge whether the target object can be gripped by the movable device.

[0041] In this step, the category of the identified target object is determined based on the object recognition result, and the gripping status of the target object relative to the movable device is further judged, for example, the gripping status of the target object is judged based on the object recognition result and / or the gripping point information of the target object.

[0042] The embodiment of the present application not only determines the object based on the confidence level output by the object recognition model, but also further determines the grippability of the target object at the same time or after object recognition based on the object recognition model, thereby improving the accuracy of object recognition, improving the success rate of the robotic arm of the movable device in gripping objects, and reducing the risk of damaging the object or the robotic arm gripper.

[0043] In one embodiment of the present application, based on the object recognition result, the object category of the target object is determined, and the gripping condition of the target object relative to the movable device is judged, including the following three solutions:

[0044] Solution (1) determines the object category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judges whether the target object is a grippable object. This method can comprehensively judge the object category from both positive and negative aspects by combining the confidence levels of grippable and non-grippable objects, thereby avoiding misidentification and the accidental grasping of pets, children, fragile objects, or dangerous items, thereby protecting the user's personal and property safety.

[0045] Solution (2) determines the target object category based on the object category in the object recognition result and the confidence level corresponding to each object category, and judges whether the target object meets the gripping conditions of the mobile device based on the gripping point information of the target object. This method can combine the gripping point information to judge whether the object of the determined object category can be gripped, thereby ensuring the success rate of gripping.

[0046] Solution (3) determines the category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judges whether the target object is a grippable object, and judges whether the target object is an object that meets the gripping conditions of the movable device based on the gripping point information of the target object. This method combines the effects of solution (1) and solution (2), and can comprehensively judge the object category from both positive and negative aspects to avoid the problem of misidentification. It can also judge whether the object that has been determined to be a grippable object category can be gripped based on the gripping point information, thereby ensuring the success rate of gripping.

[0047] The following are the introductions to the above three solutions:

[0048] For solution (1), after obtaining the object recognition result, the object category of the target object is identified based on the object recognition result, and whether the target object is a grippable object is determined.

[0049] In one embodiment of the present application, determining the object category of the target object and judging whether the target object is a grippable object based on the object category in the object recognition result and the confidence level corresponding to each object category includes:

[0050] Determine a whitelist object category and a blacklist object category in the object recognition result, wherein the whitelist object category indicates that the object category is removable and can be gripped by the mobile device, and the blacklist object category indicates that the object category is not removable and can be gripped by the mobile device;

[0051] The object category of the target object is determined according to the maximum confidence level in the whitelist object categories and the maximum confidence level in the blacklist object categories, and it is judged whether the target object is a grippable object.

[0052] In this embodiment, to ensure the safety of the user, family members, and pets and to avoid grabbing potentially damaged or dangerous items, it is necessary to list grippable and non-gripable objects based on empirical knowledge. Gripping objects should generally meet gripping feasibility requirements.

[0053] After obtaining the object recognition result, the object categories included in the object recognition result are divided into whitelist object categories and blacklist object categories, wherein the whitelist object category represents the object category that can be gripped by the mobile device, and the blacklist object category represents the object category that cannot be gripped by the mobile device. For example, as shown in Table 1,

[0054] Table 1

[0055]

[0056] Among them, fabrics include towels, socks and underwear, pets include dogs, cats, etc., sharp objects include knives, scissors and nails, and fragile objects include glass, ceramic items, etc.

[0057] Determine a maximum confidence level among the whitelist object categories and a maximum confidence level among the blacklist object categories. Furthermore, compare the maximum confidence level among the whitelist object categories with a corresponding first confidence threshold, and compare the maximum confidence level among the blacklist object categories with a corresponding second confidence threshold. Determine the object category of the target object based on the comparison results, and determine whether the target object is a grippable object.

[0058] In an embodiment of the present application, the object category of the target object is identified and whether the target object is a grippable object is determined based on the maximum confidence in the whitelist object category and the maximum confidence in the blacklist object category, thereby improving the reliability of object judgment.

[0059] In one embodiment of the present application, determining the target object of the target object based on the maximum confidence level in the whitelist object category and the maximum confidence level in the blacklist object category, and determining whether the target object is a grippable object includes:

[0060] Determine a maximum confidence level among the whitelist object categories, recorded as a first confidence level, and determine a maximum confidence level among the blacklist object categories, recorded as a second confidence level;

[0061] If the first confidence level is greater than or equal to a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, determining that the object category corresponding to the first confidence level is the object category of the target object, and the target object is a grippable object;

[0062] If the second confidence level is greater than or equal to the second confidence level threshold, determining that the object category corresponding to the second confidence level is the object category of the target object, and the target object is an ungrabable object;

[0063] If the first confidence level is less than a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, the target object is determined to be an ungrabable object.

[0064] In this embodiment, the maximum confidence level among the whitelist object categories is recorded as the first confidence level, and the maximum confidence level among the blacklist object categories is recorded as the second confidence level. For example, in the whitelist object categories of Table 1, the maximum value among a1, a2, and a3 is recorded as a, and in the blacklist object categories of Table 1, the maximum value among b1, b2, b3, and b4 is recorded as b. The first confidence threshold is recorded as A, and the second confidence threshold is recorded as B. In one embodiment, the first confidence threshold is greater than the second confidence threshold, thereby minimizing the risk of accidentally gripping ungrippable objects.

[0065] If a≥A and b<B, it indicates that the object is very likely to be a whitelist object category and very unlikely to be a blacklist object category. The object category of the target object is marked as the object category corresponding to a, and the target object is determined to be a grippable object.

[0066] If b ≥ B, there's a high probability that the object is a blacklisted object. In this case, regardless of the magnitude relationship between a and A, the target object's object category is marked as that of b, and the target object is determined to be non-grippable. In other words, regardless of the size of a, as long as b indicates a high probability of the object being non-grippable, the object is determined to be non-grippable, significantly reducing the risk of a mobile device accidentally gripping a non-grippable object.

[0067] If a<A, b<B, it is determined that the object is an unrecognized object, and the unrecognized object is determined as an ungrabable object, thereby reducing the risk of the movable device mistakenly grabbing the ungrabable object.

[0068] In the related art method, if the recognition result for a certain object is: a = 90%, b = 60%, since b < a, based on the principle of maximum confidence, the object is determined to be a grippable object, and the gripper of the movable device is controlled to grip the object. In the embodiment of the present application, however, a first confidence threshold A = 80% and a second confidence threshold B = 50% are set. If the recognition result for a certain object is: a = 90%, b = 60%, even if b < a, since b ≥ B, there is a high possibility that the object is a blacklisted object. To ensure the safety of the object or the gripper, the object is determined to be an ungrabable object.

[0069] In the embodiment of the present application, the target object category is determined to be a grippable object only when the maximum confidence of the object in the white list is greater than or equal to the first threshold confidence and the maximum confidence of the object in the black list is less than the second threshold confidence. In other cases, it is determined to be an unclippable object, making it easier for the detected object to be judged as an unclippable object, greatly reducing the risk of the movable device accidentally gripping an unclippable object, and ensuring the safety of the object or the gripper.

[0070] In one embodiment of the present application, the method further includes: acquiring first information, and adjusting the first confidence threshold and / or the second confidence threshold according to the first information;

[0071] Among them, the first information includes at least one of the following: object recognition results, security mode switch information of the mobile device; if the object recognition result is that a whitelist object category is identified in the environment to be cleaned, the first confidence threshold is reduced; if the object recognition result is that a blacklist object category is identified in the environment to be cleaned, the second confidence threshold is reduced; if it is determined that the mobile device is in security mode based on the security mode switch information of the mobile device, the second confidence threshold is reduced.

[0072] In this embodiment, the first confidence threshold and the second confidence threshold can be dynamically adjusted. For example, the first confidence threshold and the second confidence threshold can be dynamically increased or decreased based on information such as object recognition results and security mode switch information of the removable device.

[0073] For example, if a large number of objects identified in the user's home over a period of time are whitelist objects, the first confidence threshold for determining which objects can be clamped will be appropriately lowered, making it easier for the detected objects to be identified as clampable objects, allowing the mobile device to clamp more objects indoors. If blacklist objects such as pets or toys have been identified in the user's home, the second confidence threshold for determining which objects cannot be clamped will be appropriately lowered; if the user has enabled safety mode for the mobile device, the second confidence threshold for determining which objects cannot be clamped will be appropriately lowered. In this way, the detected objects will be more easily identified as non-clampable objects, ensuring the safety of the mobile device in clamping objects.

[0074] For solution (2), after obtaining the object recognition result, the object category of the target object is identified based on the object recognition result, and based on the clamping point information of the target object, it is determined whether the target object is an object that meets the clamping conditions of the movable device.

[0075] In one embodiment of the present application, the object category of the target object is determined based on the object category in the object recognition result and the confidence level corresponding to each object category, and whether the target object satisfies the gripping conditions of the movable device is determined based on the gripping point information of the target object, including:

[0076] The object category corresponding to the maximum confidence in the object recognition result is used as the object category of the target object;

[0077] Obtaining gripping point information of the target object, the gripping point information of the target object including size information of the gripping point of the target object and the distance between the gripping point and the lower edge of the target object when the target object is lifted to the loaded moving position;

[0078] Based on the size information and the distance, it is determined whether the target object satisfies the gripping conditions of the movable device.

[0079] In this embodiment, the object category corresponding to the maximum confidence in the object recognition result is used as the object category of the target object. After determining the target object, a clamping point is selected on the target object, and the size information of the height and width of the clamping point, as well as the distance between the clamping point and the lower edge of the target object when the target object is lifted to the loaded moving position, are determined, wherein the clamping point height refers to the height of the clamping point relative to the ground, and the clamping point width refers to the distance on both sides of the clamping point, and the line connecting the two sides of the clamping point is perpendicular to the clamping angle direction. Furthermore, based on the size information and distance, it is determined whether the target object is an object that meets the clamping conditions of the movable device, that is, whether the target object can be grasped by the movable device.

[0080] The embodiment of the present application can combine the gripping point information to determine whether an object of a determined object category can be gripped, thereby ensuring the success rate of gripping.

[0081] In one embodiment of the present application, the size information includes the clamping point height and the clamping point width; based on the size information and the distance, it is determined whether the target object is an object that meets the clamping conditions of the movable device, including: if the clamping point height is greater than or equal to the first height threshold and less than or equal to the second height threshold, and the clamping point width is greater than or equal to the first width threshold and less than or equal to the second width threshold, and the distance is less than or equal to the distance threshold, then the target object is an object that meets the clamping conditions of the movable device.

[0082] In this embodiment, if the height of the clamping point satisfies the first condition, the width of the clamping point satisfies the second condition, and the distance between the clamping point and the lower edge of the target object when the target object is lifted to the loaded moving position satisfies the third condition, it is determined that the target object can be grasped by the movable device.

[0083] Among them, the first condition can be that the height of the clamping point is greater than or equal to the first height threshold and less than or equal to the second height threshold, the first height threshold is related to the end accuracy of the robotic arm of the movable device, and the second height threshold is related to the height of the robotic arm of the movable device, the working radius of the robotic arm and the flexible movement distance of the robotic arm; the second condition can be that the width of the clamping point is greater than or equal to the first width threshold and less than or equal to the second width threshold, the first width threshold is related to the tolerance distance when the clamping claw of the robotic arm of the movable device is closed, and the second width threshold is related to the clamping claw stroke of the robotic arm of the movable device; the third condition can be that the distance is less than or equal to the distance threshold, and the distance threshold is related to the minimum distance from the clamping point to the sensor of the movable device when the target object is lifted to the load-carrying moving position.

[0084] In an embodiment of the present application, when the height of the clamping point, the width of the clamping point, and the distance between the clamping point and the lower edge of the target object when the target object is lifted to the loaded moving position all meet the corresponding conditions, it is determined that the target object can be grasped by the movable device, thereby improving the accuracy of object grasping and identification, and ensuring the success rate of the movable device in grasping the target object.

[0085] In one embodiment of the present application, the method further comprises:

[0086] Determining a first height threshold according to the end precision of the manipulator arm of the movable device;

[0087] Determining a second height threshold based on the height of the manipulator arm of the movable device, the working radius of the manipulator arm, and the flexible movement distance of the manipulator arm;

[0088] determining a first width threshold value based on a tolerance distance when a gripper of a manipulator arm of the movable device is closed;

[0089] determining a second width threshold according to a gripper stroke of a manipulator arm of the movable device;

[0090] The distance threshold is determined based on the minimum distance between the gripping point and the sensor of the movable device when the target object is lifted to the loaded moving position.

[0091] In this embodiment, the first height threshold is determined based on the accuracy of the end-arm of the mobile device, where this accuracy includes the cumulative accuracy of the mobile device's chassis and the arm. For example, if the end-arm accuracy is 0.5 cm, to prevent the end-arm from contacting and damaging the ground during grasping, the gripping point height cannot be lower than 0.5 cm. That is, the first height threshold H1 = 0.5 cm, requiring the gripping point height h ≥ H1.

[0092] The second height threshold is determined based on the height of the movable arm, the radius of the arm's working range, and the range of flexible movement of the arm. The arm height refers to the height of the end of the arm opposite the gripper relative to the bottom surface. For example, if the radius of the movable arm's working range is 30 cm and the flexible movement space is 10 cm, the height of the gripping point cannot be higher than 20 cm above the arm's position H. In other words, the second height threshold H2 = H + 20 cm, requiring the gripping point height h ≤ H2.

[0093] The first width threshold is determined based on the tolerance distance of the closed gripper of the mobile device's robotic arm. For example, if the distance between the two fingers of the robotic arm's closed gripper is designed to be 0, and the tolerance is ±2mm, to prevent the gripper from failing to pick up thin objects and to ensure that the object does not drop after being picked up, the first width threshold W1 should be greater than or equal to 2mm, or can also be greater than or equal to 0mm. The gripping point width w ≥ W1 is required.

[0094] The second width threshold is determined based on the gripper stroke of the mobile device's manipulator. The gripper stroke of the manipulator is W2, and the gripping point width w≤W2 is required. Otherwise, the gripping point width exceeds the gripper stroke and the manipulator gripper cannot grip the gripping point.

[0095] When the target object is lifted to the loaded mobile position, the minimum distance from the clamping point to the sensor of the movable device is used as the distance threshold. After the movable device clamps the object, the object cannot block the sensor on the chassis, otherwise it will affect the navigation and obstacle avoidance effects. Therefore, the distance threshold D is the minimum distance from the sensor of the movable device to the clamped point after the movable device clamps and lifts the object to the loaded mobile position. The distance d≤D between the clamping point and the lower edge of the target object when the target object is lifted to the loaded mobile position. In some embodiments, whether the sensor of the movable device is blocked can also be used to directly determine whether the object can be grasped by the movable device.

[0096] The embodiment of the present application comprehensively considers factors such as the accuracy of the end of the robotic arm, the working radius, the gripper stroke and closing tolerance, and the object size to determine whether the target object of a determined object type can be grasped, thereby improving the accuracy of object recognition and ensuring the success rate of the mobile device in grasping the target object.

[0097] For solution (3), the target object category is determined based on the object category in the object recognition result and the confidence level corresponding to each object category, and at the same time, it is determined whether the target object is a grippable object. After determining that the target object is a grippable object, it is further determined based on the gripping point information of the target object whether the target object is an object that meets the gripping conditions of the movable device, that is, whether the target object is capable of being gripped. In this way, when it is determined that the target object is a grippable object and meets the gripping conditions of the movable device, the movable device can be controlled to grip the target object, thereby improving the accuracy of object gripping recognition and ensuring the success rate of gripping.

[0098] In solution (3), the object category of the target object is determined based on the object category in the object recognition result and the confidence level corresponding to each object category, and at the same time, it is judged whether the target object is a grippable object. This is the same or similar to solution (1) and will not be repeated here. In solution (3), based on the gripping point information of the target object, it is judged whether the target object is an object that meets the gripping conditions of the movable device. This is the same or similar to solution (2) and will not be repeated here.

[0099] In one embodiment of the present application, user confirmation information on the object category of the target object is obtained, and model parameters of the object recognition model are adjusted according to the confirmation information. The object recognition model is used to perform object recognition based on the object information and output the object recognition result.

[0100] In this embodiment, after the target object's category is identified, the user can optionally confirm the object category. The user's confirmation of the object category is obtained. If the user confirms that the object category is correctly identified, the correctly identified object category is fed into the object recognition model for training, thereby adjusting the model parameters of the object recognition model to increase the confidence level of the correct object category. For example, if the object is identified as fabric and the user determines it is correctly identified, the value of a2 will be appropriately increased for the next recognition; if the object is identified as a pet and the user determines it is correctly identified, the value of b1 will be appropriately increased for the next recognition.

[0101] In this way, the model can be adjusted based on user confirmation information, thereby improving the confidence level of correctly identifying the object category, which is beneficial to the accuracy of subsequent object recognition.

[0102] In summary, the embodiments of the present application provide a method for comprehensively judging whether a certain object on the ground can be gripped by a household robot by combining the object category and gripping point information at the same time or after the object is identified, thereby ensuring that the object is grippable and can be gripped, and that the movement after gripping is unrestricted, thereby improving user satisfaction with movable equipment.

[0103] As a specific implementation of the above object recognition method, an embodiment of the present application provides an object recognition device, which is applied to a movable device. The movable device includes a robotic arm, which can be used to clamp objects. Figure 4 As shown, the object recognition device 400 includes: an information acquisition module 401 , an object recognition module 402 and a gripping judgment module 403 .

[0104] The information acquisition module 401 is used to obtain object information of a target object in the environment to be cleaned;

[0105] The object recognition module 402 is used to perform object recognition based on the object information and obtain an object recognition result;

[0106] The gripping determination module 403 is configured to determine the object category of the target object according to the object recognition result, and to determine whether the target object is grippable relative to the movable device.

[0107] Furthermore, the gripping judgment module 403 is specifically configured to:

[0108] Determining the object category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judging whether the target object is a grippable object; and / or,

[0109] The target object category of the target object is determined based on the object category in the object recognition result and the confidence level corresponding to each object category, and whether the target object is an object that meets the gripping conditions of the movable device is determined based on the gripping point information of the target object.

[0110] Furthermore, the gripping judgment module 403 is specifically configured to:

[0111] Determine a whitelist object category and a blacklist object category in the object recognition result, wherein the whitelist object category indicates that the object category is removable and can be gripped by the mobile device, and the blacklist object category indicates that the object category is not removable and can be gripped by the mobile device;

[0112] The object category of the target object is determined according to the maximum confidence in the whitelist object categories and the maximum confidence in the blacklist object categories, and it is judged whether the target object is a grippable object.

[0113] Furthermore, the gripping judgment module 403 is specifically configured to:

[0114] Determine a maximum confidence level among the whitelist object categories, recorded as a first confidence level, and determine a maximum confidence level among the blacklist object categories, recorded as a second confidence level;

[0115] If the first confidence level is greater than or equal to a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, determining that the object category corresponding to the first confidence level is the object category of the target object, and the target object is a grippable object;

[0116] If the second confidence level is greater than or equal to the second confidence level threshold, determining that the object category corresponding to the second confidence level is the object category of the target object, and the target object is an ungrabable object;

[0117] If the first confidence level is less than a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, the target object is determined to be an ungrabable object.

[0118] Furthermore, the information acquisition module 401 is further configured to: acquire first information;

[0119] The device also includes:

[0120] an adjustment module, configured to adjust the first confidence threshold and / or the second confidence threshold according to the first information;

[0121] Among them, the first information includes at least one of the following: object recognition results, security mode switch information of the mobile device; if the object recognition result is that a whitelist object category is identified in the environment to be cleaned, the first confidence threshold is reduced; if the object recognition result is that a blacklist object category is identified in the environment to be cleaned, the second confidence threshold is reduced; if it is determined that the mobile device is in security mode based on the security mode switch information of the mobile device, the second confidence threshold is reduced.

[0122] Furthermore, the gripping judgment module 403 is specifically configured to:

[0123] The object category corresponding to the maximum confidence in the object recognition result is used as the object category of the target object;

[0124] Obtaining gripping point information of the target object, the gripping point information of the target object including size information of the gripping point of the target object and the distance between the gripping point and the lower edge of the target object when the target object is lifted to the loaded moving position;

[0125] Based on the size information and the distance, it is determined whether the target object satisfies the gripping conditions of the movable device.

[0126] Furthermore, the size information includes the height and width of the clamping point; the clamping judgment module 403 is specifically used to:

[0127] If the height of the clamping point is greater than or equal to the first height threshold and less than or equal to the second height threshold, and the width of the clamping point is greater than or equal to the first width threshold and less than or equal to the second width threshold, and the distance is less than or equal to the distance threshold, then the target object is an object that meets the clamping conditions of the movable device.

[0128] Furthermore, the device further includes: a threshold determination module, configured to:

[0129] Determining a first height threshold according to the end precision of the manipulator arm of the movable device;

[0130] Determining a second height threshold based on the height of the manipulator arm of the movable device, the working radius of the manipulator arm, and the flexible movement distance of the manipulator arm;

[0131] determining a first width threshold value based on a tolerance distance when a gripper of a manipulator arm of the movable device is closed;

[0132] determining a second width threshold according to a gripper stroke of a manipulator arm of the movable device;

[0133] The distance threshold is determined based on the minimum distance between the gripping point and the sensor of the movable device when the target object is lifted to the loaded moving position.

[0134] Furthermore, the information acquisition module 401 is further configured to: obtain confirmation information of the user regarding the object category of the target object;

[0135] The adjustment module is further used to adjust the model parameters of the object recognition model according to the confirmation information. The object recognition model is used to perform object recognition according to the object information and output the object recognition result.

[0136] The object recognition device 400 in the embodiment of the present application can be a mobile device or a component in a mobile device, such as an integrated circuit or a chip. Figure 1 To avoid repetition, the various processes implemented in the embodiment of the object recognition method are not described here.

[0137] An embodiment of the present application also provides a removable device, which includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor. When the program or instruction is executed by the processor, the various steps of the above-mentioned object recognition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0138] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0139] The processor may include one or more processing units; optionally, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and application programs, and the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor.

[0140] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned object recognition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0142] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for object recognition, characterized in that: Applied to a movable device comprising a robotic arm for gripping an object, the method comprises: Acquiring object information of a target object in the environment to be cleaned, and performing object recognition based on the object information to obtain an object recognition result; According to the object recognition result, the object category of the target object is determined, and the gripping condition of the target object relative to the movable device is judged.

2. The method according to claim 1, characterized in that Determining the object category of the target object according to the object recognition result, and judging whether the target object can be gripped relative to the movable device, including: Determining the object category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judging whether the target object is a grippable object; and / or, The target object category of the target object is determined based on the object category in the object recognition result and the confidence corresponding to each object category, and based on the clamping point information of the target object, it is judged whether the target object is an object that meets the clamping conditions of the movable device.

3. The method according to claim 2, characterized in that Determining the object category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judging whether the target object is a grippable object, includes: Determining a whitelist object category and a blacklist object category in the object recognition result, wherein the whitelist object category indicates that the object category is grippable by the movable device, and the blacklist object category indicates that the object category is not grippable by the movable device; The object category of the target object is determined according to the maximum confidence level in the whitelist object categories and the maximum confidence level in the blacklist object categories, and whether the target object is a grippable object is determined.

4. The method according to claim 3, characterized in that The determining the object category of the target object based on the maximum confidence level in the whitelist object categories and the maximum confidence level in the blacklist object categories, and judging whether the target object is a grippable object, includes: Determining a maximum confidence level among the whitelist object categories, recorded as a first confidence level, and determining a maximum confidence level among the blacklist object categories, recorded as a second confidence level; If the first confidence level is greater than or equal to a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, determining that the object category corresponding to the first confidence level is the object category of the target object, and the target object is a grippable object; If the second confidence level is greater than or equal to a second confidence level threshold, determining that the object category corresponding to the second confidence level is the object category of the target object, and the target object is an unclampable object; If the first confidence level is less than a first confidence level threshold, and the second confidence level is less than a second confidence level threshold, the target object is determined to be an ungrabable object.

5. The method according to claim 4, characterized in that The method further comprises: Acquire first information, and adjust the first confidence threshold and / or the second confidence threshold according to the first information; Among them, the first information includes at least one of the following: the object recognition result, the security mode switch information of the movable device; if the object recognition result is that the whitelist object category is recognized in the environment to be cleaned, then the first confidence threshold is reduced; if the object recognition result is that the blacklist object category is recognized in the environment to be cleaned, then the second confidence threshold is reduced; if it is determined that the movable device is in security mode based on the security mode switch information of the movable device, then the second confidence threshold is reduced.

6. The method according to claim 2, characterized in that The determining the object category of the target object based on the object category in the object recognition result and the confidence level corresponding to each object category, and judging whether the target object is an object that meets the gripping conditions of the movable device based on the gripping point information of the target object, includes: The object category corresponding to the maximum confidence level in the object recognition result is used as the object category of the target object; Acquiring gripping point information of the target object, the gripping point information of the target object including size information of the gripping point of the target object and a distance between the gripping point and a lower edge of the target object when the target object is lifted to a loaded moving position; According to the size information and the distance, it is determined whether the target object is an object that meets the gripping condition of the movable device.

7. The method according to claim 6, characterized in that The size information includes a gripping point height and a gripping point width; and judging whether the target object satisfies the gripping conditions of the movable device based on the size information and the distance includes: If the height of the clamping point is greater than or equal to the first height threshold and less than or equal to the second height threshold, and the width of the clamping point is greater than or equal to the first width threshold and less than or equal to the second width threshold, and the distance is less than or equal to the distance threshold, then the target object is an object that meets the clamping conditions of the movable device.

8. An object recognition device, characterized in that: Applied to a movable device, the movable device including a mechanical arm for gripping an object, the device comprising: An information acquisition module, used to acquire object information of a target object in the environment to be cleaned; An object recognition module is used to perform object recognition based on the object information and obtain an object recognition result; The gripping judgment module is used to determine the object category of the target object according to the object recognition result, and to judge whether the target object can be gripped relative to the movable device.

9. A movable device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction running on the processor, and when the program or instruction is executed by the processor, the steps of the object recognition method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the object recognition method according to any one of claims 1 to 7 are implemented.

Images