Robot control method, cleaning robot, cleaning system, and storage medium

Abstract

The application provides a robot control method, a cleaning robot, a cleaning system and a storage medium. The method comprises: identifying an object to be stored; obtaining state information of the object to be stored; determining whether the object to be stored is currently in a temporary non-stored state based on the state information; and controlling the robot not to perform a storage operation on the object to be stored if it is determined that the object to be stored is in the temporary non-stored state during the execution of a task by the robot. The above method can reduce the storage operation on the object that is not suitable for storage at present, and can improve the accuracy of the robot storage decision.

Description

Technical Field

[0001] This application relates to the field of robotics, and more particularly to a robot control method, a cleaning robot, a cleaning system, and a storage medium. Background Technology

[0002] In modern home environments, robots struggle to accurately distinguish between items that truly need storage and those that are currently in use, unsuitable for storage due to user habits, or area layout. This can lead to inaccurate storage operations. For example, if a user is relaxing on the sofa with slippers at their feet, removing them from the robot's storage will leave the user with nowhere to wear shoes when they get up. Similarly, if a user habitually keeps their frequently worn slippers in the entryway or at the bottom of the shoe cabinet for easy access, forcing them into the shoe cabinet by the robot will only increase the effort required to retrieve them, adding to the inconvenience. Summary of the Invention

[0003] In view of the above, it is necessary to provide a robot control method, a cleaning robot, a cleaning system, and a storage medium that can solve the technical problem of inaccurate robot storage decisions.

[0004] In a first aspect, this application provides a robot control method that identifies an object to be stored, obtains the status information of the object to be stored, determines whether the object to be stored is currently in a state of not being stored, and controls the robot not to perform a storage operation on the object to be stored during the execution of the task if it is determined that the object to be stored is in the state of not being stored.

[0005] In some embodiments of this application, the status information includes the current activity status information of the relevant subject and the association information between the object to be collected and the current activity; determining whether the object to be collected is currently in a state of not being collected based on the status information includes: when the current activity status information is not ended and the object to be collected is associated with the current activity, determining that the object to be collected is in the state of not being collected.

[0006] In some embodiments of this application, the object to be collected is associated with the current activity, including at least one of the following conditions: the object to be collected is located within a preset range of the relevant subject related to the current activity; the object to be collected is being used by the relevant subject in the current activity.

[0007] In some embodiments of this application, the current activity status information further includes at least one of activity type and activity execution status. The activity type is a static activity or a dynamic activity. The preset range related to the current activity corresponding to the static activity is smaller than the preset range related to the current activity corresponding to the dynamic activity. The activity execution status includes one of not started, not ended, or ended.

[0008] In some embodiments of this application, the status information further includes historical occurrence time information of the current activity; determining whether the object to be collected is currently in a state of not being collected based on the status information includes: when the current activity status information is not finished, the current time matches the historical occurrence time information, and the object to be collected is associated with the current activity, determining that the object to be collected is in the state of not being collected.

[0009] In some embodiments of this application, the status information includes a functional status; determining whether the object to be stored is currently in a temporarily unstored state based on the status information includes: when the functional status indicates that the object to be stored is performing a preset function, determining that the object to be stored is in the temporarily unstored state.

[0010] In some embodiments of this application, the status information further includes historical time information of the execution of the preset function; determining whether the object to be stored is currently in a state of not being stored based on the status information includes: when the function status indicates that the object to be stored is executing the preset function, and the current time matches the historical time information, determining that the object to be stored is in the state of not being stored.

[0011] In some embodiments of this application, the status information includes the user's historical placement and / or retrieval behavior of the object to be stored; determining whether the object to be stored is currently in a temporarily unstored state based on the status information includes: determining a commonly used area of ​​the object corresponding to the object to be stored based on the historical placement and / or the historical retrieval behavior, and determining that the object to be stored is in the temporarily unstored state when the object to be stored is located in the commonly used area of ​​the object.

[0012] In some embodiments of this application, determining the commonly used area of ​​the object corresponding to the object to be stored based on the historical placement behavior and / or the historical retrieval behavior includes: determining the area that meets the preset frequency condition as the commonly used area of ​​the object based on the placement frequency and / or retrieval frequency of the object to be stored within a preset historical time period.

[0013] In some embodiments of this application, the status information includes user-configured area configuration information; the area configuration information is used to indicate the storage exemption area corresponding to the object to be stored; determining whether the object to be stored is currently in a temporarily unstored state based on the status information includes: when the object to be stored is located in the storage exemption area, determining that the object to be stored is in the temporarily unstored state.

[0014] In some embodiments of this application, when it is determined that the object to be stored is in the state of not storing, the robot is controlled to perform cleaning operations on the area surrounding the object to be stored, but not to perform storage operations on the object to be stored.

[0015] In some embodiments of this application, when updated status information is obtained, the status of the object to be stored is re-determined based on the updated status information to determine whether the object is in the state of not being stored temporarily. Once it is determined that the object to be stored is no longer in the "not to be stored" state, the storage operation is performed on the object to be stored according to the task plan.

[0016] In some embodiments of this application, the robot constructs a semantic topology map during the execution of a task; the semantic topology map contains the state information of the object to be collected.

[0017] In some embodiments of this application, the robot determines the order of operations for different areas based on the heat distribution in the semantic topology map and / or the user-configured area preference information. The heat distribution in the semantic topology map is used to characterize the degree of operation demand in the different areas. The area preference information includes the user-preferred priority operation areas. The operation includes cleaning operations in the areas and / or storage operations for the objects to be stored.

[0018] In some embodiments of this application, for the different areas, the task priority corresponding to the area with a higher task demand is higher than the task priority corresponding to the area with a lower task demand; and / or, for the different areas, the task priority corresponding to the user's preferred priority task area is higher than the task priority corresponding to other areas; and / or, for the different areas, the task priority corresponding to the area with a higher historical task demand frequency is higher than the task priority corresponding to the area with a lower historical task demand frequency; and / or, for the same area, when the objects to be stored in the area are not in the "not stored" state, the priority of the storage operation is higher than the priority of the cleaning operation.

[0019] In some embodiments of this application, if the robot receives a correction instruction sent by the user during the execution of a task, it updates the attribute information of the object to be collected corresponding to the correction instruction in the semantic topology map according to the correction instruction.

[0020] In some embodiments of this application, during the execution of a task, the robot uses a full-body vision-language-action (VLA) model to perceive the relative distance between its current position and the target object, and decouples the chassis movement and the robotic arm trajectory movement according to the relative distance to approach and collect the target object, which includes objects to be collected that are not in the temporarily uncollected state.

[0021] In some embodiments of this application, in response to a task interruption and restart, the robot uses the latent space state encoded by the VLA model and the semantic topology map to identify the current task progress, backtracks to the previous safe breakpoint according to the task progress, and uses the safe breakpoint as the recovery starting point to continue executing the task. The safe breakpoint is the state corresponding to the atomic action unit that was completed before the task interruption.

[0022] Secondly, embodiments of this application provide a cleaning robot, the cleaning robot including a memory and a processor; the memory is used to store program instructions; the processor is used to read the program instructions stored in the memory to implement the control method described in the first aspect.

[0023] Thirdly, embodiments of this application provide a cleaning system, the cleaning system comprising: a cleaning robot, the cleaning robot being used to implement the control method described in the first aspect; and a base station, the base station having a docking position for the cleaning robot to dock at, the cleaning robot being selectively docked at the base station, the base station being used at least for maintaining the cleaning robot.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the control method described above.

[0025] In the robot control scheme provided in this application embodiment, the robot determines whether the object to be stored is in a state of temporary non-storage based on the object's status information. When the object is in a state of temporary non-storage, the robot is controlled not to perform a storage operation on the object. This reduces the inaccuracy of the robot's storage decisions and minimizes unnecessary movements. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an application scenario diagram of the robot control method provided in one embodiment of this application.

[0028] Figure 2 This is a schematic diagram of the structure of a robot provided in one embodiment of this application.

[0029] Figure 3 This is a flowchart of a robot control method provided in an embodiment of this application.

[0030] Figure 4 This is a schematic diagram of the structure of a control device provided in an embodiment of this application.

[0031] Figure 5 This is a schematic diagram of the structure of a robot provided in one embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c (seven cases).

[0035] In modern home environments, robots struggle to accurately distinguish between items that truly need storage and those that are unsuitable for storage due to being in use, user habits, or area layout. This can lead to incorrect storage operations. For example, if a user is relaxing on the sofa with slippers at their feet, removing them from the robot's storage will leave the user with nowhere to wear shoes. Similarly, if a user habitually stores their frequently worn slippers in the entryway or at the bottom of the shoe cabinet for easy access, forcing them into the shoe cabinet by the robot will only increase the effort required to retrieve them, creating inconvenience.

[0036] To address the aforementioned technical problems, this application provides a robot control method that can reduce inaccurate storage operations, thereby improving the accuracy of the robot's storage decisions.

[0037] In some embodiments of this application, according to the structural form, the robot can be a composite robot with a robotic arm mounted on a mobile chassis, which has the ability to move autonomously and operate precisely.

[0038] In some embodiments of this application, the robot can be classified according to its function and purpose as a cleaning robot, an automated guided vehicle (AGV), an inspection / cruising robot, or a logistics delivery robot.

[0039] Among them, cleaning robots can be used to perform cleaning operations on target surfaces such as floors and walls, and can also use robotic arms to perform storage operations on items. For example, cleaning robots can be sweeping robots, floor washing robots, all-in-one machines that combine sweeping and floor washing, disinfection robots, or polishing robots, etc.

[0040] Automated guided vehicles (AGVs) can be used to autonomously travel along preset paths in scenarios such as warehousing or production lines, and use robotic arms to achieve material handling and automatic loading and unloading.

[0041] Inspection / cruising robots can be used to autonomously inspect designated areas and can use robotic arms to perform gripping and intervention operations on abnormal objects.

[0042] Logistics delivery robots can move autonomously in scenarios such as parks or indoors, and use robotic arms to complete the gripping, loading and fixed-point delivery of items.

[0043] The examples of robots described above are merely illustrative and are not limited to specific types in actual applications. The embodiments of this application do not limit the specific types of robots.

[0044] For example, let's take a cleaning robot as an example to illustrate the concept. Figure 1 The diagram shown is an application scenario diagram of the robot control method provided in an embodiment of this application, and also a structural schematic diagram of the cleaning system provided in an embodiment of this application.

[0045] like Figure 1 As shown, the cleaning system 1000 includes a cleaning robot 100 and a base station 200. The base station 200 has a docking position for the cleaning robot 100 to dock. The base station 200 is used at least for maintaining the cleaning robot 100. The cleaning robot 100 can optionally dock with the base station 200. After use, the cleaning robot 100 can be placed on the docking position. The cleaning robot 100 can move to the docking position by itself, or it can be manually placed on the docking position by the user, facilitating the daily storage and maintenance of the cleaning robot 100. In some embodiments, when the cleaning robot 100 is located at the docking position of the base station 200, the base station 200 can perform maintenance on the cleaning robot 100. The types of maintenance include, but are not limited to, charging, dust collection, cleaning of cleaning components, replenishment of clean water, and pumping of wastewater. It can be understood that the cleaning robot 100 can perform at least one of the following tasks within the base station 200: 1. The base station 200 charges the cleaning robot 100; 2. The base station 200 collects the debris (e.g., debris from the cleaning robot 100's dust box or wastewater tank) into its dust collection container; 3. The base station 200 cleans the cleaning components of the cleaning robot 100 (e.g., washes the mop, washes the roller, cleans the roller brush, cleans the side brush, etc.); 4. The base station 200 replenishes the cleaning robot 100's clean water tank with clean water; 5. The base station 200 collects the dirt from the cleaning robot 100's wastewater tank into its wastewater container and discharges it to the outside. The above maintenance types are merely illustrative descriptions and are not intended to limit this application.

[0046] Figure 1The scenarios shown are merely illustrative examples, and the control method provided in this application can also be applied to other scenarios. For example, in some scenarios, the cleaning system 1000 may also include other types of devices, such as terminal devices, which are communicatively connected to the cleaning robot 100 to remotely control and manage the cleaning robot 100. This application does not limit the specific application scenarios of the control method.

[0047] To illustrate the structure of the robot, a cleaning robot will be used as an example below. Figure 2 As shown.

[0048] exist Figure 2 The cleaning robot 100 includes a robot body 11 and a robotic arm 12. The robot body 11 includes a movable chassis, and the robotic arm 12 is mounted on the upper surface of the robot body 11. The movable chassis allows for autonomous movement and the execution of cleaning operations such as sweeping and / or washing. The robotic arm 12 includes multiple joint motors, such as joint motors J1, J2, J3, and J4. Through the coordinated drive of these multiple joint motors, the robotic arm 12 can achieve multi-degree-of-freedom joint movements. The end effector of the robotic arm 12 is a gripper 121, which is a flexible spatial orientation adjustment driven by the multiple joint motors. This gripper performs operations such as grasping, clamping, pushing, and releasing to adapt to objects in different orientations and postures, thereby completing the storage operation.

[0049] Understandable Figure 2 The illustrated structure does not constitute a specific limitation on the cleaning robot 100. In other embodiments of this application, the cleaning robot 100 may include more or fewer components than illustrated, or combine some components, or separate some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0050] like Figure 3 The diagram shown is a flowchart of a robot control method according to an embodiment of this application. The control method is applied to a robot, for example... Figure 2 The cleaning robot 100 shown Figure 5 The robot 500 shown is an example. The robot can employ a full-body vision-language-action (VLA) model to implement the aforementioned control method. The VLA model can be a multimodal artificial intelligence model that integrates visual perception, language understanding, and motion control capabilities within a single framework, enabling the robot to directly generate and execute corresponding physical actions based on visual input and natural language commands. The control method includes the following steps: S11, Identify the objects to be stored.

[0051] In some embodiments of this application, the robot can utilize a VLA model, combined with the robot's perception system, to identify objects to be stored in the environment.

[0052] In some embodiments, the sensing system may include a visual sensor, which may be any one or more combinations of a monocular camera, a binocular camera, or a multi-camera system.

[0053] There are various methods for using a sensing system to identify objects to be stored.

[0054] In some embodiments, the robot can use a perception system to collect environmental images, process these images using one or more algorithms such as object detection, semantic segmentation, and instance segmentation, to determine the types of objects in the environment, match these object types with preset types, and identify the object as a storage object when a match is successful. The preset type refers to a pre-determined type of object that is allowed or requires the robot to perform storage operations; this type can be determined or updated by factory configuration, user settings, or task instructions. For example, the placement of large furniture such as sofas, where the user does not require the robot to adjust the position, can be set through user settings.

[0055] In other embodiments, the robot can use a perception system to collect environmental images, perform three-dimensional reconstruction based on the environmental images to obtain point cloud data, identify the types of objects in the environment based on the point cloud data, and match the object types with preset types to determine the objects to be collected.

[0056] In some embodiments, the preset type can be set according to the object category or for a specific object instance; the preset type can also distinguish between objects that are allowed to be automatically stored and objects that are prohibited from being automatically stored. After matching the object type, the robot can also determine whether to include the object as the object to be stored in the current task based on at least one of the object's size, weight, material, fragility, cord status, container status, and the operational capability of the end effector.

[0057] In some embodiments of this application, the environment can be the target scene where the robot is located, which can be an outdoor space or an indoor space, such as an entire house, a single room, or a designated area within a room.

[0058] In some embodiments of this application, the object to be stored can be any item that needs to be stored, including but not limited to daily necessities (such as clothing, slippers, glasses, thermos cups, backpacks), catering and kitchen utensils (such as bowls, plates, cups), stationery and office supplies (such as pens, books, documents), children's products (such as toys, crawling mats), pet supplies (such as food bowls, toy balls), and electronic products (such as mobile phones, remote controls, chargers), etc.

[0059] In some embodiments, the object to be stored may be an item that the robot determines may require storage operations based on the object type, user instructions, storage rules, or task plan. The robot can perform operations such as grasping, clamping, pushing, transporting, or placing the object to be stored through its end effector. The storage location or storage rules corresponding to the object to be stored can be preset or determined based on user habits or environmental information.

[0060] S12, obtain the status information of the object to be collected.

[0061] In some embodiments of this application, the methods for obtaining the state information of the object to be stored can be flexible and diverse. For example, the methods for a robot to obtain the state information of the object to be stored may include one or more of the following: obtaining it from a pre-built semantic topology map, and / or obtaining it through a VLA model combined with the robot's perception system to detect and / or identify the object to be stored, and / or obtaining it directly through user input, and / or obtaining it from stored historical records.

[0062] It should be noted that the status information of the object to be stored is information obtained for a specific object and used to make storage decisions; the semantic topology map is used to associate the recorded object, the area where the object is located, related entities, and the relationships between the above information. The robot can read the recorded status information from the semantic topology map, and can also obtain new status information based on real-time perception, user input, or historical records, and use the new status information to update the semantic topology map.

[0063] In some embodiments of this application, when relevant entities are involved, the status information of the object to be collected may include the current activity status information of the relevant entities and the association information between the object to be collected and the current activity.

[0064] The relevant subjects include at least one of users (people) or pets; and / or, the current activity includes at least one of rest activities, reading activities, office activities, cooking activities, meal preparation activities, pet interaction activities, or play activities; the current activity status information includes at least one of the activity type and activity execution status. The activity type is a static activity or a dynamic activity. For example, a static activity could be a user reading at a desk, while a dynamic activity could be a user playing ball with a pet in the living room, i.e., an activity with a large activity range. The activity execution status can include one of not started, not finished, or finished. The robot can determine the current activity status information based on at least one of the following: current time information, historical activity time information of the relevant subjects, location information, posture information, duration, and environmental feature information corresponding to the current activity; the environmental feature information can include at least one of the following: lighting conditions, scene type, spatial layout features, temperature and humidity information, or background sound features; the association information between the object to be stored and the current activity can include at least one of the following: the object to be stored is located in the area corresponding to the current activity, or the object to be stored is an object used in the current activity.

[0065] For example, when it is detected that a user is continuously sitting near a desk, facing the desk, and there are open books or computers on the desk, it can be determined that the user is engaged in reading or working activities; when the user is not detected in the corresponding activity area for a preset period of time, it can be determined that the activity has ended.

[0066] In this embodiment, based on the current activity status information of the relevant subject and the association information between the object to be collected and the current activity, the robot can determine whether the object to be collected is related to an activity that has not yet ended, thereby determining whether the object is not suitable for collection.

[0067] In some other embodiments of this application, in addition to the current activity status information of the relevant subject and the association information between the object to be collected and the current activity, the status information of the object to be collected may also include the historical occurrence time information of the current activity.

[0068] In this embodiment, further incorporating historical occurrence time information of the current activity can improve the reliability of judging the current activity and its associated objects to be collected.

[0069] In some other embodiments of this application, the status information of the object to be stored can indicate the user's historical placement and / or historical retrieval behavior of the object to be stored.

[0070] In this embodiment, based on the user's historical placement and / or retrieval behavior of the objects to be stored, the robot can not only learn and determine the user's storage habits, but also determine whether the current object is not suitable for storage.

[0071] In other embodiments of this application, in the absence of any relevant parties involved, the status information of the object to be stored may include a functional status, which can be used to indicate whether the object is currently performing a preset function. The preset function includes at least one of charging, running, playing music, and connecting or communicating with an external device. For example, the functional status may indicate that the object is in at least one of charging, power supply, running, music playback, or connecting or communicating with an external device.

[0072] In this embodiment, based on the functional state of the object to be stored, the robot can determine whether moving or storing the object may interrupt its ongoing function.

[0073] In some other embodiments of this application, in addition to the functional status, the status information of the object to be stored may also include historical time information of the object performing a preset function.

[0074] In this embodiment, by further incorporating historical time information of the object to be stored performing preset functions, the reliability of determining whether the object to be stored is currently performing a preset function can be improved.

[0075] In some other embodiments of this application, the status information of the object to be stored may include user-configured area configuration information, which is used to indicate the storage exemption area corresponding to the object to be stored.

[0076] The methods for obtaining area configuration information can be flexible and diverse. For example, the robot can communicate and connect with user terminal devices (such as mobile phones, watches, tablets, etc.) and obtain area configuration information using the area setting function of the application (APP) running on the user terminal device, thereby realizing personalized exemption area configuration. For example, through the area setting function, the user can manually select "bottom of the entrance hall" as "frequently worn shoe area".

[0077] The exemption area in this application is a spatial range determined by the robot based on the acquired state information, used to instruct the robot not to perform storage operations on the objects to be stored in the exemption area at present.

[0078] In this embodiment, based on the user-configured area configuration information, the robot can determine the storage exemption area corresponding to the object to be stored, so as to facilitate the subsequent execution of storage exemption in the storage exemption area.

[0079] S13, Based on the status information of the object to be stored, determine whether the object is currently in a state of not being stored.

[0080] The "not yet stored" state in this application is a storage decision state determined by the robot based on the acquired state information, which is used to instruct the robot not to perform storage operations on the corresponding objects to be stored at present.

[0081] In some embodiments of this application, when the object to be stored is being used by a relevant subject or is associated with a current activity of the relevant subject that has not yet ended, the state of not storing can also be called the temporary residence state; during the duration of the temporary residence state, no storage operation is triggered on the object to be stored.

[0082] In some embodiments of this application, when the status information includes the current activity status information of the relevant subject and the association information between the object to be collected and the current activity, the robot determines whether the object to be collected is currently in a state of not being collected based on the status information of the object to be collected. This includes: when the current activity status information is not finished and the object to be collected is associated with the current activity, determining that the object to be collected is in a state of not being collected.

[0083] In some embodiments, the object to be collected is associated with the current activity, including at least one of the following conditions: the object to be collected is located within a preset range of the relevant subject related to the current activity, or the object to be collected is being used by the relevant subject in the current activity.

[0084] Static activities can be activities performed by relevant subjects primarily in a fixed location, such as rest activities, reading activities, or office activities; dynamic activities can be activities performed by relevant subjects moving within a certain area, such as cooking activities, food preparation activities, pet interaction activities, or game activities. When the current activity is a static activity, the preset range related to the current activity can be smaller than the preset range related to the current activity when the current activity is a dynamic activity.

[0085] The preset range can be a preset distance range centered on the relevant subject, a preset area corresponding to the current activity, or a preset distance range set starting from the furniture where the subject is located. For example, the preset range can be within 3 meters. Specifically, for example, if a person is lying on a sofa, the preset range is within 3 meters around the person, or within 3 meters around the sofa.

[0086] In some embodiments, the preset range can be determined based on the activity content, the movement speed of the relevant subject, the movement range of the relevant subject, or the historical activity range. The preset range corresponding to different activities can be different and does not have to be fixed to the same value. For example, if the robot detects that the user is in a resting state (such as lying on a sofa) and the object to be stored (such as slippers) is within the user's preset range (such as 3 meters), the robot can determine that the object to be stored is in a temporary stationary state, indicating that the object to be stored is being used by the user.

[0087] The temporary dwelling status can be updated according to the activity status of the relevant subject or the location of the object to be collected. When the association between the object to be collected and the current activity is no longer valid, or when the association is not detected again within a preset time period, the robot can remove the corresponding temporary dwelling status mark on the semantic topology map.

[0088] In this embodiment, when the current activity status information is not finished and the object to be stored is associated with the current activity, it can be determined that the object to be stored is associated with the current activity that has not yet ended, and the object to be stored is determined to be in a state of not storing for the time being. This can reduce unnecessary storage operations and thus reduce interference with the user's normal life.

[0089] In some other embodiments of this application, where the status information also includes historical occurrence time information of the current activity, the robot determines whether the object to be collected is currently in a state of not being collected based on the status information of the object to be collected. This includes: determining that the object to be collected is in a state of not being collected when the current activity status information is not finished, the current time matches the historical occurrence time information, and the object to be collected is associated with the current activity.

[0090] In some embodiments, the robot can determine one or more historical time windows based on the recorded activity times within a preset historical time period. When the current time falls within a historical time window, or the time difference between the current time and a historical time window is less than a preset time threshold, it can be determined that the current time matches the historical occurrence time information. Similarly, the historical time information of the object to be stored performing a preset function can also be matched using historical time windows or time difference thresholds.

[0091] In this embodiment, when the current activity status information is not yet finished, the current time matches the historical occurrence time information, and the object to be collected is associated with the current activity, the reliability of the judgment of the association between the object to be collected and the current activity can be improved, and it can be determined that the object to be collected is in a state of not collecting for the time being, which can reduce unnecessary collection operations and thus reduce the interference with the user's normal life.

[0092] In some other embodiments of this application, when the status information includes the functional status, the robot determines whether the object to be stored is currently in a state of not being stored based on the status information of the object to be stored, including: when the functional status indicates that the object to be stored is performing a preset function, the robot determines that the object to be stored is in a state of not being stored.

[0093] In this embodiment, when the function status indicates that the object to be stored is performing a preset function, it can be determined that if the object to be stored is stored, the preset function being performed will be interrupted, and the object to be stored will be determined to be in a state of not storing, which can reduce unnecessary storage operations.

[0094] In some other embodiments of this application, where the status information also includes historical time information using preset functions, the robot determines whether the object to be stored is currently in a state of not being stored based on the status information of the object to be stored, including: when the function status indicates that the object to be stored is performing a preset function and the current time matches the historical time information, the robot determines that the object to be stored is in a state of not being stored.

[0095] In this embodiment, when the function status indicates that the object to be stored is performing a preset function, and the current time matches the historical time information, the reliability of the judgment that the object to be stored is performing a preset function can be improved, and it can be determined that the object to be stored is in a state of not storing, thereby further reducing unnecessary storage operations.

[0096] In other embodiments of this application, when the status information includes the user's historical placement and / or retrieval behavior of the object to be stored, the robot determines whether the object to be stored is currently in a state of not being stored based on the status information of the object to be stored, including: determining the commonly used area of ​​the object corresponding to the object to be stored based on the historical placement and / or historical retrieval behavior, and determining that the object to be stored is in a state of not being stored when the object to be stored is located in the commonly used area of ​​the object.

[0097] For example, the robot determines the commonly used area of ​​the object corresponding to the object to be stored based on historical placement behavior and / or historical retrieval behavior, including: determining the area that meets the preset frequency condition as the commonly used area of ​​the object based on the placement frequency and / or retrieval frequency of the object to be stored in a preset historical time period.

[0098] For example, if the object to be stored is a shoe, based on the user's historical placement and / or retrieval behavior of the shoe, the robot can learn the user's habit of placing frequently worn shoes at the bottom of the entryway and determine the bottom area of ​​the entryway as the frequently used area of ​​the object, i.e., the active shoe area.

[0099] The preset frequency conditions can be customized, and this application embodiment does not impose any restrictions on them. For example, the preset frequency conditions can be that the placement frequency and / or retrieval frequency are greater than the corresponding preset frequency threshold.

[0100] In some embodiments, the robot can record the historical placement location, historical retrieval location, corresponding occurrence time, and related entities for the object to be stored, and aggregate adjacent historical locations to form candidate frequently used areas. The robot can determine the frequently used area of ​​an object based on the placement frequency, retrieval frequency, most recent occurrence time, and / or occurrence frequency within a specific time window within the candidate frequently used area. For multiple candidate frequently used areas, the robot can associate them with applicable time periods, related entities, or object uses. For example, books in a home environment are used as an example of items to be stored.

[0101] During long-term operation, the robot can record the historical placement location, historical retrieval location, corresponding time of occurrence, and related parties for books. For example, in a home environment, the robot detects that: (1) User A (an adult) often reads books in the living room sofa area between 20:00 and 22:00 in the evening and places the books near the table next to the sofa after reading; (2) User B (another adult) often reads the same book in the study desk area between 19:00 and 21:00 on weekdays and places the book near the study desk after reading; (3) User C (a child) often reads children's books in the children's room area between 14:00 and 16:00 on weekend afternoons and places the books near the child's table after reading.

[0102] The robot can record the object information, location information, time information, and subject information corresponding to the above behaviors, for example: "Book A - User A - Living room sofa side table - 20:00 to 22:00 - Leisure reading"; "Book A - User B - Study room desk - 19:00 to 21:00 - Study reading"; "Book A - User C - Children's room desktop - Weekend afternoon - Children's reading".

[0103] For example, the robot can perform spatial aggregation of historical locations. By detecting multiple pick-up and place-up locations and when the distance between multiple pick-up and place-up locations is less than a preset distance threshold, the historical locations can be aggregated into commonly used candidate regions.

[0104] For example, the robot can spatially aggregate historical locations. For instance, if the robot detects that the distance between multiple historical placement locations near the side table of the living room sofa is less than a preset distance threshold, it will aggregate these historical locations to form a first candidate frequently used area; if it detects that the distance between multiple historical placement locations near the desk in the study is less than a preset distance threshold, it will form a second candidate frequently used area; if it detects that the distance between multiple historical placement locations near the table in the children's room is less than a preset distance threshold, it will form a third candidate frequently used area.

[0105] The robot further determines the commonly used areas of the book based on historical behavior data corresponding to each candidate frequently used area. For example: For the first candidate frequently used area, the robot detected a high frequency of placement and retrieval by user A, primarily during evening leisure time. Therefore, this area was determined to be user A's frequently used area for leisure reading. For the second candidate frequently used area, the robot detected a high frequency of placement and retrieval by user B, primarily during weekday evenings. Therefore, this area was determined to be user B's frequently used area for study reading. For the third candidate frequently used area, the robot detected a high frequency of placement and retrieval by user C, primarily during weekend afternoons. Therefore, this area was determined to be user C's frequently used area for children's reading. Thus, for the same item to be stored, the robot can identify multiple candidate frequently used areas and associate them with applicable time periods, related entities, and the purpose of the item. For example: When the robot detects that user A is in the living room sofa area and the book is near user A, it can assume that the book is in the area associated with user A's reading activity. When the robot detects that user B is using the book in the study area, it can assume that the book is in user B's corresponding study and reading area. When the robot detects that user C is using the book in the children's room area, it can assume that the book is in the children's reading area. Therefore, the robot does not determine the storage location solely based on "where books are usually placed," but rather combines the object, space, time, and the relationships between related entities to determine the commonly used areas for the object in different scenarios, thus providing a basis for subsequent storage decisions.

[0106] In some embodiments, when there is a conflict between real-time perception results, explicit user configuration, user correction results, historical habits, and default storage rules, the robot can comprehensively judge the information from different sources according to preset priorities and determine the storage decision based on the information with higher priority. Specifically, in some embodiments, if the robot detects that a user is reading a book during the execution of a task, such as detecting that the user is in the sofa area, holding a book, or that the book is related to the current reading activity, the robot can determine that the book is currently in use based on the real-time perception results and temporarily refrain from performing the storage operation.

[0107] Specifically, in some embodiments, the robot can make decisions based on the principle that the user's current intention takes precedence over historical habits, and historical habits take precedence over default storage rules. Specifically, when a user specifies how to handle the items to be stored via voice commands, terminal devices, or other interactive methods, the robot can prioritize adjusting its storage decisions based on the user's current input, rather than performing storage operations based on historical habits or default storage rules.

[0108] For example, based on long-term learning, the robot determines that user A typically places a certain book on the side table next to the sofa, identifying this area as the book's frequently used reading spot. During a cleaning task, the robot detects the book on the side table, and historical habits indicate this is a frequently used area for user A; therefore, the robot can default to not moving the book to the bookshelf. However, if the user configures "all books must be put on the bookshelf" via their terminal device, the robot can use this user configuration as a new storage preference and adjust the original storage strategy generated based on historical habits. For example, the robot can reduce the weight of the frequently used area corresponding to the side table next to the sofa, or update the storage rules corresponding to the book, so that the user-configured storage strategy is prioritized in subsequent tasks.

[0109] In some embodiments, if a user issues a correction instruction to the robot’s current storage decision, such as “Don’t put this book away for now” or “Put this book here later”, the robot can update the book’s status information, area association information, or storage rules according to the correction instruction, and set the priority of the user’s correction result to be higher than historical habits and default storage rules.

[0110] In some embodiments, the robot can set corresponding decision priorities based on the reliability of different information sources. For example: For example, the correction instruction currently entered by the user reflects the user's current immediate intention and has the highest priority; the pre-configured area rules or object processing rules reflect the user's long-term clear preferences and have the next highest priority. For example, the robot's current real-time perception of the object state, subject activity state, and environmental state is used to reflect the current actual scene, and its priority is higher than historical statistical information; user habits formed based on historical placement behavior, retrieval behavior, and activity correlation are used to provide decision-making basis when the user does not actively specify; default storage rules are used to provide basic decision-making when user personalized information is lacking.

[0111] For example, when high-priority information conflicts with low-priority information, the robot can adjust the storage status, frequently used area attributes, storage target location, or task execution order of the items to be stored based on the high-priority information. For instance, if the default storage rule states that "books on the desktop should be put on the bookshelf," but the robot detects that the user is currently reading a book, it can determine not to store it for the time being based on real-time perception results. When the user further explicitly instructs "do not move this book," the robot can further update the book's storage attributes based on the user's correction, so that subsequent tasks will not repeat the same storage operation.

[0112] In this embodiment, based on the placement frequency and / or retrieval frequency of the object to be stored within a preset historical time period, the robot can determine the area that meets the preset frequency condition as the object's frequently used area, thus achieving automatic division between frequently used and infrequently used areas, facilitating the subsequent execution of differentiated storage strategies. When the object to be stored is located in the object's frequently used area, the robot determines that the object is temporarily not to be stored, thereby realizing a shift from forced tidying to adapting to user habits, improving the accuracy and intelligence of the robot's storage decisions.

[0113] In other embodiments of this application, when the status information includes user-configured area configuration information, the robot determines whether the object to be stored is currently in a state of temporary non-storage based on the status information of the object to be stored, including: when the object to be stored is located in the storage exemption area indicated by the area configuration information, it is determined that the object to be stored is in a state of temporary non-storage.

[0114] In this embodiment, when the object to be stored is located in the storage exemption area indicated by the area configuration information, the robot determines that the object to be stored is in a state of temporary non-storage, which can meet the user's personalized needs for exemption from storage in a specific area.

[0115] In some embodiments of this application, when any applicable temporary non-storage condition is met, the robot can determine that the object to be stored is in a temporary non-storage state; when none of the applicable temporary non-storage conditions are met, the robot can determine that the object to be stored is no longer in a temporary non-storage state. The fact that the object to be stored is no longer in a temporary non-storage state does not mean that the robot will immediately perform a storage operation on it; the robot can determine whether and when to perform a storage operation on the object to be stored based on task planning. Before actually executing the storage sub-task, the robot can also determine whether storage execution conditions are met based on at least one of the attributes of the object to be stored, the robot's operational capabilities, the corresponding storage location, and the safety risks in the current environment. If the storage execution conditions are met, the storage operation is performed; if not, the storage operation is performed only if the storage execution conditions are met.

[0116] S14. During the robot's task execution, if it is determined that the object to be stored is in a state of not being stored temporarily, the robot is controlled not to perform the storage operation on the object to be stored.

[0117] In some embodiments of this application, when it is determined that the object to be stored is in a state of not being stored temporarily, the robot is controlled to perform cleaning operations on the area surrounding the object to be stored, but not to perform storage operations on the object. For example, taking the active shoe area in step S13 as an example, the robot can only perform cleaning operations on this area, without performing storage operations on the shoes in this area.

[0118] In some embodiments of this application, the robot's task can be a long-term task, including at least one subtask. The subtask division method can be flexibly set according to actual needs, for example, it can be divided according to task type, execution order, or work area. For example, the long-term task may include multiple subtasks such as storage subtask, cleaning subtask, tidying subtask, and moving subtask. As another example, the long-term task may include coffee table cleaning subtask, sofa tidying subtask, and floor cleaning subtask. Each of the above subtasks is implemented by one or more actions.

[0119] In some embodiments of this application, upon obtaining updated status information, the robot can re-determine whether the object to be stored is in a temporarily unstored state based on the updated status information; when it is determined that the object to be stored is no longer in a temporarily unstored state, the robot can keep the corresponding storage sub-task in a pending state or restore it to a pending state, and determine the execution timing of the storage sub-task according to the updated task plan. The method for obtaining the updated status information can be found in the description of the method for obtaining the status information of the object to be stored.

[0120] In some embodiments of this application, the storage operation refers to the operation of moving an object to be stored from its current location to a storage location corresponding to the object, or the operation of adjusting the state of the object itself. The storage operation can be achieved through one or more actions such as grasping, placing, and tidying. For example, the storage operation could be: putting shoes into a shoe cabinet. In this process, the robot can first open the shoe cabinet door, grasp the shoes, put them into the shoe cabinet (and even place the shoes on the shelf the user usually places them on according to user habits), and then close the cabinet door. The storage operation could also be: folding clothes placed on the bed and putting them into a wardrobe. Specifically, the robot adjusts the state of the randomly placed clothes by grasping or other means until the clothes are folded, then opens the wardrobe, grasps the clothes, puts them into the wardrobe, and then closes the wardrobe.

[0121] In some embodiments of this application, the robot can construct a semantic topology map during the execution of a task. The semantic topology map contains the state information of the objects to be collected.

[0122] In some embodiments, the robot can utilize a VLA model to extract the features and spatial relationships of various objects in the environment, and combine this with historical interaction records with the user to construct a semantic topological map containing the state information of the objects to be collected. During the construction process, the robot can use the VLA model to generate and save latent space states for episode memory. The latent space states are feature representations obtained by the VLA model encoding the environment and task states during inference, effectively compressing and retaining key information in the scene.

[0123] The methods for constructing semantic topology maps can be flexible and diverse. For example, a robot can construct a semantic topology map using a perception system (such as LiDAR, visual sensors, etc.) combined with a pre-defined mapping technique. For instance, the pre-defined mapping technique could be Simultaneous Localization and Mapping (SLAM). In other embodiments, the robot can also construct semantic topology maps in other ways. This application does not limit the method of constructing semantic topology maps.

[0124] For example, the status information of objects to be tidied up can be represented as "slippers - located - next to the sofa", "phone - in - charging state", "toy - in - user - use", etc. This information is embedded into the map to form the status information in the semantic topology map. Further semantic topology maps can include information such as whether cleaning is needed, for example, "under the sofa - dusty" or "on the table - wiped clean". In addition to embedding the status information of objects to be tidied up into the semantic topology map, user habit information can also be embedded to present multi-dimensional semantic information, such as "slippers - located - next to the sofa - 8 PM" or "phone - located - on the bedside table - charging - 11 PM", thus enabling the semantic topology map to more comprehensively reflect the spatial location, status, and user usage patterns of objects in the environment.

[0125] If the robot stores a pre-built semantic topology map, the construction of the semantic topology map described here can be an update and / or improvement of the pre-built semantic topology map. The robot can perform tasks based on the pre-built semantic topology map.

[0126] In this embodiment, by constructing a semantic topology map, the environment is represented and structurally modeled, so that tasks can be rationally planned and executed based on the rich information in the semantic topology map.

[0127] The method described above for scene memorization using semantic topology maps and latent space states is merely an example. In practical applications, other types of memory media can also be used. For instance, any method capable of expressing multi-dimensional semantic information such as "object-location-state-habit" can serve as an alternative valve.

[0128] For example, in some embodiments of this application, the robot can perform scene memorization based on a three-dimensional voxel grid, which at least contains state information of the objects to be collected. By discretizing the space, the voxel grid can reduce the amount of data while retaining approximate spatial distribution information, thus requiring less memory.

[0129] For example, in some other embodiments of this application, the robot can perform scene memorization based on a large language model to generate structured text description information, which at least describes the state information of the object to be collected. In this way, by leveraging the semantic understanding and reasoning capabilities of the large language model, flexible and accurate expression and dynamic updating of scene information can be achieved.

[0130] In some embodiments of this application, the robot determines the order of operations for different areas based on heat distribution in a semantic topology map and / or user-configured area preference information. The operations include cleaning the areas and / or storing the objects to be stored.

[0131] The semantic topology map can be presented as a heat map, where heat distribution is used to characterize the degree of operational demand in different areas. For example, areas with higher heat values ​​correspond to a higher degree of operational demand than areas with lower heat values.

[0132] Area preference information can include the user's preferred task areas, which can be one or more. For example, area preference information could be "Prioritize cleaning the living room and kitchen".

[0133] For different areas, the priority of tasks in areas with higher task demand is higher than that in areas with lower task demand; and / or, for different areas, the priority of tasks in user-preferred priority areas is higher than that in other areas; and / or, for different areas, the priority of tasks in areas with higher historical task demand frequency is higher than that in areas with lower historical task demand frequency; and / or, for the same area, when the items to be stored in the area are not in a temporarily unstored state, the priority of storage operations is higher than that of cleaning operations.

[0134] In this embodiment, based on the heat distribution in the semantic topology map, the robot can determine the degree of task demand in different areas in the spatiotemporal dimension. Based on area preference information, the robot can determine the user's priority preference for tasks in a specified area and determine the order of tasks in different areas. Therefore, it can not only meet the user's personalized task needs for a region but also achieve optimization of task planning in the spatiotemporal dimension.

[0135] In some embodiments of this application, during the execution of a task, the robot uses a VLA model, combined with a perception system, to perceive the relative distance between its current position and the target object, and decouples the chassis movement and the robotic arm trajectory movement according to the relative distance, so as to approach and collect the target object, which includes objects that are not in a state of not being collected yet.

[0136] By decoupling the control of chassis movement and robotic arm trajectory movement (such as gripping / cleaning trajectory) based on relative distance, the robot can control the robotic arm in advance as it approaches the target object, so as to smoothly perform the storage operation on the target object.

[0137] For example, the robot uses an RGB-D camera to acquire environmental information and uses a VLA model to output joint control commands for chassis movement, lifting and lowering, and robotic arm joints based on the acquired environmental information, so as to achieve smooth and continuous storage operations.

[0138] In this embodiment, the robot utilizes the distance generalization capability of the VLA model to decouple the chassis movement and the robotic arm trajectory movement based on relative distance. During movement, the robotic arm is pre-adjusted synchronously, allowing it to directly perform the storage operation upon approaching the target object, eliminating the need for precise docking before storage. This not only improves storage efficiency and intelligence but also better adapts to the randomness of furniture and other target object placement, overcoming the limitation of related technologies where robots must first precisely locate themselves before performing operations.

[0139] In some embodiments of this application, if the robot receives a correction instruction sent by the user during the execution of a task, it updates the attribute information of the object to be collected corresponding to the correction instruction in the semantic topology map according to the correction instruction.

[0140] For example, if a robot mistakenly identifies a toy as "trash" and attempts to collect it while performing a task based on a semantic topology map, it can update the semantic attribute of the toy from "trash" to "toy" in the semantic topology map when it receives a correction instruction from the user indicating that the toy does not need to be collected, thus reducing the chance of repeating the same mistake.

[0141] In this embodiment, the robot updates the attribute information of the object to be collected corresponding to the correction instruction in the semantic topology map according to the correction instruction, which can realize the dynamic correction and continuous improvement of the semantic topology map and reduce the recurrence of similar problems in subsequent operations.

[0142] In some embodiments of this application, in response to a task interruption and restart, the robot uses the latent space state and semantic topology map encoded by the VLA model to identify the current task progress, backtracks to the previous safe breakpoint according to the task progress, and continues to execute the task with the safe breakpoint as the recovery starting point. The safe breakpoint is the state corresponding to the atomic action unit that was completed before the task interruption.

[0143] Among them, the atomic action unit can be the smallest indivisible execution operation (such as a single grab, a single clean, or a single move).

[0144] Compared to the "starting from scratch" strategy in related technologies, this embodiment can achieve precise breakpoint continuation scanning after the robot is picked up or interrupted by saving the latent space state and semantic topology map snapshot during the VLA model inference process. This not only improves the robot's task efficiency and intelligence level, but also enhances the robot's adaptability to complex and dynamic environments.

[0145] The above method of resuming scanning from a breakpoint using latent space states and semantic topology maps is only an example. In practical applications, robots can also achieve this through other methods.

[0146] For example, in some other embodiments of this application, in response to a task interruption and restart, the robot compares the current image frame with historical image frames to identify the current task progress, backtracks to the previous safe breakpoint according to the task progress, and continues to execute the task with the safe breakpoint as the recovery starting point. The safe breakpoint is the state corresponding to the atomic action unit that was completed before the task interruption.

[0147] In this embodiment, the image comparison method can be used to simply, quickly and accurately identify the task progress and resume scanning from breakpoints.

[0148] The robot control method provided in this application can be applied to various scenarios. The specific applications of this application will be illustrated below using scenarios such as electronic product charging, reading and office material handling, pet supplies and interactive areas, temporary clothing storage areas, kitchen food preparation and temporary storage, and children's toys and play areas.

[0149] Electronic Product Charging Scenarios: Users often carelessly place their mobile phones, tablets, or laptops on sofa armrests, bedside tables, or desks to charge. The robot uses a VLA (Visual Analog Array) model to identify when an electronic product is charging. Considering whether its location aligns with the user's long-standing charging habits (e.g., charging on the left side of the bedside table every night), the robot determines that the product is being used and therefore should not be stored away. For such products, the robot can simply clean the surrounding dust without touching or moving the device to avoid interrupting charging or causing damage. If an electronic product is not connected to a power source and remains stationary in the same location for an extended period (e.g., forgotten in a corner), the robot can determine that it is not being used and should be stored away, rather than temporarily not stored away, and will then perform the storage operation.

[0150] Reading and office material scenario: Books, notebooks, and pens are scattered on a desk, coffee table, or carpet. The robot can learn, through a VLA model, the user's reading habits over the past week during specific time periods (e.g., 8-10 PM) at the coffee table, the relatively fixed angle and position of the open books, or the presence of an open laptop and scattered documents on the table. If the robot detects that the user is sitting at the desk, carpet, or coffee table, and the current time falls within that specific time period, it can determine that the user's reading and office activities are not yet finished, and the books, notebooks, and pens awaiting storage are being used. The robot may not perform storage operations on these items, but can, based on user habits, push a nearby water glass closer and / or close the books and place them back on the shelf or in the designated area. If the current time is not within a specific time period (e.g., 3 AM) and the items are scattered, the robot can perform storage operations on the books, notebooks, and pens awaiting storage.

[0151] Pet supplies and interactive area scenario: Pet toys, food bowls, water bowls, or litter scoops are scattered on the living room floor. The robot, through VLA models, learns that pets typically spend time in the living room around 4 PM and that owners usually play with them during this time. Combined with pet activity monitoring, if the robot detects that a pet has just left or is resting nearby, it can temporarily leave pet toys uncollected and avoid collecting and cleaning items around 4 PM, the peak pet activity period. Food bowls and water bowls will remain in their original positions unless they are detected to have been knocked over or spilled, thus reducing pet anxiety.

[0152] Scenario for temporary clothing storage: Coats, backpacks, and scarves are casually tossed on the back of dining chairs, the bench at the foot of the bed, or the entryway cabinet. The robot can learn from the user's VLA model that they have a habit of storing "partially clean clothes" (clothes worn once but not yet washed), such as hanging a coat on a specific chair back instead of immediately putting it in the closet. Thus, the robot can distinguish between the "dirty laundry basket area" and the "partially clean clothes hanging area." If clothing is in the "partially clean clothes area," the robot can determine that the clothing is not to be stored immediately and will not perform any storage operations on clothing in the "partially clean clothes area" (e.g., not forcibly stuffing a coat from the chair into the closet). Instead, it will tidy up the clothing to make it neater and clean the floor below. Storage operations will only be performed if the clothing has fallen to the ground or is in an unfavorable area.

[0153] Kitchen food preparation and temporary storage scenario: Condiment bottles, cutting boards, and unwashed vegetables are placed on the kitchen countertop or island. Using a VLA model, the robot recognizes that it is dinner preparation time (e.g., 6 PM) and that the items on the countertop exhibit characteristics of food preparation (e.g., knife placement, food cutting). It determines that the condiment bottles awaiting storage are not to be stored immediately and pauses deep cleaning and tidying of the area, performing only simple stain wiping. Only after detecting that the user has left the kitchen for a period of time (e.g., returning to their room after dinner) does the robot fully initiate the storage operation of putting kitchen utensils back in their proper places and the countertop cleaning operation.

[0154] Children's toy and play area scenario: Building blocks and plush toys cover a playmat or a corner of the living room. The robot can detect and record the child's play patterns using a VLA model, for example, if the child plays in the living room every morning and goes to kindergarten in the afternoon. Based on the schedule, the robot can determine that the morning is playtime and that building blocks and plush toys are not to be put away, thus avoiding any storage operations. After the child leaves in the afternoon, the robot can determine that the building blocks and plush toys are no longer in the "not put away" state, and will categorize the toys and put them into toy boxes, and clean the playmat.

[0155] The above examples demonstrate that the control method of this application, by combining the perception capabilities of the VLA model with the user habit learning mechanism, can adaptively adjust the storage strategy in different life scenarios, thereby improving the accuracy and intelligence of the robot's storage decisions.

[0156] In the robot control scheme provided in this application embodiment, the robot determines whether the object to be stored is in a state of temporary non-storage based on the object's status information. When the object is in a state of temporary non-storage, the robot is controlled not to perform a storage operation on that object. This reduces the problem of inaccurate robot storage decisions and minimizes unnecessary movements.

[0157] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] Please see Figure 4 The diagram shown is a structural schematic of a control device provided in one embodiment of this application, which can implement the details of the control method in the above embodiments and achieve the same effect. Figure 4As shown, the control device 40 can be applied to a robot with data processing capabilities. The robot is used at least to store objects to be stored. The control device 40 includes: an identification unit 401 for identifying objects to be stored; an acquisition unit 402 for acquiring the status information of the objects to be stored; a determination unit 403 for determining, based on the status information of the objects to be stored, whether the objects to be stored are currently in a state of not being stored; and a storage unit 404 for controlling the robot not to perform storage operations on the objects to be stored if it is determined that the objects to be stored are in a state of not being stored during the robot's task execution.

[0159] Specific limitations regarding the control device 40 can be found in the limitations of the control method described above, and will not be repeated here. Each unit in the control device 40 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the robot's processor in hardware form or independently of it, or stored in the robot's memory in software form, so that the processor can call and execute the operations corresponding to each unit.

[0160] Please see Figure 5 The diagram shown is a structural schematic of a robot provided in one embodiment of this application. Figure 5 The networks in which the robot 500 shown is located include, but are not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0161] like Figure 5 As shown, the robot 500 includes a robot body 11, a robotic arm 12, a drive motor 13, a sensing system 14, a memory 15, a processor 16, a battery 17, a communication module 18, an interaction module 19, a brush sweeping component 20, and a mopping component 21.

[0162] The robot body 11 can form the basic framework of the robot 500, and the robot body 11 can include a mobile chassis. The robotic arm 12, drive motor 13, sensing system 14, memory 15, processor 16, battery 17, communication module 18, interaction module 19, brush and sweeping component 20 and mopping component 21 can be set on the robot body 11.

[0163] The robotic arm 12 can be a single robotic arm, a dual robotic arm, or a multi-robotic arm; this embodiment of the application does not impose any limitations on this. The end effector of the robotic arm 12 is provided with an end effector such as a gripper.

[0164] The drive motor 13 is used to drive the mobile chassis to move, thereby enabling the robot 500 to move autonomously, so that the robotic arm 12 can perform the storage operation of the object to be stored, and the mopping component 21 and / or brushing component 20 can perform the cleaning operation of the surface to be cleaned.

[0165] The perception system 14 includes various types of sensors, such as visual sensors (e.g., cameras and video cameras), ultrasonic sensors, lidar, collision sensors, distance sensors, counters, and gyroscopes.

[0166] Memory 15 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 16 and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0167] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 16. Non-volatile memory can include disk storage devices and flash memory. For example, flash memory can be Nand Flash.

[0168] Memory 15 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 16. The one or more computer programs include multiple instructions, which, when executed by processor 16, enable a robot control method to be executed on robot 500.

[0169] In other embodiments, such as Figure 5 The robot 500 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the robot 500.

[0170] Processor 16 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0171] The processor 16 provides computing and control capabilities. For example, the processor 16 is used to execute computer programs stored in the memory 15 to implement the robot control method described above.

[0172] Battery 17 can be used to provide power to robot 500. Robot 500 is also equipped with a charging component, which is used to obtain power from external devices (such as base stations) to charge battery 17.

[0173] The communication module 18 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR). For example, Robot 500 can communicate with terminal devices via a 5G module.

[0174] Users can interact with the robot 500 through the interaction module 19. The interaction module 19 includes components such as a switch button, speaker, microphone, and touch switch / screen. Users can control the robot 500 to start or stop working by pressing the switch button or touch switch / screen, and can also display the robot 500's working status information through the touch screen.

[0175] Robot 500 can play prompts to users through a speaker, obtain user control commands through a microphone, or locate the user's position by obtaining the user's voice.

[0176] The brush / sweeping component 20 can be a roller brush, side brush, etc., used to clean the surface to be cleaned. The brush / sweeping component 20 can be set at the bottom of the robot body 11.

[0177] The mopping component 21, for example, is a mop, which can be used to mop the surface to be cleaned. The mopping component 21 can be provided at the bottom of the robot body 11.

[0178] In some implementations, robot 500 is a sweeping and mopping robot, where the sweeping component 20 and the mopping component 21 can work together, for example, the sweeping component 20 and the mopping component 21 can work simultaneously, or the sweeping component 20 and the mopping component 21 can work alternately. Of course, the sweeping component 20 and the mopping component 21 can also work separately, that is, the sweeping component 20 can perform sweeping work alone, or the mopping component 21 can perform mopping work alone.

[0179] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on robot 500. In other embodiments of this application, robot 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0180] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can refer to the control methods in the above embodiments of this application. The computer-readable storage medium can be the robot's internal memory, such as the robot's hard drive or RAM, as described in the above embodiments. Alternatively, it can be an external storage device for the robot, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0181] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the robot, etc.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for controlling a robot, characterized in that, include: Identify items to be stored; Obtain the status information of the object to be stored, the status information including the current activity status information of the relevant subject and the association information between the object to be stored and the current activity; Based on the status information, determining whether the object to be stored is currently in a state of not being stored includes: when the current activity status information is not ended and the object to be stored is associated with the current activity, determining that the object to be stored is in the state of not being stored. If, during the process of the robot performing a task, it is determined that the object to be stored is in the state of not storing, the robot is controlled not to perform a storage operation on the object to be stored.

2. The control method according to claim 1, characterized in that, The object to be collected is associated with the current activity, including at least one of the following conditions: The object to be collected is located within a preset range related to the current activity of the relevant subject; The object to be collected is being used by the relevant entity in the current activity.

3. The control method according to claim 2, characterized in that, The current activity status information also includes at least one of activity type and activity execution status; The activity type is either a static activity or a dynamic activity; the preset range related to the current activity for the static activity is smaller than the preset range related to the current activity for the dynamic activity; The activity execution status includes one of the following: not started, not ended, or ended.

4. The control method according to claim 1, characterized in that, The status information also includes historical occurrence time information of the current activity; The step of determining whether the object to be stored is currently in a state of not being stored based on the status information includes: When the current activity status information is not finished, the current time matches the historical occurrence time information, and the object to be collected is associated with the current activity, the object to be collected is determined to be in the "not collected for the time being" state.

5. The control method according to claim 1, characterized in that, The status information includes functional status; determining whether the object to be stored is currently in a state of not being stored based on the status information includes: When the function status indicates that the object to be stored is performing a preset function, the object to be stored is determined to be in the "not stored" state.

6. The control method according to claim 5, characterized in that, The status information also includes historical time information of the execution of the preset function; The step of determining whether the object to be stored is currently in a state of not being stored based on the status information includes: When the function status indicates that the object to be stored is performing the preset function, and the current time matches the historical time information, the object to be stored is determined to be in the "not stored" state.

7. The control method according to claim 1, characterized in that, The status information includes the user's historical placement and / or retrieval behavior of the object to be stored; The step of determining whether the object to be stored is currently in a state of not being stored based on the status information includes: Based on the historical placement behavior and / or the historical retrieval behavior, the commonly used area of ​​the object corresponding to the object to be stored is determined. When the object to be stored is located in the commonly used area of ​​the object, the object to be stored is determined to be in the state of not storing.

8. The control method according to claim 7, characterized in that, The step of determining the commonly used area of ​​the object corresponding to the object to be stored based on the historical placement behavior and / or the historical retrieval behavior includes: Based on the placement frequency and / or retrieval frequency of the object to be stored within a preset historical time period, the area that meets the preset frequency condition is determined as the commonly used area of ​​the object.

9. The control method according to claim 1, characterized in that, The status information includes user-configured region configuration information; the region configuration information is used to indicate the storage exemption region corresponding to the object to be stored. The step of determining whether the object to be stored is currently in a state of not being stored based on the status information includes: When the object to be stored is located in the storage exemption area, it is determined that the object to be stored is in the state of not storing.

10. The control method according to any one of claims 1-9, characterized in that, When it is determined that the object to be stored is in the state of not storing, the robot is controlled to perform cleaning operations on the area around the object to be stored, but does not perform storage operations on the object to be stored.

11. The control method according to any one of claims 1-9, characterized in that, Upon obtaining the updated status information, the system re-determines whether the object to be stored is in the "not stored" state based on the updated status information. Once it is determined that the object to be stored is no longer in the "not to be stored" state, the storage operation is performed on the object to be stored according to the task plan.

12. The control method according to any one of claims 1-9, characterized in that, During the execution of the task, the robot constructs a semantic topology map; the semantic topology map contains the state information of the objects to be collected.

13. The control method according to claim 12, characterized in that, The robot determines the order of operations for different areas based on the heat distribution in the semantic topology map and / or the user-configured area preference information. The heat distribution in the semantic topology map is used to characterize the degree of operation demand in the different areas. The area preference information includes the user's preferred priority operation areas. The operation includes cleaning operations in the areas and / or storage operations for the objects to be stored.

14. The control method according to claim 13, characterized in that, For the different regions, the task priority corresponding to the region with higher task demand is higher than the task priority corresponding to the region with lower task demand; and / or, For the different regions, the priority of the task in the user's preferred priority region is higher than the priority of the task in other regions; And / or, For the different regions, the region with a higher historical frequency of job requests has a higher job priority than the region with a lower historical frequency of job requests. And / or, For the same area, when the items to be stored in the area are not in the "not stored" state, the storage operation has a higher priority than the cleaning operation.

15. The control method according to claim 12, characterized in that, If the robot receives a correction instruction from the user during the execution of a task, it updates the attribute information of the object to be collected in the semantic topology map according to the correction instruction.

16. The control method according to claim 12, characterized in that, During the execution of the task, the robot uses a full-body vision-language-motion (VLA) model to perceive the relative distance between its current position and the target object, and decouples the chassis movement and the robotic arm trajectory movement according to the relative distance, so as to approach and collect the target object, which includes objects to be collected that are not in the temporarily not collected state.

17. The control method according to claim 16, characterized in that, In response to a task interruption and subsequent restart, the robot utilizes the latent space state encoded by the VLA model and the semantic topology map to identify the current task progress. Based on the task progress, it backtracks to the previous safe breakpoint and uses the safe breakpoint as the recovery starting point to continue executing the task. The safe breakpoint is the state corresponding to the atomic action unit that was completed before the task interruption.

18. A cleaning robot, characterized in that, The cleaning robot includes a memory and a processor; the memory is used to store program instructions; the processor is used to read the program instructions stored in the memory to implement the control method according to any one of claims 1 to 17.

19. A cleaning system, characterized in that, The cleaning system includes: A cleaning robot, said cleaning robot being used to implement the control method according to any one of claims 1 to 17; and The base station is equipped with a docking position for the cleaning robot to dock, and the cleaning robot can selectively dock at the base station. The base station is used at least for the maintenance of the cleaning robot.

20. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method according to any one of claims 1 to 17.

Images