An automated robotic operation data collection method and system

Abstract

The application discloses an automatic robot operation data collection method and system, belongs to the technical field of data collection, and can solve the problem that the prior art cannot cover changing scenes. The method comprises the following steps: S1, determining the reachable working space of a robot according to a target task, and dividing the reachable working space into multiple space units; S2, controlling the robot to perform a task operation on a target object located in a current space unit; collecting operation data of the task operation to obtain a current sample of the current space unit; S3, determining a next space unit of a next task operation according to the current sample and historical samples of all space units, moving the target object to the next space unit, and taking the next space unit as a new current space unit; and S4, repeatedly performing S2 and S3 until the total number of samples of the reachable working space reaches a preset number or the sample distribution reaches a preset distribution state. The application is used for collecting robot operation data.

Description

Technical Field

[0001] This invention relates to a method and system for acquiring data from automated robot operations, belonging to the field of data acquisition technology. Background Technology

[0002] As robots are deployed in scenarios such as home services, power line inspections, and supermarket vending, they need to possess the ability to identify, locate, and manipulate target objects (such as grasping, pressing, twisting, carrying, and placing). The acquisition and optimization of these capabilities require training the robot's decision-making model. The sample data required for training includes multimodal operational data such as image data, point data, joint state data, force / torque sensing data, audio data, and tactile data of the robot when performing the target task.

[0003] Existing methods typically place the target object at a fixed station when collecting sample data of a robot performing a specific task, allowing the robot to repeatedly perform multiple task operations on the target object and collect multimodal operation data for each task operation. Through cyclical collection, a large amount of sample data is obtained.

[0004] Because the target object is placed in a fixed position and the task operation scenario is singular, the collected sample data cannot cover randomly changing operation scenarios (such as operation scenarios under different environments, positions, lighting conditions, and occlusion conditions), resulting in a bias in the sample data. Training the robot's decision model based on this biased sample data will lead to the decision model overfitting to specific operation scenarios, thereby reducing the robustness and generalization ability of the decision model. Summary of the Invention

[0005] This invention provides an automated robot operation data acquisition method and system, which can solve the problem that existing technologies cannot cover randomly changing operation scenarios, resulting in sample data bias and causing the robot's decision model to overfit specific scenarios.

[0006] On one hand, the present invention provides a method for acquiring automated robot operation data, the method comprising: S1. Determine the robot's reachable workspace based on the target task, and divide the reachable workspace into multiple spatial units; S2. Control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; collect the operation data of the robot performing the current task operation to obtain the current sample of the current spatial unit; S3. Based on the current sample and the historical samples of all spatial units, determine the next spatial unit corresponding to the next task operation, move the target object to the next spatial unit, and use the next spatial unit as the new current spatial unit; S4. Repeat S2 and S3 until the total number of samples in the reachable workspace reaches a preset number or the sample distribution in the reachable workspace reaches a preset distribution state.

[0007] Optionally, in S3, the next spatial unit corresponding to the next task operation is determined based on the current sample and historical samples of all spatial units, specifically including: Based on the current sample and the historical samples of all spatial units, determine the number of samples and the task failure rate for each spatial unit. The task failure rate is the percentage of samples in the spatial unit whose task completion status is failed. The next spatial unit corresponding to the next task operation is determined based on the number of samples and the task failure rate.

[0008] Optionally, the next spatial unit corresponding to the next task operation is determined based on the number of samples and the task failure rate, specifically including: The sampling weight of the spatial unit is adjusted according to the number of samples and the task failure rate; The next spatial unit corresponding to the next task operation is determined based on the sampling weights of all spatial units after the adjustment.

[0009] Optionally, adjusting the sampling weight of the spatial unit based on the number of samples and the task failure rate specifically includes: Increase the sampling weight of spatial units with a sample size less than a preset range, decrease the sampling weight of spatial units with a sample size greater than the preset range, and increase the sampling weight of spatial units with a task failure rate greater than a preset ratio.

[0010] Optionally, the next spatial unit corresponding to the next task operation is determined based on the sampling weights of all spatial units after adjustment, specifically including: The spatial unit with the highest adjusted sampling weight is determined as the next spatial unit for the next task operation.

[0011] Optionally, the number of samples for each spatial unit is determined based on the current sample and historical samples of all spatial units, specifically including: Remove historical samples from all historical samples in each spatial unit whose time interval with the current moment is longer than a preset duration; The number of samples for each spatial unit is determined based on the current samples and the remaining historical samples for each spatial unit.

[0012] Optionally, in S2, the robot is controlled to perform the current task operation on the target object located in the current spatial unit according to the target task, specifically including: The robot's trajectory is determined based on the target task and the current spatial unit where the target object is located; The robot is controlled to perform the current task operation on the target object according to the motion trajectory.

[0013] Optionally, the operation data includes at least one of the following: force sensing data of the robot, joint state data, operation image data, point cloud data, and audio data.

[0014] Optionally, in S2, the operation data collected by the robot during the current task operation includes: At multiple moments during the robot's execution of the current task, force sensing data, joint state data, operation image data, point cloud data, and audio data of the robot are collected simultaneously, and a unified timestamp is set for the operation data of different modalities.

[0015] On the other hand, the present invention provides an automated robot operation data acquisition system, the system comprising: The space partitioning module is used to determine the robot's reachable workspace based on the target task and divide the reachable workspace into multiple spatial units; The data acquisition module is used to control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; to collect the operation data of the robot performing the current task operation, and to obtain the current sample of the current spatial unit; The position changing module is used to determine the next spatial unit corresponding to the next task operation based on the current sample and the historical samples of all spatial units, move the target object to the next spatial unit, and use the next spatial unit as the new current spatial unit; The repeated sampling module is used to repeatedly perform data acquisition and location changes until the total number of samples in the reachable workspace reaches a preset number or the sample distribution in the reachable workspace reaches a preset distribution state.

[0016] The beneficial effects that this invention can produce include: This invention divides the robot's reachable workspace into multiple spatial units. By performing task operations and data collection on target objects located in the current spatial unit, a current sample is obtained. Then, based on the current sample and historical samples obtained from previous sampling, the spatial unit in which the target object should be located in the next sampling is determined. The position of the target object is then changed accordingly for each sampling, enabling sampling in changing scenarios. This significantly improves the diversity and coverage of sample distribution, avoiding the problem of overfitting of the robot's decision model to specific scenarios due to sample bias, and improving the robustness and generalization ability of the decision model. Furthermore, this invention triggers data collection via hardware and sets a unified timestamp for operation data of different modalities, achieving high-precision alignment of data from different modalities, facilitating subsequent training of the decision model. In addition, this invention is adaptable to multiple scenarios such as home services, power line inspection, and supermarket sales, exhibiting strong scalability and a wide range of applications. Attached Figure Description

[0017] Figure 1 A flowchart of an automated robot operation data acquisition method provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.

[0019] This invention provides a method for acquiring automated robot operation data, such as... Figure 1 As shown, the method includes: S1. Determine the robot's reachable workspace based on the target task, and divide the reachable workspace into multiple spatial units.

[0020] Specifically, a robot includes the robot body and end effectors (such as robotic arms, grippers, etc.). The robot is also equipped with vision sensors, point cloud sensors, force / torque sensors, audio acquisition devices, etc., to collect operational data of the robot during task operations.

[0021] Operational data includes at least one of the following: robot force sensing data, joint state data, operation image data, point cloud data, and audio data.

[0022] In this embodiment, the vision sensor is at least one of an RGB camera, an RGB-D camera, and a depth camera, used to acquire operational image data; the point cloud sensor is used to acquire point cloud data; the force / torque sensor is installed at the actuator end or joint of the end effector, used to acquire force sensing data and joint state data; and the audio acquisition device is used to acquire audio data.

[0023] Specifically, the target task can be a specific task in scenarios such as home services, power line inspection, and supermarket sales.

[0024] Among them, the target tasks for household services include grabbing items, opening and closing door handles, and tidying up items; the target tasks for power inspection include pressing buttons, turning switches, opening and closing gates (under safety constraints), reading meter data and taking photos; and the target tasks for supermarket sales include picking up and placing goods, aligning the goods with the barcode scanning area and scanning the barcode.

[0025] Different target tasks require different target objects and corresponding task operations. Taking the task of retrieving and placing goods in a supermarket as an example, the target object is the product displayed in the supermarket, and the task operations are identifying the product's location, grabbing the product, moving the product, and placing it in a designated area. Since the product's location within the supermarket may change with the display layout, if the robot's operation data is collected by placing the product in a fixed position and then trained based on that data, the robot will be unable to adapt to task scenarios with changes in the display layout.

[0026] To address the aforementioned issues, this embodiment divides the robot's reachable workspace into multiple spatial units. During the data acquisition process, the spatial unit where the target product is located is changed, and the robot's operation data is collected after each change, so that the collected operation data can cover different spatial units.

[0027] S2. Control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; collect the operation data of the robot performing the current task operation to obtain the current sample of the current spatial unit. Specifically, this includes: 1) Determine the robot's trajectory based on the target task and the current spatial unit where the target object is located; This embodiment first collects image data and / or point cloud data of the scene where the target object is located, then uses a visual detection algorithm to identify and estimate the pose of the target object, outputs the pose information of the target object, and then generates the motion trajectory of the robot end effector based on the pose information.

[0028] Specifically, the motion trajectory includes the motion trajectory corresponding to each stage, such as perception and positioning, approaching the target object, alignment / pre-grabbing, contact / clamping / operation, handling / moving, placement / release, withdrawal, termination and reset.

[0029] 2) Control the robot to perform the current task operation on the target object according to the motion trajectory; During the execution of tasks, methods such as visual servoing, force control, and hybrid control can be used to make closed-loop adjustments to the robot's operation.

[0030] 3) Collect the operation data of the robot performing the current task.

[0031] This embodiment simultaneously collects various modalities of operational data, including force sensor data, joint state data, operation image data, point cloud data, and audio data, at multiple moments during the robot's execution of the current task operation. (a) Robotic arm joint status data (joint angle, angular velocity, joint torque / current, etc.); (b) Force / torque sensor data of the robotic arm (six-dimensional force / torque or equivalent end force); (c) Clamping device joint status data (e.g., opening angle, drive current / position command / encoder feedback); (d) Force sensor data of the clamping device (e.g., gripper contact force, estimated clamping force); (e) Manipulating image data (RGB images, infrared images, etc.); (f) Point cloud data (depth point cloud data, dense / sparse point cloud data, etc.); (g) Audio data (ambient audio data, interactive audio data, etc.).

[0032] The above data together constitute the current sample of the current spatial unit.

[0033] Meanwhile, this embodiment assigns a unified timestamp to each modal data and records task stage labels, target IDs, target poses, control commands, execution status, and exception information.

[0034] To avoid asynchrony between data acquisition and task operation, this embodiment employs hardware triggering to activate devices such as visual sensors, point cloud sensors, force / torque sensors, and audio acquisition devices to collect data, achieving strict synchronization between data acquisition and task operation. Simultaneously, this embodiment sets a unified timestamp for the operational data of different modalities to avoid alignment difficulties caused by the lack of a unified time reference. Furthermore, this embodiment records the sampling time and delay parameters of different modal data and then uses an interpolation algorithm for data alignment, thereby ensuring high-precision alignment of different modal data.

[0035] This embodiment determines the end of the current task operation based on the task completion conditions (such as completion of goods retrieval and placement, arrival at the inspection point and completion of reading / taking photos, etc.), and archives all data of the current task operation in a unified format to generate a metadata index for the current sample. The metadata index includes the task scenario, target task, randomization parameters, time range, task completion status (success / failure), etc.

[0036] S3. Based on the current sample and historical samples of all spatial units, determine the next spatial unit corresponding to the next task operation, move the target object to the next spatial unit, and set the next spatial unit as the new current spatial unit. Specifically, this includes: 1) Based on the current samples and the historical samples of all spatial units, determine the number of samples and the task failure rate of each spatial unit. The task failure rate of any spatial unit is the proportion of the number of samples in the spatial unit whose task completion status is failed. This embodiment calculates the sum of the number of all samples in each spatial unit at the current time and at historical times to obtain the number of samples in each space; and determines the task failure rate of each spatial unit based on the task completion status of each sample in each spatial unit.

[0037] 2) Determine the next spatial unit corresponding to the next task operation based on the sample size of all spatial units and the task failure rate; First, the sampling weights of each spatial unit are adjusted based on the sample size and task failure rate. Specifically, in this embodiment, the sampling weights of spatial units with a sample size less than a preset range are increased, while the sampling weights of spatial units with a sample size greater than the preset range are decreased. This increases the probability of sampling insufficiently sampled spatial units and avoids redundant sampling of fully sampled spatial units. Simultaneously, the sampling weights of spatial units with a task failure rate greater than a preset ratio are increased to increase the proportion of difficult samples. Then, the spatial unit with the highest adjusted sampling weight is determined as the next spatial unit for the next task operation.

[0038] Furthermore, to reduce the impact of earlier collected operational data on the current adjustment process, this embodiment, when counting the number of samples in each space, can remove historical samples from all historical samples of each spatial unit whose time interval with the current moment is longer than a preset duration. Then, the number of samples in each spatial unit is determined based on the current sample and the remaining historical samples in each spatial unit. This allows the influence of historical samples on the adjustment process to decay over time, thereby increasing the influence of more recent moments and achieving dynamic adjustment. The preset duration can be flexibly set according to sampling requirements and actual conditions.

[0039] In addition, when determining the next spatial unit, constraints such as robot accessibility constraints, safe distance constraints, and collision constraints should also be met.

[0040] Through the above adjustments, this embodiment can achieve adaptive sampling and ensure the sampling uniformity of each spatial unit.

[0041] 3) Move the target object to the next spatial unit and set the next spatial unit as the new current spatial unit.

[0042] This embodiment can achieve the movement and rearrangement of the target object through a mobile platform, a conveying mechanism, or a manual assistance device.

[0043] S4. Repeat S2 and S3 until the total number of samples in the reachable workspace reaches the preset number or the sample distribution in the reachable workspace reaches the preset distribution state.

[0044] The total number of samples and the preset distribution can be flexibly set according to sampling requirements and actual conditions. To avoid operational data bias, a uniform distribution should be used to ensure that the number of samples in each spatial unit is similar. Alternatively, this embodiment can terminate the loop according to a preset number of repetitions, a preset collection time, or a preset number of successful samples.

[0045] Another embodiment of the present invention provides an automated robot operation data acquisition system, the system comprising: The space partitioning module is used to determine the robot's reachable workspace based on the target task and divide the reachable workspace into multiple spatial units; The data acquisition module is used to control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; to collect the operation data of the robot performing the current task operation, and to obtain the current sample of the current spatial unit; The position changing module is used to determine the next spatial unit corresponding to the next task operation based on the current sample and the historical samples of all spatial units, move the target object to the next spatial unit, and use the next spatial unit as the new current spatial unit; The repeated sampling module is used to repeatedly perform data acquisition and location changes until the total number of samples in the reachable workspace reaches a preset number or the sample distribution in the reachable workspace reaches a preset distribution state.

[0046] This embodiment divides the robot's reachable workspace into multiple spatial units. By performing task operations and data collection on the target object located in the current spatial unit, a current sample is obtained. Then, based on the current sample obtained from the current sampling and the historical samples obtained from each previous sampling, the spatial unit in which the target object should be located in the next sampling is determined. Accordingly, the position of the target object is changed in each sampling, realizing sampling under changing scenarios. This significantly improves the diversity and coverage of sample distribution, avoiding the problem of overfitting of the robot's decision model to specific scenarios caused by sample bias, and is conducive to improving the robustness and generalization ability of the decision model. On this basis, this embodiment triggers data collection through hardware triggering and sets a unified timestamp for the operation data of different modalities, achieving high-precision alignment of data of different modalities, which facilitates subsequent training of the decision model. In addition, this embodiment is adaptable to multiple scenarios such as home services, power inspection, and supermarket vending, with strong scalability and a wide range of applications.

[0047] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for acquiring data from the operation of an automated robot, characterized in that, The method includes: S1. Determine the robot's reachable workspace based on the target task, and divide the reachable workspace into multiple spatial units; S2. Control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; collect the operation data of the robot performing the current task operation to obtain the current sample of the current spatial unit; S3. Based on the current sample and the historical samples of all spatial units, determine the next spatial unit corresponding to the next task operation, move the target object to the next spatial unit, and use the next spatial unit as the new current spatial unit; S4. Repeat S2 and S3 until the total number of samples in the reachable workspace reaches a preset number or the sample distribution in the reachable workspace reaches a preset distribution state. S3 determines the next spatial unit corresponding to the next task operation based on the current sample and the historical samples of all spatial units, specifically including: Based on the current sample and the historical samples of all spatial units, determine the number of samples and the task failure rate for each spatial unit. The task failure rate is the percentage of samples in the spatial unit whose task completion status is failed. The next spatial unit corresponding to the next task operation is determined based on the number of samples and the task failure rate. Based on the current sample and the historical samples of all spatial units, determine the number of samples for each spatial unit, specifically including: Remove historical samples from all historical samples in each spatial unit whose time interval with the current moment is longer than a preset duration; The number of samples for each spatial unit is determined based on the current samples and the remaining historical samples for each spatial unit.

2. The method according to claim 1, characterized in that, The next spatial unit corresponding to the next task operation is determined based on the sample size and the task failure rate, specifically including: The sampling weight of the spatial unit is adjusted according to the number of samples and the task failure rate; The next spatial unit corresponding to the next task operation is determined based on the sampling weights of all spatial units after the adjustment.

3. The method according to claim 2, characterized in that, Adjusting the sampling weight of the spatial unit based on the number of samples and the task failure rate specifically includes: Increase the sampling weight of spatial units with a sample size less than a preset range, decrease the sampling weight of spatial units with a sample size greater than the preset range, and increase the sampling weight of spatial units with a task failure rate greater than a preset ratio.

4. The method according to claim 2, characterized in that, The next spatial unit corresponding to the next task operation is determined based on the adjusted sampling weights of all spatial units, specifically including: The spatial unit with the highest adjusted sampling weight is determined as the next spatial unit for the next task operation.

5. The method according to claim 1, characterized in that, S2 controls the robot to perform the current task operation on the target object located in the current spatial unit according to the target task, specifically including: The robot's trajectory is determined based on the target task and the current spatial unit where the target object is located; The robot is controlled to perform the current task operation on the target object according to the motion trajectory.

6. The method according to claim 1, characterized in that, The operational data includes at least one of the following: force sensing data of the robot, joint state data, operational image data, point cloud data, and audio data.

7. The method according to claim 6, characterized in that, S2 collects the operation data of the robot performing the current task, specifically including: At multiple moments during the robot's execution of the current task, force sensing data, joint state data, operation image data, point cloud data, and audio data of the robot are collected simultaneously, and a unified timestamp is set for the operation data of different modalities.

8. A system based on the automated robot operation data acquisition method according to any one of claims 1 to 7, characterized in that, The system includes: The space partitioning module is used to determine the robot's reachable workspace based on the target task and divide the reachable workspace into multiple spatial units; The data acquisition module is used to control the robot to perform the current task operation on the target object located in the current spatial unit according to the target task; to collect the operation data of the robot performing the current task operation, and to obtain the current sample of the current spatial unit; The position changing module is used to determine the next spatial unit corresponding to the next task operation based on the current sample and the historical samples of all spatial units, move the target object to the next spatial unit, and use the next spatial unit as the new current spatial unit; The repeated sampling module is used to repeatedly perform data acquisition and location changes until the total number of samples in the reachable workspace reaches a preset number or the sample distribution in the reachable workspace reaches a preset distribution state.

Images