An agent-based mechanical arm data acquisition and strategy deployment integrated system

Abstract

The application discloses a kind of mechanical arm data acquisition and strategy deployment integrated system based on agent, it is related to robot control and intelligent data processing technical field.The system includes cloud control unit, edge execution unit and equipment access unit, specifically including task analysis and disintegration module, data acquisition arrangement module, multi-camera equipment, mechanical arm controller, data quality detection and sample construction module, strategy selection module, strategy encapsulation and version management module, cloud edge collaborative deployment module and execution monitoring and abnormal disposal module etc..The system can automatically complete the joint acquisition of multi-camera image, end cartesian pose sequence and joint space trajectory, sample construction, strategy selection and edge deployment according to natural language task description, and trigger re-sampling, retry or rollback when execution timeout or data acquisition exception occurs.The scheme improves the data availability, deployment efficiency and abnormal recovery capability in the mechanical arm grabbing scene.

Description

Technical Field

[0001] This invention relates to the fields of robot control, intelligent data processing, and cloud-edge collaborative deployment technology, specifically to an integrated system for data acquisition and strategy deployment of a robotic arm based on an intelligent agent. Background Technology

[0002] Robotic arm grasping systems typically include a vision perception unit, a control execution unit, a trajectory planning unit, and a deployment and operation unit. In existing technologies, the acquisition of training samples, organization of task instructions, selection of model toolchains, and deployment of edge execution for robotic arm grasping tasks are usually performed by different systems or by different personnel. Taking multi-camera grasping tasks as an example, image data is generally acquired independently by the camera system, the end effector Cartesian pose sequence and joint space trajectory are recorded independently by the robotic arm controller, while natural language task descriptions, task labels, and execution results are stored in other business systems. There is a lack of unified organization and alignment mechanisms among these various data types.

[0003] The existing solutions have at least the following problems.

[0004] First, the data acquisition links from multiple sources are scattered. For robotic arm grasping tasks, image frames, multi-view videos, end-effector pose sequences, and joint trajectories are often acquired and stored separately, making it difficult to establish a stable temporal correspondence between different types of data under the same task.

[0005] Second, the task text and execution data have a weak correlation. Natural language task descriptions cannot be automatically converted into structured acquisition plans, making it difficult to establish a unified correlation between task objectives, capture constraints, acquisition steps, and strategy selection conditions and subsequent execution data.

[0006] Third, strategy selection relies on human experience. Faced with different objects to be grasped, different working environments, and different recognition accuracy requirements, it is usually necessary to manually decide whether to use a visual language action model, a visual language model, an object detection model, an image segmentation model, or a combination of several tools. This selection efficiency is low, and the results of different people's choices are inconsistent.

[0007] Fourth, the deployment process has a low degree of automation. After the cloud model or policy is updated, the encapsulation of the policy package, the distribution to the edge industrial control machine, the version switching and rollback processing still rely on manual operation, which can easily lead to problems such as long deployment time, untraceable versions and untimely anomaly recovery.

[0008] Fifth, there is a lack of closed-loop handling for anomalies. Common anomalies during robotic arm grasping tasks include execution timeouts, missing image frames, timestamp misalignments, missing trajectory segments, and interruptions in the acquisition process. Existing technologies often only display alarms and lack automated handling mechanisms for re-acquisition, rollback, retries, or manual intervention.

[0009] Therefore, there is a need for an integrated system that can simultaneously perform task decomposition, automated data acquisition, sample construction, strategy selection, cloud-edge collaborative deployment, and anomaly handling in robotic arm grasping scenarios, in order to improve data acquisition efficiency, strategy deployment efficiency, and task execution stability. Summary of the Invention

[0010] In view of this, the present invention provides an integrated solution for data acquisition and strategy deployment of a robotic arm based on an intelligent agent, aiming to solve or at least partially solve the following technical problems:

[0011] 1. How to convert natural language task descriptions into executable robotic arm data acquisition and deployment plans.

[0012] 2. How to uniformly acquire, label, and align multi-camera images, end-effector Cartesian pose sequences, and joint spatial trajectories.

[0013] 3. How to automatically select the appropriate strategy or model toolchain based on task characteristics and data collection results.

[0014] 4. How to automatically distribute policy packages to edge industrial control computers and drive robotic arms to execute them under a cloud-edge collaborative architecture.

[0015] 5. How to automatically trigger re-sampling, retry, rollback, or manual takeover when execution timeout or data acquisition anomaly occurs.

[0016] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides an integrated system for data acquisition and strategy deployment of a robotic arm based on an intelligent agent. This system includes: a cloud control unit, an edge execution unit, and a device access unit. Wherein: The cloud-based control unit includes: 1. Task parsing and decomposition module, used to receive natural language crawling task descriptions and generate structured task objects containing task objectives, crawling constraints, collection requirements and termination conditions.

[0017] 2. Data acquisition and orchestration module, used to generate acquisition scheduling instructions based on structured task objects, and determine the triggering timing and sampling parameters for multi-camera image acquisition, end-effector Cartesian pose acquisition, and joint spatial trajectory acquisition.

[0018] 3. Data quality inspection and sample construction module, used for time alignment, task labeling, sample encapsulation and anomaly detection of image data, pose sequences, joint trajectories and task descriptions.

[0019] 4. Strategy selection module, used to determine the target strategy from visual language action model, visual language model, object detection model, segmentation model and their combination toolchain based on task object, sample features, target object features and model capability information.

[0020] 5. The strategy encapsulation and version management module is used to encapsulate the target strategy into a deployable strategy package and record the strategy version, configuration version, timeout threshold, and rollback target version information.

[0021] 6. The cloud-edge collaborative deployment module is used to distribute policy packages to edge execution units and receive deployment results and running status.

[0022] The edge execution unit includes: 1. Data acquisition agent, used to drive the multi-camera and robotic arm control interface to perform data acquisition according to the acquisition scheduling instructions.

[0023] 2. Policy loading agent, used to receive and load policy packages.

[0024] 3. The execution monitoring and anomaly handling module is used to monitor the robotic arm's grasping execution status, detect execution timeouts and data acquisition anomalies, and trigger pause, retry, re-acquisition, rollback, or manual takeover.

[0025] 4. Robotic arm execution interface module, used to send control commands to the robotic arm controller and receive execution feedback.

[0026] The device access unit includes: 1. Multi-camera device used to capture multi-view image frames or video streams during the capture process.

[0027] 2. Robotic arm controller, used to provide end-effector Cartesian pose sequence, joint space trajectory and control execution feedback.

[0028] 3. End effector, used to perform the grasping action.

[0029] Preferably, the task parsing and decomposition module parses the natural language capture task description into task objects through preset task parsing rules or intelligent agent reasoning processes. The task objects include at least a task identifier, capture target, capture area, target placement area, perception requirements, collection requirements, and anomaly handling strategies.

[0030] Preferably, the data acquisition and arrangement module generates a unique task ID for each task object and writes the task ID, camera number, frame number, trajectory sampling number and timestamp into a unified metadata structure to establish the association between image data, pose sequence, joint trajectory and task text.

[0031] Preferably, the data quality detection and sample construction module includes a time synchronization unit, a sample slicing unit, and an anomaly detection unit. The time synchronization unit aligns the image frame timestamps, pose sampling timestamps, and joint sampling timestamps according to a unified clock or mapping rule. The sample slicing unit divides the continuously acquired data into three stages: pre-grabbing, during-grabbing, and post-grabbing, based on the task steps. The anomaly detection unit detects missing frames, misaligned timestamps, missing trajectory segments, missing labels, or interrupted acquisition based on preset thresholds.

[0032] Preferably, the strategy selection module selects a target strategy from the candidate toolchains based on the target type, perception requirements, and execution constraints recorded in the task object, combined with sample quality results and model capability descriptions. Specifically, when the task requires target localization, a toolchain containing target detection or image segmentation capabilities is preferred; when the task requires direct generation of action planning from natural language descriptions, a toolchain containing visual language action capabilities is preferred; and when the task requires language understanding to assist perception, a toolchain containing visual language understanding capabilities is preferred.

[0033] Preferably, the strategy encapsulation and version management module encapsulates the target strategy into a strategy package, which includes at least a model file, configuration file, preprocessing rules, postprocessing rules, deployment script, timeout threshold, data anomaly threshold, and rollback pointer.

[0034] Preferably, the cloud-edge collaborative deployment module distributes the policy package to the edge execution unit through the network communication link, and requests the edge execution unit to return the policy loading status, version verification result, and deployment completion status.

[0035] Preferably, the execution monitoring and anomaly handling module sets a total timeout threshold and a stage timeout threshold for the execution process. When any threshold is triggered, a timeout event is output. It also sets a frame integrity threshold, a time synchronization error threshold, and a trajectory continuity threshold for the acquisition process. When any threshold is not met, a data acquisition anomaly event is output. For timeout events and data acquisition anomaly events, the execution monitoring and anomaly handling module executes pause, retry, re-acquisition, historical rollback strategy packages, or initiates manual takeover according to preset handling rules.

[0036] Secondly, the present invention also provides an integrated method for data acquisition and strategy deployment of a robotic arm based on an intelligent agent. Using the aforementioned system, data acquisition and strategy deployment of the robotic arm are performed. This method includes the following steps: S1 receives a natural language description of the data crawling task input by the user and generates a structured task object; S2, based on the structured task object, decompose the capture task and generate a collection plan and strategy selection conditions; S3 controls multiple camera devices to acquire image data and simultaneously acquires the Cartesian pose sequence and joint space trajectory of the robotic arm end effector; S4, perform time alignment, task labeling, and sample construction on the image data, pose sequence, joint space trajectory, and task description; S5, Perform data quality detection on the collected results to determine whether there are missing frames, abnormal timestamps, abnormal trajectories, abnormal labels, or collection interruptions; S6. Determine the target strategy based on the structured task object, data quality results, and model capability information; S7, the target strategy is encapsulated into a strategy package and sent to the edge execution unit. After the edge execution unit loads the strategy package, it drives the robotic arm to perform the grasping task. S8 monitors the execution and data acquisition status during execution. When an execution timeout or data acquisition anomaly is detected, it triggers pause, retry, reacquisition, rollback, or manual takeover.

[0037] Compared with the prior art, the present invention has at least the following beneficial technical effects: This invention can simultaneously achieve task decomposition, automated data acquisition, sample construction, strategy selection, cloud-edge collaborative deployment, and anomaly handling in robotic arm grasping scenarios, thereby improving data acquisition efficiency, strategy deployment efficiency, and task execution stability.

[0038] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0042] Figure 1 A schematic diagram of an integrated system architecture for data acquisition and strategy deployment of a robotic arm based on an intelligent agent, provided in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram illustrating the deployment relationship between the cloud control unit, edge execution unit, and device access unit provided in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the data acquisition and sample construction process for a robotic arm grasping task provided in an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram illustrating the strategy selection, strategy package encapsulation, and cloud-edge collaborative deployment process provided in this embodiment of the invention.

[0046] Figure 5 This is a schematic diagram of the execution timeout and data acquisition anomaly handling process provided in an embodiment of the present invention.

[0047] Figure 6 This is a schematic diagram of the end-to-end timing of a robotic arm grasping task provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0050] Combination Figure 1 and Figure 2 As shown, this invention provides an integrated system for data acquisition and strategy deployment of a robotic arm based on an intelligent agent. This embodiment of the integrated system includes a cloud server (i.e., a cloud control unit), an edge industrial control computer (i.e., an edge execution unit), and a device access unit (including a robotic arm controller, a multi-camera device, and an end effector). Wherein: The cloud server includes a task parsing and decomposition module, a data acquisition and orchestration module, a data quality detection and sample construction module, a strategy selection module, a strategy encapsulation and version management module, and a cloud-edge collaborative deployment module.

[0051] The edge industrial control computer includes a data acquisition agent, a policy loading agent, an execution monitoring and anomaly handling module, and a robotic arm execution interface module.

[0052] Multiple cameras are positioned around the robotic arm's workspace to acquire images of the grasping area from multiple perspectives. The robotic arm controller is connected to the robotic arm body to output the end effector's Cartesian pose sequence and joint space trajectory, and to receive control commands from the robotic arm's execution interface module.

[0053] In this embodiment, the user first inputs a natural language description of the task, such as "grab the black workpiece on the conveyor belt and place it in the left-hand turnover box". The task parsing and decomposition module analyzes the task description to obtain a structured task object. The structured task object includes one or more of the following: task identifier, description of the grasped object, target placement area, execution stage division, collection requirements, perception requirements, and anomaly handling strategy.

[0054] Specifically, the task parsing and decomposition module predefines the JSON data format for structured task objects. This JSON data format includes fields such as task identifier, crawling target, crawling object attributes, crawling area, target placement area, execution stage, collection requirements, perception requirements, termination conditions, and exception handling strategies. Upon receiving the user's natural language crawling task description, the task parsing and decomposition module inputs the natural language crawling task description, the predefined JSON data format, and field filling rules into the large language model. The large language model performs semantic understanding and field extraction on the crawling object, object attributes, starting position, target position, action intent, collection requirements, and constraints in the task description, and outputs structured JSON text according to the predefined JSON data format. The task parsing and decomposition module performs field integrity and format validity checks on the structured JSON text. When missing fields or format errors exist, the module completes and corrects them based on preset default values, context information, or by re-calling the large language model, ultimately obtaining a structured task object that can be used by the data acquisition and orchestration module.

[0055] The data acquisition and orchestration module generates acquisition scheduling instructions based on structured task objects. These instructions include camera activation time, acquisition frequency, pose sampling frequency, joint trajectory sampling frequency, stage slicing rules, and acquisition termination conditions. Specifically, after receiving a structured task object, the module extracts the task identifier, grasping area, execution stage, acquisition requirements, perception requirements, and termination conditions, and generates acquisition scheduling instructions according to preset field-to-scheduling parameter mapping rules. Specifically, it determines the cameras participating in acquisition based on the grasping area and perception requirements; determines the image acquisition frequency, end-effector Cartesian pose sampling frequency, and joint trajectory sampling frequency based on the acquisition requirements and the device's supported sampling range; determines the stage slicing rules based on the robotic arm execution status, end-effector opening / closing status, and grasping completion signal corresponding to each execution stage; determines the camera activation time based on the task start event; and determines the acquisition termination conditions based on task completion, release completion, execution timeout, or abnormal termination events. Finally, the task identifier, camera number, various sampling frequencies, stage slicing rules, and acquisition termination conditions are encapsulated into acquisition scheduling instructions and sent to the data acquisition agent for execution.

[0056] The data acquisition agent controls multiple camera devices to start acquiring image data according to the acquisition scheduling instructions, and at the same time reads the Cartesian pose sequence and joint space trajectory of the robotic arm end effector through the robotic arm controller.

[0057] During the acquisition process, each image frame is written with a task ID, camera number, frame number, and first timestamp; each pose sampling point is written with a task ID, pose number, and second timestamp; and each joint trajectory sampling point is written with a task ID, trajectory number, and third timestamp.

[0058] After receiving the above data, the data quality inspection and sample construction module performs alignment processing based on a unified time reference to establish the correspondence between image frames, pose sampling points, and joint trajectory sampling points. The specific alignment process is as follows: Upon receiving image frames, end-effector Cartesian pose sampling points, and joint trajectory sampling points, the data quality inspection and sample construction module first groups the data under the same capture task according to the task ID, and then sorts the image frame sequence, pose sequence, and joint trajectory sequence according to the first, second, and third timestamps, respectively. Subsequently, a unified reference timeline is generated using the image frame timestamp or a preset sampling period. At each reference time point, the sampling point with the smallest time difference is found from the pose sequence and joint trajectory sequence. When the time difference is less than a preset synchronization error threshold, the corresponding image frame, end-effector Cartesian pose sampling point, and joint trajectory sampling point are written into the same alignment record. For cases where inconsistent sampling frequencies result in no identical timestamps, the data quality inspection and sample construction module can perform linear interpolation based on two adjacent sampling points to obtain the pose data or joint trajectory data corresponding to the reference time point. After alignment, an alignment index table containing the task ID, reference timestamp, image frame identifier, pose sampling identifier, joint trajectory sampling identifier, and synchronization error value is generated to establish the correspondence between image frames, pose sampling points, and joint trajectory sampling points.

[0059] The data quality inspection and sample construction module further performs sample slicing on the collected data according to the stages of the grasping task. Taking a complete grasp as an example, it can be divided into the target approach stage, the grasping execution stage, and the transport and placement stage. The image data, pose sequences, joint trajectories, and task text descriptions of each stage are encapsulated into stage sample packages or whole task sample packages.

[0060] In this embodiment, the specific process of sample slicing is as follows: After completing time alignment, the data quality detection and sample construction module performs sample slicing on continuously acquired data under the same task ID based on the alignment index table. The data quality detection and sample construction module can determine the start and end points of each task stage according to the stage slicing rules in the acquisition scheduling instruction, the robotic arm execution status, the end effector opening and closing status, the grasping completion signal, and the preset time window.

[0061] For example, when the robotic arm's end effector gradually approaches the target before closing, the corresponding time interval is marked as the target approach stage; when the end effector performs a closing action and generates a grasping confirmation signal, the corresponding time interval is marked as the grasping execution stage; when the robotic arm carries the target object to the target placement area and performs a release action, the corresponding time interval is marked as the transport and placement stage. After determining the start and end times of each stage, the data quality inspection and sample construction module extracts the corresponding image frames, end effector Cartesian pose sampling points, and joint trajectory sampling points from the alignment index table according to the start and end times, and encapsulates them together with the task text description, stage label, timestamp range, and execution result into a stage sample package; at the same time, multiple stage sample packages can also be merged according to the task ID to form a complete task sample package.

[0062] In this embodiment, the anomaly detection unit determines data acquisition anomalies according to the following rules: 1. When the number of missing image frames in a preset acquisition window of a certain camera exceeds the first threshold, it is determined to be an image frame missing anomaly; 2. When the offset between the image timestamp and the pose timestamp exceeds the second threshold, it is determined to be a timestamp misalignment anomaly; 3. When the number of joint trajectory sampling points is lower than the preset lower limit or the interval between adjacent sampling points exceeds the third threshold, it is determined to be an abnormal trajectory segment. 4. When a required label field is missing in a task object, it is determined to be a label missing exception; 5. If the data acquisition agent fails to return the data acquisition status within a preset time, it is determined to be a data acquisition interruption error.

[0063] Preferably, the acquisition window can be set to 1 second, and the first threshold can be set to 3 frames; when a camera misses more than 3 image frames within the 1-second acquisition window, it is determined to be an image frame missing anomaly.

[0064] The second threshold can be set to 50 milliseconds; when the offset between the image timestamp and the corresponding pose timestamp is greater than 50 milliseconds, it is determined to be a timestamp misalignment anomaly.

[0065] The third threshold can be set to 100 milliseconds; when the time interval between adjacent joint trajectory sampling points is greater than 100 milliseconds, or the number of joint trajectory sampling points is lower than the minimum number corresponding to the preset sampling frequency, it is judged as a trajectory segment missing anomaly. For the acquisition agent status return timeout, the preset time can be set to 2 seconds; when the acquisition agent does not return the acquisition status within 2 seconds, it is judged as an acquisition interruption anomaly.

[0066] The above thresholds are for illustrative purposes only and can be adjusted based on camera frame rate, robotic arm control cycle, network latency, and task accuracy requirements.

[0067] If any of the above anomalies are detected, the data quality inspection and sample construction module will associate the anomaly with the task ID and send an anomaly event to the execution monitoring and anomaly handling module. For anomalies during the data acquisition phase, the execution monitoring and anomaly handling module can output a re-acquisition command to re-trigger the data acquisition for the corresponding phase.

[0068] After the sample construction is completed, the strategy selection module selects the target strategy based on the task object and sample features. When the task description emphasizes the category of the grasping object and the target localization accuracy requirement is high, the strategy selection module prioritizes the perception toolchain composed of the object detection model and the image segmentation model; when the task needs to be directly mapped from natural language tasks to action planning, the strategy selection module prioritizes the visual language action model; when the task needs to perform semantic understanding of the scene and assist in judging the grasping steps, the strategy selection module can use the visual language model as an auxiliary tool and combine it with the detection model. Preferably, the object detection model can adopt the YOLO series object detection model to identify the category of the grasping object and output the target bounding box; the image segmentation model can adopt the SAM3 segmentation model to obtain the pixel-level contour or mask region of the grasping object; the visual language model adopts the Qwen series multimodal model to perform semantic matching and scene understanding of the task text and the grasping scene image; the visual language action model adopts pi0.5 or a robot basic model with visual language to action mapping capability to generate action planning or control instructions based on the image, task text and robotic arm status. The above model is only an example. In actual deployment, other models or combined toolchains with the same functions can be selected according to the type of robotic arm, camera configuration, computing resources, grasping accuracy requirements and model inference latency.

[0069] The strategy encapsulation and version management module generates a strategy package after determining the target strategy. The strategy package includes at least the model file, configuration file, environment dependencies, preprocessing rules, postprocessing rules, timeout threshold, exception judgment threshold, and rollback target version.

[0070] Preferably, the preprocessing rules include rules for image scaling, image normalization, camera distortion correction, coordinate system transformation, timestamp synchronization, task text encoding, and robot arm state standardization, used to convert multi-camera images, task text, and robot arm states into an input format recognizable by the target strategy. The post-processing rules include rules for target detection box filtering, segmentation mask filtering, grasping point calculation, action sequence smoothing, collision constraint checking, execution result judgment, and abnormal result marking, used to convert the model output into grasping instructions or task results executable by the robot arm. Anomaly judgment thresholds may include model inference timeout thresholds, target confidence thresholds, grasping point deviation thresholds, execution phase timeout thresholds, image frame missing thresholds, time synchronization error thresholds, and trajectory continuity thresholds. The rollback target version is used to indicate the historical stable strategy package version switched to by the edge control computer when strategy loading fails, model inference times out continuously, grasping fails continuously, or the number of data acquisition anomalies exceeds a preset number.

[0071] By encapsulating the above processing rules and thresholds along with the policy package, it can be ensured that the policies determined in the cloud run on the edge industrial control computer in accordance with consistent input processing, output processing, anomaly judgment and rollback logic.

[0072] Subsequently, the cloud-edge collaborative deployment module sends the policy package to the edge industrial control machine and requires the edge industrial control machine to perform version verification, dependency verification, and loading confirmation.

[0073] The policy loading agent receives the policy package in the edge industrial control computer and completes its local loading. Subsequently, the robotic arm execution interface module generates robotic arm execution instructions based on the loaded policy package and sends them to the robotic arm controller to drive the robotic arm to perform the grasping task. During execution, the execution monitoring and exception handling module continuously monitors the total execution time, stage execution time, grasping completion flag, and acquisition status.

[0074] In this embodiment, the execution monitoring and exception handling module uses a two-level judgment for execution timeout: 1. If the execution time of the entire crawling task exceeds the total timeout threshold, a task-level timeout event will be generated; 2. If the execution time of a certain stage exceeds the stage timeout threshold, a stage-level timeout event will be generated.

[0075] When a phase-level timeout occurs, pause and retry should be prioritized. If multiple retries fail or a task-level timeout occurs, a rollback should be performed, switching the current policy package to a historical stable version and redeploying it. If the task still cannot be completed after the rollback, manual takeover should be initiated.

[0076] Through the aforementioned system, task parsing, data acquisition and arrangement, sample construction, strategy selection, and deployment of the robotic arm's grasping task form a unified closed loop. Compared with traditional manual configuration and deployment methods, this embodiment can significantly reduce human intervention and improve data availability and deployment efficiency.

[0077] Furthermore, combined Figures 3 to 6 As shown, the present invention also provides an integrated method for data acquisition and strategy deployment of a robotic arm based on an intelligent agent, comprising the following steps.

[0078] Receive a natural language crawling task description and generate a structured task object.

[0079] Based on the structured task object, the task steps are broken down to generate a data collection plan, perception requirements, and strategy selection conditions.

[0080] Control multiple camera devices to acquire image frames, and simultaneously acquire the Cartesian pose sequence and joint space trajectory of the robotic arm end effector.

[0081] The acquired image frames, pose sequences, joint trajectories, and task text are uniformly labeled and time-aligned.

[0082] Perform data quality checks on the aligned acquisition results and construct a joint sample package based on the capture phase.

[0083] Determine the target strategy or toolchain based on the task object, sample quality, and model capability description.

[0084] The target strategy is encapsulated into a strategy package and then distributed to the edge industrial control computer via the cloud-edge collaborative link.

[0085] After loading the strategy package on the edge industrial control computer, the robotic arm is driven to perform the grasping task, and execution timeouts and data acquisition anomalies are continuously monitored.

[0086] When an execution timeout or data acquisition anomaly is detected, pause, retry, reacquisition, rollback, or manual takeover are triggered.

[0087] Each step in this embodiment can be implemented entirely by software, or partially by software and partially by hardware circuitry. Without departing from the inventive concept, the order of the steps can also be adjusted according to actual needs.

[0088] As can be seen from the description of the above embodiments, the present invention provides an integrated system and method for data acquisition and strategy deployment of a robotic arm based on an intelligent agent, the technical advantages of which include the following: 1. Integrate natural language task parsing, automated data acquisition, sample construction, strategy selection, and edge deployment into a unified closed loop to improve the integration of the robotic arm grasping task chain.

[0089] 2. Improve the consistency of association between multi-camera images, end-effector pose sequences, and joint trajectories by using unified task IDs, unified timestamps, and unified metadata structures.

[0090] 3. By automating data acquisition and arrangement and data quality inspection, the effectiveness and usability of sample collection can be improved, and the cost of manual sample screening can be reduced.

[0091] 4. By automatically selecting policies and automatically delivering policy packages, the path from policy selection to edge effectiveness is shortened, improving deployment efficiency.

[0092] 5. By performing timeout detection and data acquisition anomaly judgment, rollback, re-acquisition, retry, or manual takeover can be performed in a timely manner, improving the stability of task execution and recovery efficiency.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as products such as systems, methods, or electronic devices. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An integrated system for data acquisition and strategy deployment of a robotic arm based on intelligent agents, characterized in that, The system includes: a cloud control unit, an edge execution unit, and a device access unit; wherein: The cloud-based control unit includes: The task parsing and decomposition module is used to receive natural language crawling task descriptions and generate structured task objects; The data acquisition and arrangement module is used to generate acquisition scheduling instructions based on the structured task object, and to determine the triggering timing and sampling parameters for multi-camera image acquisition, end-effector Cartesian pose acquisition, and joint spatial trajectory acquisition. The data quality inspection and sample construction module is used to perform time alignment, sample encapsulation, and anomaly detection on image data, pose sequences, joint trajectories, and task descriptions. The strategy selection module is used to select the target strategy based on the task object, sample features, target object features, and model capability information. The strategy encapsulation and version management module is used to encapsulate the target strategy into a deployable strategy package and record the strategy version, configuration version, timeout threshold, and rollback target version information. The cloud-edge collaborative deployment module is used to distribute policy packages to the edge execution unit and receive deployment results and running status. The edge execution unit includes: The execution monitoring and anomaly handling module is used to monitor the robotic arm's grasping execution status, detect execution timeouts and data acquisition anomalies, and trigger handling actions. The device access unit includes: A multi-camera device is used to acquire image data of the robotic arm grasping process according to the acquisition scheduling instructions, and to feed the image data back to the cloud control unit and the edge execution unit.

2. The system according to claim 1, characterized in that, The task parsing and decomposition module parses the natural language crawling task description into a structured task object through preset task parsing rules or intelligent agent reasoning process. The structured task object includes at least a task identifier, a crawling target, a crawling area, a target placement area, a perception requirement, a collection requirement, and an anomaly handling strategy.

3. The system according to claim 1, characterized in that, The data acquisition and orchestration module generates a unique task ID for each task object and writes the task ID, camera number, frame number, trajectory sampling number, and timestamp into a unified metadata structure to establish the association between image data, pose sequence, joint trajectory, and task text. The acquisition scheduling instructions generated by the data acquisition and orchestration module include camera start time, acquisition frequency, pose sampling frequency, joint trajectory sampling frequency, stage slicing rules, and acquisition end conditions.

4. The system according to claim 1, characterized in that, The data quality detection and sample construction module includes a time synchronization unit, a sample slicing unit, and an anomaly detection unit. The time synchronization unit aligns the image frame timestamps, pose sampling timestamps, and joint sampling timestamps according to a unified clock or mapping rule. The sample slicing unit divides the continuously acquired data into three stages: pre-capture, mid-capture, and post-capture, according to the task steps. The anomaly detection unit detects missing frames, misaligned timestamps, missing trajectory segments, missing labels, or interrupted acquisition based on preset thresholds.

5. The system according to claim 1, characterized in that, The strategy selection module selects a target strategy from the candidate toolchain; wherein, when the task requires target localization, a toolchain containing target detection or image segmentation capabilities is used; when the task requires direct generation of action planning from natural language description, a toolchain containing visual language action capabilities is used; and when the task requires assisted perception through language understanding, a toolchain containing visual language understanding capabilities is used.

6. The system according to claim 1, characterized in that, The strategy encapsulation and version management module encapsulates the target strategy into a deployable strategy package. The strategy package includes at least a model file, configuration file, preprocessing rules, postprocessing rules, deployment script, timeout threshold, data anomaly threshold, and rollback pointer.

7. The system according to claim 1, characterized in that, The cloud-edge collaborative deployment module distributes the policy package to the edge execution unit through the network communication link and requests the edge execution unit to return the policy loading status, version verification result, and deployment completion status.

8. The system according to claim 1, characterized in that, The execution monitoring and anomaly handling module sets a total timeout threshold and a stage timeout threshold for the execution process. When any threshold is triggered, a timeout event is output. It also sets a frame integrity threshold, a time synchronization error threshold, and a trajectory continuity threshold for the acquisition process. When any threshold is not met, a data acquisition anomaly event is output. For timeout events and data acquisition anomaly events, the execution monitoring and anomaly handling module initiates manual takeover or one or more handling actions from the execution strategy package according to preset handling rules.

9. The system according to claim 1, characterized in that, The edge execution unit further includes: The data acquisition agent is used to drive the control interfaces of multiple cameras and robotic arms to perform data acquisition according to the acquisition scheduling instructions. The policy loading agent is used to receive and load policy packages; The robotic arm execution interface module is used to send control commands to the robotic arm controller and receive execution feedback; The device access unit also includes: The robotic arm controller is used to provide end-effector Cartesian pose sequence, joint space trajectory and control execution feedback, as well as control the movement of the end-effector; An end effector is used to perform the grasping action.

10. A method for integrating data acquisition and strategy deployment of a robotic arm based on an intelligent agent, characterized in that, Using the system of any one of claims 1–9, the method comprises: Receive a natural language crawling task description and generate a structured task object; Based on the structured task object, the capture task is decomposed, and a collection plan and strategy selection conditions are generated; Control multiple camera devices to acquire image data, and simultaneously acquire the Cartesian pose sequence and joint space trajectory of the robotic arm end effector; Temporal alignment, task labeling, and sample construction are performed on image data, pose sequences, joint space trajectories, and task descriptions. Perform data quality checks on the collected results; Determine the target strategy based on the task object, data quality results, and model capability information; The target strategy is encapsulated into a strategy package and distributed to the edge execution unit; After loading the strategy package at the edge execution unit, the robotic arm is driven to perform the grasping task. When an execution timeout or data acquisition anomaly is detected, exception handling is triggered.

Images