Cloud edge terminal cooperative mechanical arm intelligent perception and control system and method

Abstract

The application discloses a cloud edge end cooperative mechanical arm intelligent perception and control system and method, and belongs to the technical field of robot vision control and cloud edge end cooperation. A shape perception lightweight vision detection algorithm is introduced at the edge end, high-precision extraction of the positions and geometric contours of an end, a target object and an obstacle is realized, and real-time mapping to a monitoring interface and a cloud end is realized. The cloud end generates movement and obstacle avoidance instructions in real time according to multi-dimensional perception feedback, thereby realizing intelligent and high-robust operation of the mechanical arm in a dynamic scene, and solving the problems of weak edge perception geometric capability, incomplete multi-target perception, high cloud edge cooperation closed loop delay and the like of an existing cloud edge end mechanical arm system.

Description

Technical Field

[0001] This invention belongs to the field of robot vision control and cloud-edge-device collaboration technology, specifically relating to a cloud-edge-device collaborative intelligent perception and control system and method for robotic arms. Background Technology

[0002] In fields such as industrial manufacturing, warehousing and logistics, and scientific research and education, robotic arms often need to perform target grasping and obstacle avoidance operations in dynamic, unstructured environments. Traditional robotic arm control systems typically employ a centralized architecture, with visual perception, path planning, and motion control all performed on a local industrial computer or host computer. However, this architecture has significant limitations: on the one hand, complex visual algorithms and path planning consume a large amount of computing power, resulting in poor system real-time performance; on the other hand, the computing power of terminal devices is limited, making it difficult to run high-precision visual perception models. With the development of edge computing and cloud computing, cloud-edge-device collaborative architectures have been introduced into robot control. However, existing cloud-edge-device robotic arm systems generally suffer from the following drawbacks: Limited edge perception and insufficient geometric shape perception: Existing lightweight object detection algorithms (such as YOLO-Tiny, NanoDet, PP-PicoDet, etc.) primarily focus on balancing parameter quantity and speed, lacking sufficient ability to perceive the geometric shape of objects. In robotic arm operations, it is not only necessary to know the category and rough location of the target object, but also to accurately obtain the geometric contours and aspect ratios of the target object and obstacles in order to plan the grasping posture and obstacle avoidance path. Existing algorithms cannot provide reliable geometric dimension information with extremely low parameter quantities.

[0003] Lack of multi-target collaborative perception and closed-loop linkage: Existing systems often only use the edge to identify targets, failing to incorporate the robotic arm end effector, target, and obstacles into a unified edge perception framework. This results in the cloud lacking real-time and accurate feedback on the end pose and obstacle contours when planning paths, making it difficult to achieve tightly coupled dynamic obstacle avoidance and grasping closed-loop control.

[0004] The decoupling of heterogeneous communication and algorithms is not flexible enough: In the multi-level heterogeneous network of K210, microcontroller and cloud server, the algorithm task allocation and communication protocol design are not reasonable enough, resulting in a large delay in the control commands issued by the cloud decision, which cannot meet the millisecond-level response requirements of the robotic arm for real-time obstacle avoidance in dynamic scenarios.

[0005] Therefore, under the stringent resource constraints of edge devices, designing a robotic arm collaborative control system that can explicitly enhance geometric and multi-target perception capabilities and deeply integrate cloud decision-making and edge feedback has become an urgent technical problem to be solved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a cloud-edge-device collaborative intelligent perception and control system and method for robotic arms. By introducing a lightweight shape-aware visual detection algorithm at the edge, it achieves high-precision extraction of the position and geometric contours of the end effector, target objects, and obstacles, and maps them to the monitoring interface and the cloud in real time. The cloud generates motion and obstacle avoidance commands in real time based on multi-dimensional perception feedback, thereby enabling intelligent and robust operation of the robotic arm in dynamic scenarios. This solves the problems of weak edge perception geometric capabilities, incomplete multi-target perception, and high latency in cloud-edge collaborative closed-loop systems in existing cloud-edge-device robotic arm systems.

[0007] The technical solution of this invention is as follows: On the one hand, the present invention provides a cloud-edge-device collaborative intelligent sensing and control system for robotic arms, including an edge sensing module, a host computer monitoring and relay module, a cloud decision-making module, and an execution control module.

[0008] The edge perception module is used to perceive three types of targets: the end effector of the robotic arm, the target object, and the obstacle, and to acquire multi-dimensional edge perception data and send it to the execution control module.

[0009] The cloud-based decision-making module is used to receive edge-aware multidimensional data forwarded by the host computer monitoring and relay module, generate control commands, and send them to the host computer monitoring and relay module.

[0010] The host computer monitoring and relay module is used to receive edge-aware multidimensional data forwarded by the execution control module and control commands output by the cloud decision module, and forward the control commands to the execution control module and the edge-aware multidimensional data to the cloud decision module; construct and dynamically update the virtual scene based on the edge-aware multidimensional data, receive the angles of each servo joint of the robotic arm fed back by the execution control module, and obtain the actual motion trajectory of the robotic arm based on the fed-back angles of each servo joint of the robotic arm, thereby realizing the visualization of the virtual scene and the actual motion trajectory of the robotic arm.

[0011] The execution control module is used to receive edge-sensing multi-dimensional data and forward it to the host computer monitoring and relay module; receive control commands output by the host computer monitoring and relay module and execute the action of grasping the target object or avoiding obstacles, and feed back the angle of each servo joint of the robotic arm to the host computer monitoring and relay module.

[0012] Furthermore, the edge perception module includes a vision unit and a lightweight shape perception algorithm unit.

[0013] The vision unit is used to acquire RGB images of the robotic arm's working area in real time and send them to the lightweight shape perception algorithm unit.

[0014] The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet network to simultaneously detect and geometrically model three types of targets—the robotic arm end effector, the target object, and the obstacle—from the RGB image of the robotic arm's working area. Finally, it outputs the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline as edge perception multidimensional data, and sends them to the execution control module via the I2C interface.

[0015] Furthermore, the cloud-based decision-making module includes a cloud server and an intelligent planning algorithm unit. The cloud server communicates with the PC in the host computer monitoring and relay module via a Socket network, receives edge-aware multi-dimensional data forwarded by the PC, and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking and path planning algorithms. It generates a grasping posture based on the coordinates of the robotic arm end and the target object and geometric contour box in the received edge perception multidimensional data. It also plans a collision-free joint trajectory in real time based on the obstacle coordinates and geometric contour box. The grasping posture and collision-free joint trajectory are converted into control commands and sent to the PC host computer in the host computer monitoring and relay module.

[0016] Furthermore, the host computer monitoring and relay module includes a PC host computer and a monitoring interface.

[0017] The PC host computer communicates bidirectionally with the execution control module via a serial port and with the cloud decision module via a Socket network. It is used to receive edge-aware multidimensional data and forward it to the cloud decision module, and to receive control commands and forward them to the execution control module. Based on the edge-aware multidimensional data, it renders and dynamically updates the target objects, obstacles, and robotic arm in the virtual scene in real time and sends the virtual scene to the monitoring interface. It receives the angles of each servo joint of the robotic arm fed back by the execution control module, obtains the actual motion trajectory of the robotic arm end effector, and sends it to the monitoring interface synchronously.

[0018] The monitoring interface is used to display targets, obstacles, and robotic arms in the virtual scene, while also displaying the actual movement trajectory of the robotic arm's end effector, thus achieving digital twin.

[0019] Furthermore, the execution control module includes a motion control board and a six-axis robotic arm.

[0020] The motion control board receives multi-dimensional data from edge sensing and forwards it to the host computer monitoring and relay module; it receives... Control commands from the PC host computer are parsed and output as PWM signals to drive the servo joints of the six-axis robotic arm to perform actions such as grasping targets or avoiding obstacles; at the same time, the angles of each servo joint are read and reported to the PC host computer via serial port, completing the entire cloud-edge-device collaborative closed loop.

[0021] Furthermore, the improved Shape-Aware NanoDet network improves upon the original standard lightweight backbone network ShuffleNetV2 by introducing a dual-path structure in its Stage 3 and Stage 4 stages. Specifically, Stage 3 and Stage 4 are replaced with a dual-path feature extraction backbone network. The original structure of Stage 3 and Stage 4 consists of ordinary blocks and downsampling blocks. The dual-path feature extraction backbone network embeds the dual-path structure into each block, replacing the ordinary blocks and downsampling blocks of Stage 3 and Stage 4 with dual-path ordinary blocks and dual-path downsampling blocks. At the same time, an adaptive weighted feature pyramid network is added after Stage 4.

[0022] The dual-path ordinary block first splits the input feature map into two channels at a 1:1 ratio, resulting in two split feature maps. Dual-path feature extraction is then performed on one of the split feature maps, including a first semantic path and a first geometric path. The first semantic path is a first depthwise separable convolutional group, and the first geometric path is a lightweight edge enhancement module. Finally, the feature maps output by the semantic path and the geometric path are concatenated by channels to form a first fused feature. The first fused feature is then integrated with the other split feature map by channel dimension to obtain the output feature map.

[0023] The dual-path downsampling block includes a dual-path downsampling branch and a dual-path downsampling and convolution branch. In the dual-path downsampling branch, the input feature map is processed through the second semantic path and the second geometric path for feature extraction and channel concatenation to obtain the second fused feature. In the dual-path downsampling and convolution branch, the input feature map is processed through the third semantic path and the third geometric path for feature extraction and channel concatenation to obtain the third fused feature. Finally, the second fused feature and the third fused feature are concatenated to obtain the output feature map.

[0024] The adaptive weighted feature pyramid network automatically learns a weight vector for the output feature maps of Stage 3 and Stage 4. And perform a weighted summation of the output feature maps of Stage 3 and Stage 4: (2); in, For the level number, Output feature map for Stage 3 or Stage 4; The output feature map of the adaptive weighted feature pyramid network; Let be the weight vector, where =1; This refers to either an upsampling operator or a downsampling operator.

[0025] Furthermore, the first depthwise separable convolutional group sequentially includes convolution and depthwise separable convolution; the lightweight edge enhancement module first performs asymmetric convolution in parallel on the input feature map, and then concatenates the output features of the two asymmetric convolutions by channel, and completes channel integration by pointwise fusion convolution.

[0026] The second semantic path sequentially includes a depthwise separable convolution and a convolution; the third semantic path sequentially includes a convolution, a depthwise separable convolution, and a convolution; both the second and third geometric paths sequentially include a lightweight edge enhancement module and a depthwise separable convolution.

[0027] Furthermore, the improved Shape-Aware NanoDet network is trained using a geometry-sensitive joint loss function, which is: (3); in, For geometry-sensitive joint loss function, For EIoU loss, For key point distance loss, This is due to the loss in aspect ratio consistency. The keypoint distance loss weighting coefficient; This is the aspect ratio consistency loss weighting coefficient.

[0028] (4); in, This represents the area of ​​the intersection between the predicted bounding box and the ground truth bounding box. The area of ​​the region where the predicted bounding box and the ground truth bounding box are combined is defined as the predicted bounding box, which includes the geometric outline bounding box of the target object and the geometric outline bounding box of the obstacle. , , These are the center point coordinate vector, width, and height of the prediction bounding box, respectively; , , These are the center point coordinate vectors of the real bounding box, its width, and its height, respectively. The squared Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box is given. This is the diagonal length of the smallest bounding rectangle between the predicted bounding box and the ground truth bounding box. and These represent the squared difference in width and the squared difference in height between the predicted bounding box and the actual bounding box, respectively. and The width and height of the minimum bounding rectangle between the predicted bounding box and the ground truth bounding box.

[0029] (5); in, For smoothing L1 loss function; This is the set of distance parameters for the prediction box, including the distance from the center point of the prediction box to the four sides; This is a set of distance parameters for the ground truth bounding boxes, including the distances from the center point of the ground truth bounding box to the four sides.

[0030] (6).

[0031] On the other hand, the present invention also provides a cloud-edge-device collaborative intelligent sensing and control method for robotic arms, which is based on a cloud-edge-device collaborative intelligent sensing and control system for robotic arms, and includes the following steps: Step 1: Power on a cloud-edge-device collaborative intelligent sensing and control system for robotic arms. The vision unit collects RGB images of the robotic arm's working area in real time and sends them to the lightweight shape perception algorithm unit.

[0032] Step 2: The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet network to simultaneously detect the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline from the RGB image of the robotic arm's working area as edge perception multidimensional data. This data is then sent to the motion control board via the I2C interface and transmitted to the PC host computer via the serial port.

[0033] Step 3: The PC host computer receives edge-sensing multidimensional data, renders and dynamically updates the target objects, obstacles and robotic arms in the virtual scene in real time based on the edge-sensing multidimensional data, and sends the virtual scene to the monitoring interface for visualization to realize digital twin; the edge-sensing multidimensional data is forwarded to the cloud decision module through the Socket network.

[0034] Step 4: The cloud server receives the edge-aware multidimensional data forwarded by the PC host computer and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking algorithm and path planning algorithm. Based on the coordinates of the robotic arm end effector and the coordinates of the target object and the geometric contour box in the received edge-aware multidimensional data, it generates the grasping posture and plans the collision-free joint trajectory in real time based on the coordinates of the obstacle and the geometric contour box. The grasping posture and the collision-free joint trajectory are converted into control commands.

[0035] Step 5: The intelligent planning algorithm unit sends control commands to the PC host computer in the upper computer monitoring and relay module. The PC host computer receives the control commands and forwards them to the motion control board. It outputs PWM signals to drive each servo joint of the six-axis robotic arm to perform the action of grasping the target object or avoiding obstacles. At the same time, it reads the angle of each servo joint of the six-axis robotic arm and reports it to the PC host computer via the serial port.

[0036] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: 1. Superior edge geometry awareness: This invention deploys the original SA-NanoDet algorithm on the edge K210, especially through... (Aspect Ratio Consistency Loss) enables the model to generate detection boxes that accurately fit the actual geometric contours of objects with extremely low parameter count (<1MB). The aspect ratio error of the generated bounding boxes is ≤4.8%, which is 62% higher than the existing NanoDet model. This directly provides high-quality prior information on geometric dimensions for cloud-based obstacle avoidance and grasping posture planning.

[0037] 2. Multi-target collaborative perception and digital twin closed loop: This invention is the first to realize joint detection and geometric modeling of "end-target-obstacle" at the edge, and maps multi-dimensional geometric information to the host computer's three-dimensional monitoring interface in real time, realizing true virtual-real synchronization and digital twin, which greatly enhances the system's observability and human-computer interaction capabilities.

[0038] 3. High real-time performance and high security cloud-edge collaboration: Lightweight, high-precision geometric perception is retained at the edge, while complex tracking and global obstacle avoidance planning are placed in the cloud. Through an optimized heterogeneous communication link of "K210 (I2C) -> 8051 (serial port) -> PC (Socket) -> cloud", the cloud computing power is fully utilized, and the extremely low latency of edge feedback is guaranteed, enabling the robotic arm to have strong and robust obstacle avoidance and grasping capabilities in dynamic and complex environments. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall architecture of a cloud-edge-device collaborative intelligent sensing and control system for a robotic arm, as described in an embodiment of the present invention.

[0040] Figure 2 This is a rendering of the robotic arm, target object, and obstacles in the virtual-real combined monitoring interface of the host computer in an embodiment of the present invention. Figure 3 This is a schematic diagram of the architecture of the improved Shape-Aware NanoDet (SA-NanoDet) network in an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of the dual-path backbone network structure of the SA-NanoDet algorithm in the edge perception module of this invention. Figure 5 This is a schematic diagram illustrating the working principle of each component in the geometrically sensitive joint loss function in an embodiment of the present invention.

[0042] Among them, (a) is a schematic diagram of the EIoU loss principle; (b) is a schematic diagram of the key point distance loss principle; (c) is a schematic diagram of the aspect ratio consistency loss principle; and (d) is a comparison diagram of loss quantization of geometrically sensitive joint loss function.

[0043] Figure 6 This is a flowchart of a cloud-edge-device collaborative intelligent sensing and control method for a robotic arm, as described in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Example 1: A cloud-edge-device collaborative intelligent sensing and control system for robotic arms, suitable for dynamic operation scenarios, capable of simultaneously and with high precision sensing of the geometric and positional information of the robotic arm's end effector, the target object, and obstacles, and realizing real-time obstacle avoidance and precise grasping based on cloud-based decision-making, is proposed. Figure 1 As shown, it includes an edge perception module, a host computer monitoring and relay module, a cloud decision-making module, and an execution control module.

[0046] The edge perception module is used to perceive three types of targets: the end effector of the robotic arm, the target object, and the obstacle, and to acquire multi-dimensional edge perception data and send it to the execution control module.

[0047] The edge perception module includes a K210 vision unit and a lightweight shape perception algorithm unit.

[0048] The K210 vision unit is used to acquire RGB images of the robotic arm's working area in real time and send them to the lightweight shape perception algorithm unit.

[0049] The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet (SA-NanoDet) network to simultaneously detect and geometrically model three types of targets—the robotic arm end effector, the target object, and the obstacle—from the RGB image of the robotic arm's working area. Finally, it outputs the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline as edge perception multidimensional data, and sends them to the execution control module via the I2C interface.

[0050] The cloud-based decision-making module is used to receive edge-aware multidimensional data forwarded by the host computer monitoring and relay module, generate control commands, and send them to the host computer monitoring and relay module.

[0051] The cloud-based decision-making module includes a cloud server and an intelligent planning algorithm unit. The cloud server communicates with the PC in the host computer monitoring and relay module via a Socket network, receives edge-aware multi-dimensional data forwarded by the PC, and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking and path planning algorithms. It generates a grasping posture based on the coordinates of the robotic arm end and the target object and geometric contour box in the received edge perception multidimensional data. It also plans a collision-free joint trajectory in real time based on the obstacle coordinates and geometric contour box. The grasping posture and collision-free joint trajectory are converted into control commands and sent to the PC host computer in the host computer monitoring and relay module.

[0052] In this embodiment, the cloud-based decision-making module is developed based on C#. It listens for TCP connections on the PC host computer and, upon receiving multi-dimensional edge sensing data uploaded by the edge sensing module (edge ​​end), executes intelligent decision-making: (1) Grasping posture generation: Since the geometric contour box of the target object transmitted by the edge perception module (edge ​​end) contains accurate aspect ratio consistency loss. After constraining the aspect ratio of the contour, the intelligent planning algorithm unit can directly determine the actual geometric size of the target object based on the aspect ratio of the contour, and then calculate the opening degree and approach angle as the grasping posture, avoiding the clamping offset caused by traditional centroid point grasping; the opening degree refers to the degree of opening and closing of the two fingers of the end gripper of the robotic arm, which is converted from the actual length and width of the contour of the target object to obtain the corresponding gripper opening and closing distance, characterizing the opening range required for the gripper to adapt to the shape of the target object.

[0053] (2) Dynamic obstacle avoidance planning: The edge perception module (edge ​​end) expands the geometric outline of the obstacle outward by a safe distance (e.g., 2cm) as the obstacle avoidance equivalent envelope box. Combined with the improved A* algorithm or artificial potential field method, path optimization is performed in the joint space to obtain the collision-free joint trajectory. If the edge perception module (edge ​​end) reports that the position of the obstacle has changed, the cloud decision module can replan the collision-free joint trajectory in real time in the next control cycle (<100ms).

[0054] (3) The cloud decision module converts the grasping posture and collision-free joint trajectory into control commands (six-axis angle and speed commands) and sends them to the PC host computer.

[0055] The host computer monitoring and relay module is used to receive edge-aware multidimensional data forwarded by the execution control module and control commands output by the cloud decision module, and forward the control commands to the execution control module and the edge-aware multidimensional data to the cloud decision module; construct and dynamically update the virtual scene based on the edge-aware multidimensional data, receive the angles of each servo joint of the robotic arm fed back by the execution control module, and obtain the actual motion trajectory of the robotic arm based on the fed-back angles of each servo joint of the robotic arm, thereby realizing the visualization of the virtual scene and the actual motion trajectory of the robotic arm, and completing the closed loop.

[0056] The host computer monitoring and relay module includes a PC host computer and a monitoring interface.

[0057] The PC host computer communicates bidirectionally with the execution control module via a serial port and with the cloud decision module via a Socket network. It receives edge-aware multidimensional data and forwards it to the cloud decision module, and receives control commands and forwards them to the execution control module. Based on the edge-aware multidimensional data, it renders and dynamically updates the target objects, obstacles, and robotic arm in the virtual scene in real time and sends the virtual scene to the monitoring interface. Figure 2 As shown, the actual motion trajectory of the robotic arm is obtained by receiving the angles of each servo joint of the robotic arm from the execution control module and synchronously sent to the monitoring interface.

[0058] The monitoring interface is used to display targets, obstacles, and robotic arms in the virtual scene, while also displaying the actual movement trajectory of the robotic arm's end effector, thus achieving digital twin.

[0059] In this example, the PC host computer is developed based on Python 3.9 and uses the pyserial library to conduct full-duplex serial communication with the 8051 control board at 9600bps. The PC host computer parses the coordinates of the robotic arm's end effector. The coordinates of the target object and the aspect ratio of its geometric outline. Obstacle coordinates and aspect ratio of their geometric outlines ,in The coordinates of the geometric outline of the target object. The length of the geometric outline of the target object. The width of the geometric outline of the target object; Let be the coordinates of the geometric outline of the obstacle. The length of the geometric outline of the obstacle. Define the width of the geometric outline of the obstacle; then perform two parallel tasks: (1) Data uplink: The above edge-aware multidimensional data is encapsulated into JSON format through the Socket network and reported to the cloud server.

[0060] (2) Real-Virtual Synchronization: In the 3D monitoring scene built by PyOpenGL, the length and width of the robotic arm, target object and obstacle are dynamically generated to form a cube or cylinder with the same scaling ratio to wrap the model, so as to realistically restore the size relationship of the physical world and draw the actual movement trajectory of the robotic arm in the interface.

[0061] The execution control module is used to receive edge-sensing multi-dimensional data and forward it to the host computer monitoring and relay module; receive control commands output by the host computer monitoring and relay module and execute the action of grasping the target object or avoiding obstacles, and feed back the angle of each servo joint of the robotic arm to the host computer monitoring and relay module.

[0062] The execution control module includes an 8051 motion control board and a six-axis robotic arm.

[0063] The 8051 motion control board receives multi-dimensional data from edge sensing and forwards it to the host computer monitoring and relay module; it receives... Control commands from the PC host computer are parsed and output as PWM signals to drive the servo joints of the six-axis robotic arm to perform actions such as grasping targets or avoiding obstacles. At the same time, the angles of each servo joint of the six-axis robotic arm are read and reported to the PC host computer via serial port, completing the entire cloud-edge-device collaborative closed loop.

[0064] like Figure 3 As shown, the improved Shape-Aware NanoDet (SA-NanoDet) network improves upon the original standard lightweight backbone network ShuffleNetV2 by introducing a dual-path structure in its Stage 3 and Stage 4 stages. Specifically, Stage 3 and Stage 4 are replaced with a dual-path feature extraction backbone network. The original structures of Stage 3 and Stage 4 consist of ordinary blocks and downsampling blocks. The dual-path feature extraction backbone network embeds the dual-path structure into each block without changing the channel variation rules of the original blocks. In other words, the ordinary blocks and downsampling blocks of Stage 3 and Stage 4 are replaced with dual-path ordinary blocks and dual-path downsampling blocks. At the same time, an adaptively weighted feature pyramid network (FPN) is added after Stage 4.

[0065] like Figure 4 As shown, the dual-path ordinary block first splits the input feature map into two split feature maps by a 1:1 channel split. Dual-path feature extraction is then performed on one of these split feature maps, including a first semantic path (main path) and a first geometric path (auxiliary path). The first semantic path uses a first depthwise separable convolutional group to extract rich semantic context information, ensuring classification accuracy. The first geometric path uses a lightweight edge enhancement module (LEEM) to focus on extracting edge gradient features in the horizontal and vertical directions, strengthening the network's perception of object contours and shapes. Finally, the feature maps output from the semantic and geometric paths are concatenated to form a first fusion feature that contains both semantic information and rich geometric details. This first fusion feature is then integrated with the other split feature map by channel dimension to obtain the output feature map.

[0066] The first depthwise separable convolutional group includes convolution and depthwise separable convolution in sequence; the lightweight edge enhancement module (LEEM) first performs asymmetric convolution in parallel on the input feature map, with convolution kernels of 1×3 and 3×1 respectively, and then concatenates the output features of the two asymmetric convolutions through channels, and completes channel integration by using pointwise fusion convolution.

[0067] The process of concatenating the feature maps output from the semantic path and the geometric path is represented as follows: (1); in, The original input features are the semantic path and the geometric path. The output fusion features, This represents a channel splicing operation. For depthwise separable convolution, It is an asymmetric convolution.

[0068] The dual-path downsampling block includes a dual-path downsampling branch and a dual-path downsampling and convolution branch. In the dual-path downsampling branch, the input feature map is processed through the second semantic path and the second geometric path for feature extraction and channel concatenation to obtain the second fused feature. In the dual-path downsampling and convolution branch, the input feature map is processed through the third semantic path and the third geometric path for feature extraction and channel concatenation to obtain the third fused feature. Finally, the second fused feature and the third fused feature are concatenated to obtain the output feature map.

[0069] The second semantic path sequentially includes a depthwise separable convolution and a convolution; the third semantic path sequentially includes a convolution, a depthwise separable convolution and a convolution; both the second and third geometric paths sequentially include a lightweight edge enhancement module (LEEM) and a depthwise separable convolution.

[0070] The Adaptively Weighted Feature Pyramid Network (FPN) employs an improved PANet structure in its feature fusion part, but introduces an adaptive weighting mechanism. For the output feature maps of Stage 3 and Stage 4 (input features from different levels), a weight vector is automatically learned. And perform a weighted summation of the output feature maps of Stage 3 and Stage 4: (2); in, For the level number, Output feature map for Stage 3 or Stage 4; The output feature map of the adaptive weighted feature pyramid network; Let be the weight vector, where =1, which allows the adaptive weighted feature pyramid network to dynamically assign higher weights to feature scales that are more important for judging the shape of the current target; These are upsampling or downsampling operators used to unify the size of feature maps at different levels, ensuring that multi-scale features can be weighted and fused.

[0071] The improved Shape-Aware NanoDet network is trained using a geometry-sensitive joint loss function, which is: (3); in, For geometry-sensitive joint loss function, For EIoU loss, For key point distance loss, This is due to the loss in aspect ratio consistency. The keypoint distance loss weighting coefficient is used to balance the proportion of different loss terms, as described in this embodiment of the invention. =0.5; The aspect ratio consistency loss weight coefficient is a core parameter controlling the shape learning intensity, as described in this embodiment of the invention. The value is 0.8. The working principle of each component in the geometrically sensitive joint loss function is as follows: Figure 5 As shown. Among them. This loss function is used to constrain the overall overlap and center offset of the detection boxes, maximizing the overlap area between the predicted and ground truth boxes and reducing their overlap deviation; it also constrains the center point of the predicted box to converge towards the center point of the ground truth box, eliminating center offset errors; and it initially corrects the aspect ratio of the predicted box, alleviating the problem of large-scale shape distortion. This loss function is responsible for global coarse correction, solving the basic localization problem of overall misalignment and large-scale mismatch of the detection boxes. This loss function is used to finely correct the distances between the four corner points of the contour and the boundary, aligning the coordinates of the four corner points of the predicted and ground truth bounding boxes one by one, and correcting corner offset errors; it constrains the horizontal and vertical distances from the four sides of the predicted bounding box to the target center, matching the boundary scale; and it repairs local contour detail distortion, outputting an optimized predicted bounding box with higher contour fit. This loss function focuses on contour details, compensating for... The loss only focuses on the overall overlap and ignores the local deviations at corner points. It is used to unify the aspect ratio of irregular objects and correct shape distortion. It forces the predicted bounding box to maintain the same aspect ratio as the ground truth bounding box, thereby suppressing the stretching / compression distortion of the bounding box from the root. It corrects the contour shape distortion of irregular objects, adapts to the detection of irregular and non-rectangular targets, and improves the network's ability to perceive the geometric shape of objects.

[0072] Using the aforementioned EIoU loss, and considering overlap area, center point distance, and aspect ratio differences, basic localization supervision is provided: (4); in, This represents the area of ​​the intersection between the predicted bounding box and the ground truth bounding box. The area of ​​the region where the predicted bounding box and the ground truth bounding box are combined is defined as the predicted bounding box, which includes the geometric outline bounding box of the target object and the geometric outline bounding box of the obstacle. , , These are the center point coordinate vector, width, and height of the prediction bounding box, respectively; , , These are the center point coordinate vectors, width, and height of the actual bounding box (label box), respectively. The squared Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box is given. This is the diagonal length of the smallest bounding rectangle between the predicted bounding box and the ground truth bounding box, used to normalize the center point distance. and These represent the squared difference in width and the squared difference in height between the predicted bounding box and the actual bounding box, respectively. and The width and height of the minimum bounding rectangle between the predicted bounding box and the ground truth bounding box.

[0073] The keypoint distance loss is: (5); in, The smoothing L1 loss function is a commonly used regression loss function. This is the set of distance parameters for the prediction box, including the distance from the center point of the prediction box to the four sides; This is the set of distance parameters for the ground truth bounding boxes, including the distances from the center point of the ground truth bounding box to its four sides. In addition to the four corner points of the predicted bounding box, it further supervises the difference between the distances from the center point of the predicted bounding box to its four sides and the ground truth bounding box, which can more finely constrain the shape of the box. The aspect ratio consistency loss is the core of this invention. This loss function forces the aspect ratio of the predicted box to approximate that of the true box, and it is defined as the absolute difference in logarithmic space: (6); By introducing aspect ratio consistency loss The improved Shape-Aware NanoDet network is explicitly guided to learn the shape prior of the object, and the generated bounding box can better fit the actual geometric contour of the object.

[0074] In this embodiment, the SA-NanoDet model is trained using PyTorch on a K210 vision unit. The dataset includes the robotic arm end effector, various target objects, and common obstacles, with their bounding boxes and precise aspect ratios annotated. During training, the input resolution is set to 320×240 (adapted to K210 resolution), and the AdamW optimizer is selected.

[0075] After training, the model was quantized using NNCM (using quantization-aware training to compensate for accuracy in the geometric path feature layer) with INT8 quantization, and then converted to the .kmodel format supported by K210 using kme_converter. In actual testing on K210, the quantized model size was approximately 950KB, and the inference time was approximately 85ms (≥11FPS), perfectly adapting to the memory and computing power constraints of K210. K210 not only outputs the coordinates of the target center point but also outputs high-precision contour aspect ratios. The compressed multi-target pose and geometry data packets are sent to the 8051 control board in real time via the I2C interface (100kHz).

[0076] Example 2: A cloud-edge-device collaborative intelligent sensing and control method for robotic arms is implemented based on a cloud-edge-device collaborative intelligent sensing and control system for robotic arms, such as... Figure 6 As shown, it includes the following steps: Step 1: Power on a cloud-edge-device collaborative intelligent sensing and control system for robotic arms. The K210 vision unit collects RGB images of the robotic arm's working area in real time and sends them to the lightweight shape perception algorithm unit.

[0077] Step 2: The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet network to simultaneously detect the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline from the RGB image of the robotic arm's working area as edge perception multidimensional data. This data is then sent to the 8051 motion control board via the I2C interface and transmitted to the PC host computer via the serial port.

[0078] Step 3: The PC host computer receives edge-sensing multidimensional data, renders and dynamically updates the target objects, obstacles and robotic arms in the virtual scene in real time based on the edge-sensing multidimensional data, and sends the virtual scene to the monitoring interface for visualization to realize digital twin; the edge-sensing multidimensional data is forwarded to the cloud decision module through the Socket network.

[0079] Step 4: The cloud server receives the edge-aware multidimensional data forwarded by the PC host computer and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking algorithm and path planning algorithm. Based on the coordinates of the robotic arm end effector and the coordinates of the target object and the geometric contour box in the received edge-aware multidimensional data, it generates the grasping posture and plans the collision-free joint trajectory in real time based on the coordinates of the obstacle and the geometric contour box. The grasping posture and the collision-free joint trajectory are converted into control commands.

[0080] Step 5: The intelligent planning algorithm unit sends control commands to the PC host computer in the upper computer monitoring and relay module. The PC host computer receives the control commands and forwards them to the 8051 motion control board, outputting PWM signals to drive each servo joint of the six-axis robotic arm to perform the action of grasping the target object or avoiding obstacles. At the same time, it reads the angle of each servo joint of the six-axis robotic arm and reports it to the PC host computer via the serial port, completing the entire cloud-edge-device collaborative closed loop.

[0081] During execution, the K210 vision unit continuously provides feedback on the actual pose of the end effector and the real-time position of obstacles. If an obstacle intrudes into the planned path, the cloud immediately replans and corrects the trajectory, forming a dynamic closed-loop control of "edge perception - cloud decision-making - end-side execution - edge feedback".

[0082] Although the functions and working processes of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific functions and working processes described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these are within the protection scope of the present invention.

Claims

1. A cloud-edge-device collaborative intelligent sensing and control system for robotic arms, characterized in that, It includes an edge sensing module, a host computer monitoring and relay module, a cloud decision-making module, and an execution control module; The edge perception module is used to perceive three types of targets: the end effector of the robotic arm, the target object, and the obstacle, and to acquire multi-dimensional edge perception data and send it to the execution control module. The cloud-based decision-making module is used to receive edge-aware multi-dimensional data forwarded by the host computer monitoring and relay module, generate control commands, and send them to the host computer monitoring and relay module. The host computer monitoring and relay module is used to receive edge-aware multidimensional data forwarded by the execution control module and control commands output by the cloud decision module, and forward the control commands to the execution control module and the edge-aware multidimensional data to the cloud decision module; construct and dynamically update the virtual scene based on the edge-aware multidimensional data, receive the angles of each servo joint of the robotic arm fed back by the execution control module, and obtain the actual motion trajectory of the robotic arm based on the fed-back angles of each servo joint of the robotic arm, thereby realizing the visualization of the virtual scene and the actual motion trajectory of the robotic arm; The execution control module is used to receive edge-sensing multi-dimensional data and forward it to the host computer monitoring and relay module; receive control commands output by the host computer monitoring and relay module and execute the action of grasping the target object or avoiding obstacles, and feed back the angle of each servo joint of the robotic arm to the host computer monitoring and relay module.

2. The cloud-edge-device collaborative intelligent sensing and control system for robotic arms according to claim 1, characterized in that, The edge perception module includes a vision unit and a lightweight shape perception algorithm unit; The vision unit is used to acquire RGB images of the robotic arm's working area in real time and send them to the lightweight shape perception algorithm unit. The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet network to simultaneously detect and geometrically model three types of targets—the robotic arm end effector, the target object, and the obstacle—from the RGB image of the robotic arm's working area. Finally, it outputs the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline as edge perception multidimensional data, and sends them to the execution control module via the I2C interface.

3. The cloud-edge-device collaborative intelligent sensing and control system for robotic arms according to claim 1, characterized in that, The cloud-based decision-making module includes a cloud server and an intelligent planning algorithm unit; The cloud server communicates with the PC host computer in the host computer monitoring and relay module through the Socket network, receives the edge perception multidimensional data forwarded by the PC host computer and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking and path planning algorithms. It generates a grasping posture based on the coordinates of the robotic arm end and the target object and geometric contour box in the received edge perception multidimensional data. It also plans a collision-free joint trajectory in real time based on the obstacle coordinates and geometric contour box. The grasping posture and collision-free joint trajectory are converted into control commands and sent to the PC host computer in the host computer monitoring and relay module.

4. The cloud-edge-device collaborative intelligent sensing and control system for robotic arms according to claim 1, characterized in that, The host computer monitoring and relay module includes a PC host computer and a monitoring interface; The PC host computer communicates bidirectionally with the execution control module via a serial port and bidirectionally with the cloud decision module via a Socket network. It is used to receive edge-sensing multidimensional data and forward it to the cloud decision module, receive control commands and forward them to the execution control module; render and dynamically update the target objects, obstacles and robotic arms in the virtual scene in real time based on the edge-sensing multidimensional data, and send the virtual scene to the monitoring interface; receive the angles of each servo joint of the robotic arm fed back by the execution control module to obtain the actual motion trajectory of the robotic arm end effector and send it synchronously to the monitoring interface. The monitoring interface is used to display targets, obstacles, and robotic arms in the virtual scene, while also displaying the actual movement trajectory of the robotic arm's end effector, thus achieving digital twin.

5. The cloud-edge-device collaborative intelligent sensing and control system for robotic arms according to claim 1, characterized in that, The execution control module includes a motion control board and a six-axis robotic arm; The motion control board receives multi-dimensional data from edge sensing and forwards it to the host computer monitoring and relay module; it receives... Control commands from the PC host computer are parsed and output as PWM signals to drive the servo joints of the six-axis robotic arm to perform actions such as grasping targets or avoiding obstacles; at the same time, the angles of each servo joint are read and reported to the PC host computer via serial port, completing the entire cloud-edge-device collaborative closed loop.

6. The cloud-edge-device collaborative intelligent sensing and control system for a robotic arm according to claim 2, characterized in that, The improved Shape-Aware NanoDet network improves upon the original standard lightweight backbone network ShuffleNetV2 by introducing a dual-path structure in its Stage 3 and Stage 4 stages. This means that Stage 3 and Stage 4 are replaced with a dual-path feature extraction backbone network. The original structure of Stage 3 and Stage 4 consists of ordinary blocks and downsampling blocks. The dual-path feature extraction backbone network embeds the dual-path structure into each block, that is, it replaces the ordinary blocks and downsampling blocks of Stage 3 and Stage 4 with dual-path ordinary blocks and dual-path downsampling blocks. At the same time, an adaptive weighted feature pyramid network is added after Stage 4. The dual-path ordinary block first splits the input feature map into two channels at a 1:1 ratio, resulting in two split feature maps. Dual-path feature extraction is then performed on one of the split feature maps, including a first semantic path and a first geometric path. The first semantic path is a first depthwise separable convolutional group, and the first geometric path is a lightweight edge enhancement module. Finally, the feature maps output from the semantic path and the geometric path are concatenated by channels to form a first fused feature. The first fused feature is then integrated with the other split feature map by channel dimension to obtain the output feature map. The dual-path downsampling block includes a dual-path downsampling branch and a dual-path downsampling and convolution branch. In the dual-path downsampling branch, the input feature map is extracted through the second semantic path and the second geometric path, and then the channels are concatenated to obtain the second fused feature. In the dual-path downsampling and convolution branches, the input feature map is processed by the third semantic path and the third geometric path for feature extraction and channel concatenation to obtain the third fusion feature; finally, the second fusion feature and the third fusion feature are concatenated to obtain the output feature map. The adaptive weighted feature pyramid network automatically learns a weight vector for the output feature maps of Stage 3 and Stage 4. And perform a weighted summation of the output feature maps of Stage 3 and Stage 4: （2）； in, For the level number, Output feature map for Stage 3 or Stage 4; The output feature map of the adaptive weighted feature pyramid network; Let be the weight vector, where =1; This refers to either an upsampling operator or a downsampling operator.

7. The cloud-edge-device collaborative intelligent sensing and control system for a robotic arm according to claim 6, characterized in that, The first depthwise separable convolutional group includes convolution and depthwise separable convolution in sequence; the lightweight edge enhancement module first performs asymmetric convolution on the input feature map in parallel, then concatenates the output features of the two asymmetric convolutions by channel, and completes channel integration by pointwise fusion convolution; The second semantic path sequentially includes a depthwise separable convolution and a convolution; the third semantic path sequentially includes a convolution, a depthwise separable convolution, and a convolution; both the second and third geometric paths sequentially include a lightweight edge enhancement module and a depthwise separable convolution.

8. The cloud-edge-device collaborative intelligent sensing and control system for a robotic arm according to claim 2, characterized in that, The improved Shape-Aware NanoDet network is trained using a geometry-sensitive joint loss function, which is: （3）； in, For geometry-sensitive joint loss function, For EIoU loss, For key point distance loss, This is due to the loss in aspect ratio consistency. The keypoint distance loss weighting coefficient; The aspect ratio consistency loss weighting coefficient; （4）； in, This represents the area of ​​the intersection between the predicted bounding box and the ground truth bounding box. The area of ​​the region where the predicted bounding box and the ground truth bounding box are combined is defined as the predicted bounding box, which includes the geometric outline bounding box of the target object and the geometric outline bounding box of the obstacle. , , These are the center point coordinate vector, width, and height of the prediction bounding box, respectively; , , These are the center point coordinate vectors of the real bounding box, its width, and its height, respectively. The squared Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box is given. This is the diagonal length of the smallest bounding rectangle between the predicted bounding box and the ground truth bounding box. and These represent the squared difference in width and the squared difference in height between the predicted bounding box and the actual bounding box, respectively. and The width and height of the minimum bounding rectangle between the predicted bounding box and the ground truth bounding box; （5）； in, For smoothing L1 loss function; This is the set of distance parameters for the prediction box, including the distance from the center point of the prediction box to the four sides; This is a set of ground truth box distance parameters, including the distances from the center point of the ground truth box to its four sides; （6）。 9. A cloud-edge-device collaborative intelligent sensing and control method for a robotic arm, implemented based on the cloud-edge-device collaborative intelligent sensing and control system for a robotic arm as described in any one of claims 1-8, characterized in that, Includes the following steps: Step 1: Power on a cloud-edge-device collaborative intelligent sensing and control system for robotic arms. The vision unit collects RGB images of the robotic arm's working area in real time and sends them to the lightweight shape perception algorithm unit. Step 2: The lightweight shape perception algorithm unit uses an improved Shape-Aware NanoDet network to simultaneously detect the coordinates of the robotic arm end effector, the coordinates of the target object and its geometric outline, and the coordinates of the obstacle and its geometric outline from the RGB image of the robotic arm's working area as edge perception multidimensional data. This data is then sent to the motion control board via the I2C interface and transmitted to the PC host computer via the serial port. Step 3: The PC host computer receives edge-sensing multi-dimensional data, renders and dynamically updates the target objects, obstacles, and robotic arm in the virtual scene in real time based on the edge-sensing multi-dimensional data, and sends the virtual scene to the monitoring interface for visualization to realize digital twin; the edge-sensing multi-dimensional data is forwarded to the cloud decision module through the Socket network; Step 4: The cloud server receives the edge-aware multidimensional data forwarded by the PC host computer and forwards it to the intelligent planning algorithm unit. The intelligent planning algorithm unit has built-in target tracking algorithm and path planning algorithm. Based on the coordinates of the robotic arm end and the coordinates of the target object and the geometric contour box in the received edge-aware multidimensional data, it generates the grasping posture and plans the collision-free joint trajectory in real time based on the coordinates of the obstacle and the geometric contour box. The grasping posture and the collision-free joint trajectory are converted into control commands. Step 5: The intelligent planning algorithm unit sends control commands to the PC host computer in the upper computer monitoring and relay module. The PC host computer receives the control commands and forwards them to the motion control board. It outputs PWM signals to drive each servo joint of the six-axis robotic arm to perform the action of grasping the target object or avoiding obstacles. At the same time, it reads the angle of each servo joint of the six-axis robotic arm and reports it to the PC host computer via the serial port.

Images