Remote mechanical arm control system and method based on multimode communication

Abstract

The invention provides a remote mechanical arm control system and method based on multi-mode communication, the system comprises a user side and an equipment side, the user side and the equipment side establish communication connection through a multi-mode communication system, and the system comprises a main channel, an emergency channel and a cloud platform; the equipment end comprises an image acquisition device, a mechanical arm and a controller; the user side sends a control instruction to the controller through the main channel, and the controller is used for controlling the image acquisition device to acquire a field image according to the control instruction, performing mechanical arm control planning based on the field image, generating a mechanical arm control signal to control the mechanical arm to execute operation, acquiring field state information and sending the field state information to the user side through the main channel; the cloud platform allocates communication resources of the main channel; the controller monitors the communication state of the main channel, switches to the emergency channel when the communication state is abnormal, and synchronizes the final effective control instruction snapshot of the main channel to the emergency channel; according to the scheme, the reliability of remote operation of the mechanical arm can be guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote control, in particular to a remote mechanical arm control system and method based on multi-mode communication. BACKGROUND

[0002] With the development of science and technology, the demand for automation and intelligent equipment in power systems is increasing. In the existing technology, the positioning of the spacer clamp and the cable is mainly controlled by mechanical adjustment. For example, patent document CN120127544A provides a spacer installation platform installation method of a four-split conductor spacer installation robot, which includes a platform support plate, a hollow rotating platform support plate, a hollow rotating platform, a guide shaft, an elevator, an electromagnet and a telescopic ejector pin. The platform support plate is arranged above the hollow rotating platform support plate and connected by bolts. The hollow rotating platform passes through the center of the hollow rotating platform support plate and is connected to the lower surface of the platform support plate by bolts. The guide shaft is arranged on the lower surface of the hollow rotating platform support plate. The elevator is located at the bottom and its upper end is connected to the lower end of the hollow rotating platform through the elevator lead screw. The electromagnet is arranged on the upper surface of the platform support plate. The telescopic ejector pin is arranged on the upper surface of the platform support plate. However, the installation scene of the spacer may be located in high altitude and mountainous areas, which is not convenient for on-site control. SUMMARY

[0003] The present application provides a remote mechanical arm control system and method based on multi-mode communication, which can improve the reliability and accuracy of remote control of the mechanical arm.

[0004] A remote mechanical arm control system based on multi-mode communication, comprising a user end and a device end, a communication connection is established between the user end and the device end through a multi-mode communication system, the multi-mode communication system comprises a main channel, an emergency channel and a cloud platform; The device end comprises an image acquisition device, a mechanical arm and a controller; The user end is used to send control instructions to the controller through the main channel, the controller is used to control the image acquisition device to acquire the field image according to the control instructions, to plan the mechanical arm control based on the field image, to generate the mechanical arm control signal to control the mechanical arm to perform the operation, and to acquire the field state information and send it to the user end through the main channel; The cloud platform is used to allocate the communication resources of the main channel; The controller is also used to monitor the communication state of the main channel, switch to the emergency channel when the communication state is abnormal, and synchronize the last valid control instruction snapshot of the main channel to the emergency channel.

[0005] Further, the main channel is in wireless communication mode, and the emergency channel is in wired communication mode.

[0006] Further, the field image comprises an RGB image and a depth image; The controller is further configured to: establish a mapping relationship between the RGB image and the depth image, construct a field search space, and obtain a camera intrinsic matrix and distortion coefficients; perform target recognition according to the RGB image, determine an obstacle and a task area, and map the task area to a three-dimensional space according to the mapping relationship to obtain a task area space; generate a task path in the task area space according to a task execution sequence, and determine a task starting point three-dimensional coordinate; obtain an obstacle two-dimensional coordinate, map the obstacle two-dimensional coordinate to a three-dimensional space according to the mapping relationship to obtain an obstacle three-dimensional coordinate; perform path planning according to the starting point three-dimensional coordinate of the robot arm, the obstacle three-dimensional coordinate, the task starting point three-dimensional coordinate, and the field search space to obtain a walking path, integrate the walking path with the task path to obtain an overall planning path; select a target point in the task path; calculate a robot arm pose sequence and a joint angle sequence in the task path according to the camera intrinsic matrix, the robot arm base coordinate, and the target point; generate the robot arm control signal according to the overall planning path, the pose sequence, and the joint angle sequence to control the robot arm to perform an operation.

[0007] Further, the controller is further configured to: initialize a search tree, the search tree taking the starting point three-dimensional coordinate of the robot arm as a root node; establish an obstacle list according to the obstacle three-dimensional coordinate; iteratively perform the following steps until a stop condition is met: randomly select a random node in the field search space that is not in the obstacle list, search for a nearest neighbor node of the random node in the search tree, expand a preset step length from the neighbor node to the random node to obtain a new node; select nodes in a neighborhood range of the new node based on the search tree to obtain a neighborhood node set; traverse each pending node in the neighborhood node set, calculate a first cost of connecting from the root node to the pending node and then to the new node, select a pending node corresponding to the minimum first cost as a parent node of the new node, and add the new node to the search tree; traversing each pending node in the set of neighborhood nodes, calculating a second cost from the root node to the new node to the pending node, if there is a second cost less than the minimum first cost, taking the new node as the parent node of the corresponding pending node; when the coordinates of the new node coincide with the three-dimensional coordinates of the task starting point, or the distance between the coordinates of the new node and the three-dimensional coordinates of the task starting point is less than a preset distance, stopping the current iteration, backtracking from the three-dimensional coordinates of the task starting point to the three-dimensional coordinates of the starting point of the robot arm, and recording the currently obtained parent node as a candidate planning path; when the stop condition is met, selecting the shortest path from the candidate planning path as the final walking path.

[0008] Further, the controller is further configured to: selecting a plurality of non-coplanar feature points in a preset neighborhood range of the target point, obtaining two-dimensional coordinates of the feature points in the RGB image according to the mapping relationship, and combining the two-dimensional coordinates of the feature points and the three-dimensional coordinates to form a coordinate point pair; calculating a rotation vector and a translation vector under the corresponding target point based on the PnP algorithm, the camera intrinsic matrix and the distortion coefficient, the rotation vector and the translation vector forming the pose of the corresponding target point; interpolating based on the pose of each target point to obtain a pose sequence; solving the corresponding joint angle by using the Jacobian pseudo-inverse algorithm according to the pose sequence to obtain the joint angle sequence.

[0009] Further, the cloud platform is configured to: constructing a first level queue, a second level queue, a third level queue and a fourth level queue, the first level queue adopting a pre-allocated fixed time slot mechanism and a forced preemption mechanism, the second level queue adopting a dynamic bandwidth priority strategy, the third level queue adopting a dynamic bandwidth sub-priority strategy and a time slice round robin mechanism, and the fourth level queue adopting an idle trigger mechanism; according to the task level of the transmission data between the user end and the device end, using the first level queue, the second level queue, the third level queue and the fourth level queue for task scheduling.

[0010] Further, the transmission data between the user end and the device end includes the control instruction, the field state information, the emergency instruction, the backup data and the software update data; the field state information includes field video stream data and sensing data; the control instruction and the emergency instruction are based on the pre-allocated fixed time slot mechanism of the first level queue to exclusively occupy the pre-allocated time slot, and interrupt the tasks of the second level queue, the third level queue and the fourth level queue; The live video stream data is allocated bandwidth based on a dynamic bandwidth priority strategy of the second level queue; The sensing data occupies the remaining bandwidth based on a dynamic bandwidth sub-priority strategy of the third level queue, and is sent in rotation with the second level queue according to a preset time slice length; The backup data and software update data are allocated a second preset percentage of the remaining bandwidth for transmission based on an idle trigger mechanism of the fourth level queue when the load of the main channel is lower than a first preset percentage and lasts for more than a preset time length.

[0011] Further, the controller is further configured to: According to a preset frequency, collect the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate of the main channel; According to the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate, determine whether a switching condition is met, and switch to the emergency channel when the switching condition is met.

[0012] Further, the controller is further configured to determine whether a recovery condition is met according to the signal strength and packet loss rate of the main channel, and switch to the main channel and synchronize the last valid control instruction snapshot of the emergency channel to the main channel if the recovery condition is met.

[0013] A remote mechanical arm control method based on multi-mode communication applied to the above system, comprising: The user end sends control instructions to the controller through the main channel; The controller controls the image acquisition device to acquire live images according to the control instructions, plans mechanical arm control based on the live images, generates mechanical arm control signals to control the mechanical arm to perform operations, and sends live state information to the user end through the main channel; The cloud platform allocates communication resources of the main channel; The controller monitors the communication state of the main channel, switches to the emergency channel when the communication state is abnormal, and synchronizes the last valid control instruction snapshot of the main channel to the emergency channel.

[0014] Further, the live images include RGB images and depth images; Planning mechanical arm control based on the live images, generating mechanical arm control signals to control the mechanical arm to perform operations, comprises: Establishing a mapping relationship between the RGB images and the depth images, constructing a live search space, and obtaining a camera intrinsic matrix and distortion coefficients; Target recognition is performed based on the RGB image to determine obstacles and task areas, and the task area is mapped to three-dimensional space according to the mapping relationship to obtain the task area space; Based on the task execution order, a task path is generated in the task area space, and the three-dimensional coordinates of the task starting point are determined. Obtain the two-dimensional coordinates of the obstacle, and map the two-dimensional coordinates of the obstacle to three-dimensional space according to the mapping relationship to obtain the three-dimensional coordinates of the obstacle; Based on the three-dimensional coordinates of the starting point of the robotic arm, the three-dimensional coordinates of the obstacle, the three-dimensional coordinates of the task starting point, and the on-site search space, a path is planned to obtain a walking path. The walking path is then integrated with the task path to obtain the overall planned path. Select a target point in the task path; Based on the camera intrinsic parameter matrix, the base coordinates of the robotic arm, and the target point, calculate the pose sequence and joint angle sequence of the robotic arm in the task path; The robot arm control signals are generated based on the overall planning path, pose sequence, and joint angle sequence to control the robot arm to perform operations.

[0015] Furthermore, based on the three-dimensional coordinates of the robotic arm's starting point, the three-dimensional coordinates of the obstacles, the three-dimensional coordinates of the task starting point, and the on-site search space, path planning is performed to obtain the walking path, including: Initialize the search tree, with the three-dimensional coordinates of the starting point of the robotic arm as the root node; Create an obstacle list based on the three-dimensional coordinates of the obstacles; Iteratively execute the following steps until the stopping condition is met: Randomly select a node that is not in the obstacle list within the on-site search space, search for the nearest neighbor node to the random node in the search tree, and expand the direction of the neighbor node toward the random node by a preset step size to obtain a new node; Based on the search tree, select nodes within the neighborhood range of the new node to obtain a set of neighborhood nodes; Traverse each undetermined node in the neighborhood node set, calculate the first cost of connecting from the root node to the undetermined node and then to the new node, select the undetermined node with the minimum first cost as the parent node of the new node, and add the new node to the search tree; Traverse each undetermined node in the neighborhood node set, calculate the second cost from the root node to the new node and then to the undetermined node. If there exists a second cost less than the minimum first cost, then the new node is taken as the parent node of the corresponding undetermined node. When the coordinates of the new node coincide with the three-dimensional coordinates of the task starting point, or the distance between the coordinates of the new node and the three-dimensional coordinates of the task starting point is less than a preset distance, stopping the current iteration, backtracking from the three-dimensional coordinates of the task starting point to the three-dimensional coordinates of the starting point of the mechanical arm, and recording the currently obtained parent node as a candidate planning path; When the stop condition is met, selecting the shortest path from the candidate planning path as the final walking path.

[0016] Further, based on the camera intrinsic matrix, the base coordinates of the mechanical arm, and the target point, a pose sequence and a joint angle sequence of the mechanical arm in the task path are calculated, including: Selecting a plurality of non-coplanar feature points in a preset neighborhood range of the target point, obtaining two-dimensional coordinates of the feature points in the RGB image according to the mapping relationship, and combining the two-dimensional coordinates and the three-dimensional coordinates of the feature points to form coordinate point pairs; Based on the PnP algorithm, a rotation vector and a translation vector under the corresponding target point are calculated according to the coordinate point pairs, the camera intrinsic matrix, and the distortion coefficients, and the rotation vector and the translation vector form the pose of the corresponding target point; Interpolating based on the poses of each target point to obtain a pose sequence; According to the pose sequence, the corresponding joint angle is solved by using the pseudo-inverse algorithm of Jacobian to obtain the joint angle sequence.

[0017] Further, the cloud platform allocates communication resources of the main channel, including: A first level queue, a second level queue, a third level queue, and a fourth level queue are constructed, the first level queue adopts a pre-allocated fixed time slot mechanism and a forced preemption mechanism, the second level queue adopts a dynamic bandwidth priority strategy, the third level queue adopts a dynamic bandwidth sub-priority strategy and a time slice rotation mechanism, and the fourth level queue adopts an idle trigger mechanism; According to the task level of the transmission data between the user end and the device end, the first level queue, the second level queue, the third level queue, and the fourth level queue are used for task scheduling.

[0018] Further, the transmission data between the user end and the device end includes the control instruction, the field state information, the emergency instruction, the backup data, and the software update data; the field state information includes field video stream data and sensing data; The control instruction and the emergency instruction are based on the pre-allocated fixed time slot mechanism of the first level queue to exclusively occupy the pre-allocated time slot, and interrupt the tasks of the second level queue, the third level queue, and the fourth level queue; The field video stream data is based on the dynamic bandwidth priority strategy of the second level queue to allocate bandwidth; The sensing data occupies the remaining bandwidth based on the third-level queue dynamic bandwidth sub-priority strategy, and is sent in rotation with the second-level queue according to a preset time slice length; The backup data and software update data are based on the idle trigger mechanism of the fourth-level queue, and when the load of the main channel is lower than a first preset percentage and lasts for more than a preset time length, a second preset percentage of the remaining bandwidth is allocated to the backup data and software update data transmission.

[0019] Further, the controller monitors the communication state of the main channel, and switches to the emergency channel when the communication state is abnormal, comprising: The controller collects the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate of the main channel according to a preset frequency; According to the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate, it is judged whether the switching condition is met, and when the switching condition is met, the emergency channel is switched to.

[0020] Further, after switching to the emergency channel, further comprising: The controller judges whether the recovery condition is met according to the signal strength and packet loss rate of the main channel, and if the recovery condition is met, switches to the main channel and synchronizes the last valid control instruction snapshot of the emergency channel to the main channel.

[0021] The remote mechanical arm control system and method based on multi-mode communication provided by the application have at least the following beneficial effects: (1) The communication connection between the user end and the device end is established based on the main channel and the emergency channel, so that the user end can realize remote control of the mechanical arm of the device end, and when the network fluctuates, the channel can be automatically switched to ensure the reliability of the communication, thereby ensuring the reliability of the remote operation of the mechanical arm, and is particularly suitable for positioning of spacer clamps and cables in areas inconvenient for on-site control; (2) The device end combines an image acquisition device and an improved path planning method, so that the mechanical arm realizes full path planning from starting walking to task completion, realizes accurate positioning and control of the mechanical arm, and ensures the reliability of task implementation; (3) The PnP algorithm and the interpolation algorithm are fused to realize control of the mechanical arm pose and joint angle, which is suitable for complex task operation of the mechanical arm; (4) The communication resources of the main channel are allocated through the cloud platform to ensure the real-time performance of important instructions, fully utilize the bandwidth resources, and realize efficient data interaction between the device end and the user end. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1A structural schematic diagram of an embodiment of a remote mechanical arm control system based on multi-mode communication provided by the present application.

[0023] Figure 2 A flow chart of an embodiment of a remote mechanical arm control method based on multi-mode communication provided by the present application.

[0024] Figure 3 A flow chart of an embodiment of mechanical arm control in a remote mechanical arm control method based on multi-mode communication provided by the present application.

[0025] Figure 4 A flow chart of an embodiment of switching channels in a remote mechanical arm control method based on multi-mode communication provided by the present application. DETAILED DESCRIPTION

[0026] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0027] REFERENCE Figure 1 In some embodiments, a remote mechanical arm control system based on multi-mode communication is provided, which includes a user end 1 and a device end 2, and a communication connection is established between the user end 1 and the device end 2 through a multi-mode communication system, wherein the multi-mode communication system includes a main channel 3, an emergency channel 4 and a cloud platform 5. The device end 2 includes an image acquisition device 21, a mechanical arm 22 and a controller 23. The user end 1 is configured to send a control instruction to the controller 23 through the main channel, the controller 23 is configured to control the image acquisition device 21 to acquire a live image according to the control instruction, perform mechanical arm control planning based on the live image, generate a mechanical arm control signal to control the mechanical arm to perform an operation, and acquire live state information and send the live state information to the user end 1 through the main channel 3. The cloud platform 5 is configured to allocate communication resources of the main channel 3. The controller 23 is further configured to monitor a communication state of the main channel 3, switch to the emergency channel 4 when the communication state is abnormal, and synchronize a last valid control instruction snapshot of the main channel 3 to the emergency channel 4.

[0028] Specifically, the main channel 3 is a wireless communication mode, and the emergency channel 4 is a wired communication mode.

[0029] The main channel 3 adopts a 5G CPE (Customer Premises Equipment) device with a dual-SIM card aggregation transmission function, realizes mobile + telecommunication dual-operator redundancy, and monitors signal quality in real time. When the signal of the main card is interrupted or weaker than the threshold, it automatically switches to the backup card to ensure zero interruption of business. The built-in IPSec / L2TP protocol provides end-to-end encryption for remote device monitoring data, preventing man-in-the-middle attacks.

[0030] The emergency channel 4 adopts a 900MHz image and data integrated data transmission CPE, supports video + data bidirectional transparent transmission, and has strong penetration in the 900MHz frequency band, suitable for long-distance communication in complex environments (such as mountainous areas and building groups). The self-defined emergency instruction set design has a fixed starting symbol value 0xA - instruction type - target device ID - recovery strategy - crc16 check code. Instructions such as full emergency stop, single-function emergency stop, and self-recovery.

[0031] In some embodiments, the device end 2 also includes a switch 24, a CPE device 25, and an image and data integrated data transmission CPE device 26. The controller 23 is connected to the switch 24, the switch 24 is connected to the CPE device 25, and the communication connection of the main channel is established based on the CPE device 25. The communication connection of the emergency channel is established based on the image and data integrated data transmission CPE device 26.

[0032] The user end 1 is installed on the ground and communicates with the cloud platform 5 through a wireless network and communicates with the controller 23 through an emergency channel to realize data interaction, such as control instructions sent by the user end. The CPE device (Customer Premises Equipment) is installed on the device end 2 and communicates with the cloud platform 5 through a 5G network and is connected to the controller 23 through a wired network through the switch 24, and is responsible for signal forwarding. The controller 23 and the image acquisition device 21 are both installed on the device end 2, and the controller 23 is connected to the image acquisition device 21 to obtain video streams in real time. The controller 23 is equipped with an NPU (Neural Processing Unit) chip processor and a self-defined training detection model.

[0033] Reference Figure 2 In some embodiments, a multi-mode communication-based remote mechanical arm control method applied to the above system is provided, which includes: S1, the user end sends a control instruction to the controller through the main channel; S2, the controller controls the image acquisition device to acquire the on-site image according to the control instruction, plans the mechanical arm control based on the on-site image, generates a mechanical arm control signal to control the mechanical arm to perform an operation, and sends the on-site state information to the user end through the main channel; S3, the cloud platform allocates communication resources of the main channel; S4, the controller monitors the communication state of the main channel, and switches to the emergency channel when the communication state is abnormal, and synchronizes the last valid control instruction snapshot of the main channel to the emergency channel.

[0034] Specifically, in step S1, under normal circumstances, the user end interacts with the device end through the main channel, including sending control instructions to the device end, receiving the on-site state information returned by the device end, etc.

[0035] Further, in step S2, the on-site images collected by the image collection device include RGB images and depth images.

[0036] Reference Figure 3 , based on the on-site images, mechanical arm control planning is carried out to generate mechanical arm control signals to control the mechanical arm to perform operations, including: S21, the mapping relationship between the RGB image and the depth image is established, the on-site search space is constructed, and the camera intrinsic matrix and the distortion coefficient are obtained; S22, target recognition is carried out according to the RGB image, the obstacles and the task area are determined, and the task area is mapped to the three-dimensional space according to the mapping relationship to obtain the task area space; S23, according to the task execution sequence, a task path is generated in the task area space, and a task starting point three-dimensional coordinate is determined; S24, the two-dimensional coordinates of the obstacles are obtained, and the two-dimensional coordinates of the obstacles are mapped to the three-dimensional space according to the mapping relationship to obtain the three-dimensional coordinates of the obstacles; S25, according to the starting point three-dimensional coordinate of the mechanical arm, the three-dimensional coordinate of the obstacle, the task starting point three-dimensional coordinate and the on-site search space, path planning is carried out to obtain a walking path, and the walking path is integrated with the task path to obtain an overall planning path; S26, a target point is selected in the task path; S27, according to the camera intrinsic matrix, the base coordinate of the mechanical arm and the target point, the pose sequence and the joint angle sequence of the mechanical arm in the task path are calculated; S28, the mechanical arm control signals are generated according to the overall planning path, the pose sequence and the joint angle sequence to control the mechanical arm to perform operations.

[0037] Specifically, in step S21, the mapping relationship between the RGB image and the depth image is established, that is, the two-dimensional coordinates provided by the RGB image are combined with the pixel depth values provided by the depth image through coordinate transformation, and the mapping relationship is as follows: ; (1) Where, P 3Drepresents the three-dimensional coordinates corresponding to the pixel (u, v), u, v are two-dimensional coordinates of the pixel, D （u,v） represents the depth value corresponding to the pixel (u, v), K is the camera intrinsic matrix.

[0038] The camera intrinsic matrix K and the distortion coefficient are obtained by calibration.

[0039] Further, the mapping relationship between the RGB image and the depth image is established, the three-dimensional space of the scene can be constructed, and the search space of the scene is obtained.

[0040] Further, in step S22, the controller uses improved YOLOv5s as a target recognition detection model, performs probability calibration on the output confidence through Platt Scaling, improves the numerical reliability, and uses INT8 quantization deployment to reduce the demand for computing resources.

[0041] The controller performs target recognition according to the RGB image, determines the obstacle and the task area, and the task area is an area in which the robot arm needs to perform a task. In the task area, the robot arm needs to perform a series of consecutive actions. For example, the task can be the installation of a four-split conductor spacer clamp and the positioning of a cable in a power system. After target recognition based on the two-dimensional RGB image, the corresponding task area is obtained, and the mapping relationship can be mapped to the three-dimensional space to obtain the task area space.

[0042] Further, in step S23, a task path is generated in the task area space according to the task execution sequence, that is, the sequence of operation nodes, and a task starting point three-dimensional coordinate is determined.

[0043] Further, in step S24, target recognition is performed based on the two-dimensional RGB image to obtain obstacle two-dimensional coordinates, and the obstacle two-dimensional coordinates are mapped to the three-dimensional space according to the mapping relationship to obtain obstacle three-dimensional coordinates.

[0044] Further, in step S25, path planning is performed according to the starting point three-dimensional coordinate of the robot arm, the obstacle three-dimensional coordinate, the task starting point three-dimensional coordinate, and the search space of the scene to obtain a walking path, including: S251, initializing a search tree, the search tree taking the starting point three-dimensional coordinate of the robot arm as a root node; S252, establishing an obstacle list according to the obstacle three-dimensional coordinate; S253, iteratively performing the following steps until a stop condition is met: randomly selecting a random node in the search space of the scene that is not in the obstacle list, searching for a nearest neighbor node of the random node in the search tree, and expanding a preset step length from the neighbor node to the random node to obtain a new node; Based on the search tree, nodes in the neighborhood range of the new node are selected to obtain a neighborhood node set; Each pending node in the neighborhood node set is traversed, a first cost from the root node connecting to the pending node and then connecting to the new node is calculated, a pending node corresponding to the minimum first cost is selected as the parent node of the new node, and the new node is added to the search tree; Each pending node in the neighborhood node set is traversed, a second cost from the root node to the new node and then to the pending node is calculated, and if there is a second cost smaller than the minimum first cost, the new node is taken as the parent node of the corresponding pending node; When the new node reaches or approaches the task starting point three-dimensional coordinate, the current iteration is stopped, the task starting point three-dimensional coordinate is taken as the starting point three-dimensional coordinate of the robot arm, and the candidate planning path is recorded as the candidate planning path; S254, when the stop condition is met, the shortest path is selected from the candidate planning path as the final walking path.

[0045] Specifically, in step S251, a search tree T is initialized, and the starting point three-dimensional coordinate of the robot arm is taken as the root node of the search tree T, that is, the starting point of the entire walking path. The iteration number and the preset step length and other parameters are set.

[0046] Further, in step S252, an obstacle list is established according to the obstacle three-dimensional coordinates, and the obstacle list contains the center coordinates and the radius of the obstacle.

[0047] Further, in step S253, in each iteration, a random node q_rand not in the obstacle list is randomly selected in the field search space, the nearest neighbor node q_near of the random node q_rand in the search tree T is searched, a preset step length is expanded from the neighbor node q_near to the random node q_rand to obtain a new node q_new, nodes in the neighborhood range (such as a fixed radius or k nearest neighbors) of the new node q_new are selected based on the search tree T to form a neighborhood node set X_near, each pending node X in the neighborhood node set X_near is traversed, a first cost from the root node connecting to the pending node X and then connecting to the new node q_new is calculated, and the minimum first cost is selected. The pending node X corresponding to the minimum first cost is taken as the parent node of the new node q_new, the neighborhood node set X_near is traversed again, a second cost from the root node to the new node q_new and then to the pending node X is calculated, and if there is a second cost smaller than the minimum first cost, the new node q_new is taken as the parent node of the corresponding pending node X, so as to realize the optimization of the search tree T.

[0048] determining whether the new node q_new reaches or approaches the three-dimensional coordinate of the task starting point, if the coordinate of the new node q_new coincides with the three-dimensional coordinate of the task starting point, the new node q_new reaches the three-dimensional coordinate of the task starting point, if the distance between the new node q_new and the three-dimensional coordinate of the task starting point is less than a preset distance, the new node q_new approaches the three-dimensional coordinate of the task starting point, at this time, the current iteration is stopped, and the three-dimensional coordinate of the task starting point is backtracked to the three-dimensional coordinate of the starting point of the robot arm, the currently obtained parent node is recorded as a candidate planning path, and the current iteration is completed.

[0049] Further, in step S254, the stop condition is that the number of iterations reaches a preset number, at this time, a plurality of candidate planning paths are obtained, and the shortest path is selected as the final walking path from the plurality of candidate planning paths.

[0050] Further, the obtained walking path is a path for the robot arm to walk from the starting point to the task area, and the end point of the walking path, i.e., the starting point of the task, is integrated with the starting point of the task path to obtain an overall planning path.

[0051] Further, in step S26, a target point is selected in the task path, and the target point is a task point at which the robot arm needs to perform a task.

[0052] Further, in step S27, a pose sequence and a joint angle sequence of the robot arm in the task path are calculated according to the camera intrinsic matrix, the base coordinates of the robot arm, and the target point, including: S271, a plurality of non-coplanar feature points in a preset neighborhood range of the target point are selected, and two-dimensional coordinates of the feature points are obtained according to the mapping relationship, and the two-dimensional coordinates and three-dimensional coordinates of the feature points are combined to form a coordinate point pair; S272, based on a PnP algorithm (Perspective-n-Point), a rotation vector and a translation vector under the corresponding target point are calculated according to the coordinate point pair, the camera intrinsic matrix, and the distortion coefficient, and the rotation vector and the translation vector form a pose of the corresponding target point; S273, interpolation is performed based on the poses of each target point to obtain a pose sequence; S274, the corresponding joint angle is solved by using a Jacobian pseudo-inverse algorithm according to the pose sequence to obtain the joint angle sequence.

[0053] Specifically, in step S271, a plurality of non-coplanar feature points in a preset neighborhood range of the target point are selected, the feature points are at least four, and three-dimensional coordinates corresponding to the feature points are obtained, two-dimensional coordinates of each feature point are obtained according to the established mapping relationship, and the two-dimensional coordinates and three-dimensional coordinates of the feature points are combined to form a coordinate point pair.

[0054] Further, in step S272, based on the PnP algorithm, the rotation vector and the translation vector under the corresponding target point are calculated according to the coordinate point pair, the camera intrinsic matrix and the distortion coefficient. The specific method is: according to the camera intrinsic matrix, the distortion parameter and the coordinates of the coordinate point pair, a projection equation set is constructed, and the SVD decomposition method is used to solve the projection set to obtain the rotation vector and the translation vector of the target point. The rotation vector and the translation vector constitute the pose of the corresponding target point, and the pose represents the position and attitude of the robot arm.

[0055] Further, in step S273, the poses of the plurality of target points in the task path are generated, and interpolation is performed on the poses to obtain a pose sequence, so that the position of the robot arm smoothly transitions from one target point to another target point.

[0056] Further, in step S274, according to the pose sequence, the joint angle corresponding to each pose is solved based on the pseudo-inverse algorithm of the Jacobian to obtain the joint angle sequence. Assuming that there are N poses in the pose sequence, numbered 0, 1, 2, …, N-1, wherein the pseudo-inverse algorithm of the Jacobian is used to solve the corresponding joint angle, including: setting an initial joint angle; for the i-th pose (i=1, 2, 3, …, N-1) in the pose sequence, taking the i-1-th pose as the initial pose for current calculation, calculating the current pose error, that is, the difference between the i-th pose and the i-1-th pose; deriving the joint angle obtained last time according to the i-th pose (deriving the initial joint angle according to the first pose), to obtain a linear velocity Jacobian matrix; obtaining an angular velocity Jacobian matrix according to the angular velocity of the joint, and combining the linear velocity Jacobian matrix to obtain a Jacobian matrix; calculating a pseudo-inverse matrix of the Jacobian matrix; multiplying the pseudo-inverse matrix of the Jacobian matrix by the current pose error to obtain a joint angle change amount, and adding the joint angle change amount to the joint angle obtained last time to obtain the joint angle corresponding to the current pose; combining the initial joint angle and the N-2 joint angles calculated and obtained to obtain the joint angle sequence.

[0057] Further, in step S28, the robot arm control signal is generated according to the overall planning path, the pose sequence and the joint angle sequence to control the robot arm to perform an operation, so that the robot arm walks to the task area according to the walking path, and at each target point in the task area, the position and the joint angle are controlled to perform a corresponding task.

[0058] Further, in step S3, the cloud platform allocates communication resources of the main channel, including: The first level queue adopts a pre-allocated fixed time slot mechanism and a forced preemption mechanism, the second level queue adopts a dynamic bandwidth priority strategy, the third level queue adopts a dynamic bandwidth sub-priority strategy and a time slice rotation mechanism, and the fourth level queue adopts an idle trigger mechanism. According to the task level of the data transmission between the user end and the device end, the first level queue, the second level queue, the third level queue and the fourth level queue are used for task scheduling.

[0059] Further, the data transmission between the user end and the device end includes the control instruction, the field state information, the emergency instruction, the backup data and the software update data; the field state information includes field video stream data and sensing data. The control instruction and the emergency instruction exclusively occupy the pre-allocated time slot based on the pre-allocated fixed time slot mechanism of the first level queue, and interrupt the tasks of the second level queue, the third level queue and the fourth level queue. The field video stream data is allocated bandwidth based on the dynamic bandwidth priority strategy of the second level queue. The sensing data occupies the remaining bandwidth based on the dynamic bandwidth sub-priority strategy of the third level queue, and is sent in rotation according to a preset time slice length. The backup data and the software update data are allocated a second preset percentage of the remaining bandwidth for transmission based on the idle trigger mechanism of the fourth level queue when the load of the main channel is lower than a first preset percentage and lasts for more than a preset length of time.

[0060] Specifically, when the user end and the device end interact with each other, the control instruction and the emergency instruction have the highest priority and exclusively occupy the pre-allocated time slot, i.e. a time period that is periodically or statically allocated to the control instruction and the emergency instruction and cannot be occupied by other tasks, and resource allocation is completed before system initialization or task start, rather than being dynamically applied. When there is a task in the first level queue, the tasks in the second level queue, the third level queue and the fourth level queue can be interrupted.

[0061] Further, the field video stream data is allocated bandwidth based on the dynamic bandwidth priority strategy of the second level queue, and a minimum bandwidth is reserved for the third level queue.

[0062] Further, the sensing data occupies the remaining bandwidth based on the dynamic bandwidth sub-priority strategy of the third level queue, and is sent in rotation with the second level queue according to a preset time slice length, i.e. the maximum continuous execution time length of each task is set, and the tasks are executed in rotation according to the ready queue, and are queued again after the execution time length.

[0063] Further, the backup data and software update data are based on the idle trigger mechanism of the fourth queue, when the load of the main channel is lower than the first preset percentage and lasts more than a preset time length, a second preset percentage of the remaining bandwidth is allocated to the backup data and software update data transmission.

[0064] Further, referring to Figure 4 , in step S4, the controller monitors the communication state of the main channel, and switches to the emergency channel when the communication state is abnormal, including: S41, the controller collects the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate of the main channel according to a preset frequency; S42, whether the switching condition is met is determined according to the signal strength, packet loss rate, transmission delay and cyclic redundancy check error rate, and the emergency channel is switched to when the switching condition is met.

[0065] Further, after switching to the emergency channel, further comprising: The controller determines whether the recovery condition is met according to the signal strength and packet loss rate of the main channel, and switches to the main channel and synchronizes the last valid control instruction snapshot of the emergency channel to the main channel if the recovery condition is met.

[0066] Specifically, in step S42, the switching condition is that at least two times of sampling meet: the signal strength is less than a preset minimum signal strength and the packet loss rate is greater than a first preset percentage and lasts more than a first preset time length, and / or the cyclic redundancy check error rate is greater than a second preset percentage or the transmission delay is greater than a preset time delay.

[0067] The recovery condition is that the signal strength is greater than a preset strength value and lasts a second preset time length, and the packet loss rate is less than a third preset percentage.

[0068] The remote mechanical arm control system and method based on multi-mode communication provided by the above embodiments at least have the following beneficial effects: (1) The communication connection between the user end and the device end is established based on the main channel and the emergency channel, so that the user end can realize remote control of the mechanical arm of the device end, and when the network fluctuates, the channel can be automatically switched to ensure the reliability of the communication, thereby ensuring the reliability of the remote operation of the mechanical arm, and it is particularly suitable for positioning the spacer rod clamp and the cable in areas where it is inconvenient to control on site; (2) The device end combines the image acquisition device and the improved path planning method, so that the mechanical arm realizes the whole path planning from starting to walking to task completion, realizes the accurate positioning and control of the mechanical arm, and ensures the reliability of the task implementation; (3) The PnP algorithm and the interpolation algorithm are fused to realize the control of the mechanical arm pose and joint angle, and are suitable for complex task operations of the mechanical arm; (4) The communication resources of the main channel are allocated through the cloud platform to guarantee the real-time performance of important instructions, fully utilize the bandwidth resources, and realize efficient data interaction between the device end and the user end.

[0069] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments. It is apparent that those skilled in the art can, without departing from the spirit or scope of the application, make various changes and modifications of the application. Thus, the application is intended to encompass all such changes and modifications as fall within the scope of the claims, together with all equivalents thereof.

Claims

1. A remote robotic arm control system based on multimode communication, characterized in that, It includes a user terminal and a device terminal, and the user terminal and the device terminal establish a communication connection through a multi-mode communication system, which includes a main channel, an emergency channel and a cloud platform; The device includes an image acquisition device, a robotic arm, and a controller; The user terminal is used to send control commands to the controller through the main channel. The controller is used to control the image acquisition device to acquire on-site images according to the control commands, perform robotic arm control planning based on the on-site images, generate robotic arm control signals to control the robotic arm to perform operations, and collect on-site status information to send to the user terminal through the main channel. The cloud platform is used to allocate communication resources for the main channel; The controller is also used to monitor the communication status of the main channel, switch to the emergency channel when the communication status is abnormal, and synchronize the last valid control command snapshot of the main channel to the emergency channel.

2. The system according to claim 1, characterized in that, The main channel is in wireless communication mode, and the emergency channel is in wired communication mode.

3. A remote robotic arm control method based on multimode communication applied to the system as described in claim 1 or 2, characterized in that, include: The user terminal sends control commands to the controller via the main channel; The controller controls the image acquisition device to acquire on-site images according to the control command, performs robotic arm control planning based on the on-site images, generates robotic arm control signals to control the robotic arm to perform operations, and sends on-site status information to the user terminal through the main channel. The cloud platform allocates communication resources for the main channel; The controller monitors the communication status of the main channel. When the communication status is abnormal, it switches to the emergency channel and synchronizes the last valid control command snapshot of the main channel to the emergency channel.

4. The method according to claim 3, characterized in that, The on-site images include RGB images and depth images; Based on the on-site images, a robotic arm control plan is developed, and robotic arm control signals are generated to control the robotic arm to perform operations, including: Establish the mapping relationship between the RGB image and the depth image, construct the field search space, and obtain the camera intrinsic parameter matrix and distortion coefficients; Target recognition is performed based on the RGB image to determine obstacles and task areas, and the task area is mapped to three-dimensional space according to the mapping relationship to obtain the task area space; Based on the task execution order, a task path is generated in the task area space, and the three-dimensional coordinates of the task starting point are determined. Obtain the two-dimensional coordinates of the obstacle, and map the two-dimensional coordinates of the obstacle to three-dimensional space according to the mapping relationship to obtain the three-dimensional coordinates of the obstacle; Based on the three-dimensional coordinates of the starting point of the robotic arm, the three-dimensional coordinates of the obstacle, the three-dimensional coordinates of the task starting point, and the on-site search space, a path is planned to obtain a walking path. The walking path is then integrated with the task path to obtain the overall planned path. Select a target point in the task path; Based on the camera intrinsic parameter matrix, the base coordinates of the robotic arm, and the target point, calculate the pose sequence and joint angle sequence of the robotic arm in the task path; The robotic arm control signals are generated based on the overall planning path, pose sequence, and joint angle sequence to control the robotic arm to perform operations.

5. The method according to claim 4, characterized in that, Based on the three-dimensional coordinates of the robotic arm's starting point, the three-dimensional coordinates of the obstacles, the three-dimensional coordinates of the task's starting point, and the on-site search space, path planning is performed to obtain the walking path, including: Initialize the search tree, with the three-dimensional coordinates of the starting point of the robotic arm as the root node; An obstacle list is created based on the three-dimensional coordinates of the obstacles; Iteratively execute the following steps until the stopping condition is met: Randomly select a node that is not in the obstacle list within the on-site search space, search for the nearest neighbor node to the random node in the search tree, and expand the node from the neighbor node toward the random node by a preset step size to obtain a new node; Based on the search tree, select nodes within the neighborhood range of the new node to obtain a set of neighborhood nodes; Traverse each undetermined node in the neighborhood node set, calculate the first cost of connecting from the root node to the undetermined node and then to the new node, select the undetermined node with the minimum first cost as the parent node of the new node, and add the new node to the search tree; Traverse each undetermined node in the neighborhood node set, calculate the second cost from the root node to the new node and then to the undetermined node. If there exists a second cost less than the minimum first cost, then the new node is taken as the parent node of the corresponding undetermined node. When the coordinates of the new node coincide with the three-dimensional coordinates of the task starting point, or when the distance between the coordinates of the new node and the three-dimensional coordinates of the task starting point is less than a preset distance, the current iteration stops, and the process backtracks from the three-dimensional coordinates of the task starting point to the three-dimensional coordinates of the robot arm's starting point, recording the currently obtained parent node as a candidate planning path. When the stopping condition is met, the shortest path is selected from the candidate planned paths as the final walking path.

6. The method according to claim 4, characterized in that, Based on the camera intrinsic parameter matrix, the robot arm's base coordinates, and the target point, calculate the robot arm's pose sequence and joint angle sequence along the task path, including: Select multiple non-coplanar feature points within a preset neighborhood of the target point, obtain the two-dimensional coordinates of the feature points in the RGB image according to the mapping relationship, and form a coordinate point pair by combining the two-dimensional coordinates and the three-dimensional coordinates of the feature points. Based on the PnP algorithm, the rotation vector and translation vector of the corresponding target point are calculated according to the coordinate point pair, the camera intrinsic parameter matrix and the distortion coefficient. The rotation vector and translation vector form the pose of the corresponding target point. Interpolation is performed based on the pose of each target point to obtain a pose sequence; Based on the pose sequence, the corresponding joint angles are solved using the Jacobi pseudo-inverse algorithm to obtain the joint angle sequence.

7. The method according to claim 3, characterized in that, The cloud platform allocates communication resources for the main channel, including: Construct a first-level queue, a second-level queue, a third-level queue, and a fourth-level queue. The first-level queue adopts a pre-allocated fixed time slot mechanism and a forced preemption mechanism. The second-level queue adopts a dynamic bandwidth priority strategy. The third-level queue adopts a dynamic bandwidth secondary priority strategy and a time slice round-robin mechanism. The fourth-level queue adopts an idle trigger mechanism. Based on the task level of data transmission between the user terminal and the device terminal, task scheduling is performed using the first-level queue, the second-level queue, the third-level queue, and the fourth-level queue.

8. The method according to claim 7, characterized in that, The data transmitted between the user terminal and the device terminal includes the control commands, the on-site status information, emergency commands, backup data, and software update data; the on-site status information includes on-site video stream data and sensor data. The control commands and the emergency commands exclusively occupy the pre-allocated time slots based on the pre-allocated fixed time slot mechanism of the first-level queue, and interrupt the tasks of the second-level queue, the third-level queue, and the fourth-level queue. The live video stream data is allocated bandwidth based on the dynamic bandwidth priority strategy of the second-level queue; The sensor data occupies the remaining bandwidth based on the third-level queue dynamic bandwidth second-priority strategy, and is sent in rotation with the second-level queue according to a preset time slice length. The backup data and software update data are based on the idle triggering mechanism of the fourth-level queue. When the load of the main channel is lower than the first preset percentage and continues for more than the preset duration, the remaining bandwidth is allocated to transmit the backup data and software update data.

9. The method according to claim 3, characterized in that, The controller monitors the communication status of the main channel and switches to the emergency channel when the communication status is abnormal, including: The controller collects the signal strength, packet loss rate, transmission delay, and cyclic redundancy check error rate of the main channel according to a preset frequency. The handover conditions are determined based on the signal strength, packet loss rate, transmission delay, and cyclic redundancy check error rate. If the handover conditions are met, the system switches to the emergency channel.

10. The method according to claim 9, characterized in that, After switching to the emergency exit, the following is also included: The controller determines whether the recovery conditions are met based on the signal strength and packet loss rate of the main channel. If the recovery conditions are met, it switches to the main channel and synchronizes the last valid control command snapshot of the emergency channel to the main channel.

Images