Wheel-leg cluster robot system based on gesture control

Abstract

A wheel-legged cluster robot control system comprises: a kinematics module for controlling a plurality of wheel-legged robots by establishing a closed-chain kinematics model; the distributed formation control module is used for distributed formation control based on a rigid graph theory and deriving a speed driving control law to maintain formation stability; the multi-mode gesture recognition module is used for realizing man-machine interaction and analyzing a gesture instruction through a layered state machine structure; and the communication positioning module is used for integrating the communication framework and the positioning module and coordinating the movement of the multiple robots in real time. According to the formation control rate based on the rigid graph theory, each robot in the system independently calculates the motion control quantity of the robot, and stable formation of a robot cluster in a dynamic environment can be ensured; through combination of gestures and robot control instructions, multi-mode and flexible instruction control is realized, and autonomy and task execution efficiency of a robot cluster are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control, in particular to a wheel-legged cluster robot system based on gesture control. BACKGROUND

[0002] In recent years, robot technology has made significant progress in industrial automation, military reconnaissance, logistics transportation and service fields. Multi-robot formation control technology has received widespread attention because it can efficiently complete complex tasks and enhance system robustness through the coordinated operation of cluster robots. Wheel-legged robots have become an ideal carrier for formation control in unstructured environments due to their combination of high-speed movement and terrain adaptability.

[0003] Existing formation control techniques mainly rely on pre-programming, remote control devices or computer interface inputs. For example, pre-programmed multi-robot collaborative handling has been widely used in industrial assembly lines; unmanned aerial vehicles achieve reconnaissance and strike missions through remote control instructions in the military field; and multi-robots complete goods sorting and path planning through central server scheduling in logistics and warehousing.

[0004] At the same time, gesture control as a new interactive method has emerged in single robot operation. For example, simple task control of service robots is achieved through Kinect devices; and flight paths are adjusted through gestures in the field of unmanned aerial vehicles.

[0005] Existing formation control methods are mostly based on centralized architecture and rely on pre-set communication topology, making it difficult to achieve adaptive collaboration in dynamic environments. In order to extend single robot gesture control to robot formation and achieve dynamic adjustment and task allocation, the following problems exist:

[0006] First, the multi-level mapping mechanism of the instruction object. Gesture instructions must accurately distinguish between controlling the entire robot formation or a single robot. Second, task attribute decoupling and coordination constraints. Gesture instructions must consider the distinction between formation tasks and individual robot tasks in execution, and there are differences in attributes between robot formations and individual robots. Third, dynamic behavior coordination mechanism. The balance between executing instructions and maintaining formation structure for individual robots must be considered. SUMMARY

[0007] To solve the above technical problems, the purpose of the present application is to provide a wheel-legged robot formation control method, gesture interaction method and system to solve the existing problems of gesture-controlled robot formation. The gesture space information of the controller is obtained using a visual scheme and mapped to the instruction space, combining gesture instructions and robot formation control instructions to achieve precise control of the robot formation.

[0008] To achieve the above purpose, the present application adopts the following scheme.

[0009] A wheel-legged swarm robot control system, comprising:

[0010] A kinematics module for controlling a plurality of wheel-legged robots by establishing a closed kinematics model;

[0011] A distributed formation control module for deriving a velocity-driven control law to maintain formation stability based on a rigid graph theory-based distributed formation control;

[0012] A multi-modal gesture recognition module for realizing human-robot interaction by parsing gesture instructions through a hierarchical state machine structure and sending them to a communication network;

[0013] A communication positioning module for integrating a communication framework and a positioning module to coordinate multi-robot motion in real time.

[0014] Optionally, the distributed formation control module performs the following steps:

[0015] Step 1, calculate the robot position through an extended Kalman filter (EKF) algorithm, broadcast to the communication network and receive other robot position information;

[0016] Step 2, abstract the formation target and the current formation into a weighted graph structure, and solve the optimal position assignment using the Hungarian algorithm;

[0017] Step 3, construct a gradient control law based on the rigid graph theory, derive a velocity-driven control law through a single-integral model, and calculate the robot's own control quantity in the global common coordinate system to maintain formation stability.

[0018] Optionally, the multi-modal gesture recognition module performs the following steps:

[0019] Step 1, classify gestures into static gestures, dynamic finger states, hand spatial poses, and trajectory features;

[0020] Step 2, parse gesture instructions using a hierarchical state machine structure, divide gesture semantics into formation control and individual control modalities, and further subdivide them into motion control and behavior control submodalities.

[0021] Optionally, the communication framework includes a Robot Operating System 2 (ROS2) framework, and the positioning module includes an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system.

[0022] A wheel-legged swarm robot formation control system, comprising:

[0023] A physical layer composed of a plurality of wheel-legged robots, each robot forming a dynamic reconfigurable formation topology through a Wi-Fi communication network;

[0024] The control layer comprises: a top-level formation controller and a single-machine controller; the top-level formation controller calculates a global control variable based on a rigidity graph theory, and the single-machine controller generates a driving signal by fusing a gesture instruction;

[0025] The perception layer comprises: a positioning perception module and a visual perception module; the positioning perception module adopts UWB and IMU fusion positioning, and the visual perception module extracts gesture features based on an RGB-D camera and a deep learning algorithm;

[0026] The communication layer constructs a communication topology model based on a ROS2 DDS protocol stack, and supports real-time data sharing and instruction broadcasting of multiple robots.

[0027] Optionally, a state equation of the positioning perception module is as follows:

[0028]

[0029] wherein x and y are position coordinates of the robot in a global coordinate system, is a heading angle of the robot in the global coordinate system, v and w are linear speed and angular speed of the robot, b is an initial deviation of the heading angle, and the above variables sequentially constitute a state vector .

[0030] Optionally, the top-level formation controller comprises: a positioning node, a formation node, and a gesture node.

[0031] The positioning node receives UWB and IMU data, and checks geometric consistency of a formation shape.

[0032] The formation node calculates a global control variable based on a rigidity graph theory, and fuses a gesture instruction to generate a synthesized control variable.

[0033] The gesture node receives visual perception data, classifies gesture features through a deep learning model, and maps the gesture features to an instruction space.

[0034] A gesture interaction method of a wheel-leg cluster robot system comprises the following steps:

[0035] Step 1: acquiring user gesture images through an RGB-D camera, and extracting static gesture, finger trajectory, and palm spatial posture features;

[0036] Step 2: detecting a hand region by using a MeidaPipe Hand Pose Landmark model, and analyzing gesture semantics by using multiple methods;

[0037] Step 3: analyzing gesture instructions through a hierarchical state machine, mapping gesture semantics to a preset instruction space, and supporting a standard motion control mode, a switching motion control mode, and a behavior instruction trigger.

[0038] Optionally, the hierarchical state machine further subdivides the formation control and individual control modes into motion control and behavior control sub-modes; and maps classified gestures to specific control commands according to the sub-modes to achieve stable command parsing.

[0039] Further features include: integrating the parsed control commands with the Robot Operating System 2 (ROS2) framework to coordinate multiple wheeled robots; and using an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system to provide real-time positioning for multiple wheeled robots.

[0040] Compared with the prior art, the present invention has the following beneficial technical effects:

[0041] By employing a formation control law based on rigid graph theory, each robot in the system independently calculates its own motion control quantities, ensuring stable formation of the robot swarm in dynamic environments. Combining RGB-D cameras with deep learning algorithms, gestures are accurately recognized and converted into control commands in real time, significantly improving the accuracy and response speed of human-computer interaction. Through the combination of gestures and robot control commands, multimodal and flexible command control is achieved, enhancing the autonomy and task execution efficiency of the robot swarm. Attached Figure Description

[0042] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.

[0043] Figure 1 This is a schematic diagram of the wheeled robot structure in one embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a controller planning algorithm in one embodiment of the present invention;

[0045] Figure 3 This is a system overall architecture diagram in one embodiment of the present invention;

[0046] Figure 4 This is a control module architecture diagram in one embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the gesture control process in one embodiment of the present invention. Detailed Implementation

[0048] The following is in conjunction with the appendix Figures 1 to 5The present invention will be further described in detail below with reference to the embodiments. It is to be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The technical solution of this invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] Unless otherwise stated, the exemplary embodiments / exemplifications shown are to be understood as providing exemplary features of various details that provide ways in which the technical concept of the invention can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / exemplifications may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concept of the invention.

[0051] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0052] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0053] For descriptive purposes, the present invention may use spatial relative terms such as “below,” “under,” “below,” “down,” “above,” “above,” “higher,” and “side (e.g., in a “sidewall”)” to describe the relationship between one component and another component as shown in the accompanying drawings. In addition to the orientations depicted in the drawings, the spatial relative terms are also intended to encompass different orientations of the device during use, operation, and / or manufacture. For example, if the device in the drawings is flipped, a component described as “below” or “under” another component or feature would subsequently be positioned “above” said other component or feature. Thus, the exemplary term “below” can encompass both “above” and “below” orientations. Furthermore, the device may be otherwise positioned (e.g., rotated 90 degrees or in other orientations), thus interpreting the spatial relative descriptive terms used herein accordingly.

[0054] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0055] In one embodiment, the present invention provides a control system for a wheeled legged swarm robot, comprising:

[0056] The kinematics module is used to control multiple wheeled robots by establishing a closed-loop kinematic model;

[0057] The distributed formation control module is used for distributed formation control based on rigid graph theory, and derives the speed-driven control law to maintain formation stability.

[0058] A multimodal gesture recognition module is used to realize human-computer interaction. It parses gesture commands through a hierarchical state machine structure and sends them to the communication network.

[0059] The communication and positioning module integrates the communication framework and the positioning module to coordinate the movement of multiple robots in real time.

[0060] Optionally, the distributed formation control module performs the following steps:

[0061] Step 1: Calculate the robot's position using the Extended Kalman Filter (EKF) algorithm, broadcast it to the communication network, and receive position information from other robots;

[0062] Step 2: Abstract the formation target and the current formation into a weighted graph structure, and use the Hungarian algorithm to solve for the optimal position assignment;

[0063] Step 3: Construct a gradient control law based on rigid graph theory, derive the velocity-driven control law through a single integral model, and calculate the robot's own control quantity in the global common coordinate system to maintain formation stability.

[0064] Optionally, the multimodal gesture recognition module performs the following steps:

[0065] Step 1: Classify gestures into static gestures, dynamic finger states, hand spatial posture, and trajectory features;

[0066] Step 2: Use a hierarchical state machine structure to parse gesture commands, divide gesture semantics into formation control and individual control modes, and further subdivide them into motion control and behavior control sub-modalities.

[0067] Optionally, the communication framework includes a Robot Operating System 2 (ROS2) framework, and the positioning module includes an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system.

[0068] A swarm control system for wheeled legged robots includes:

[0069] The physical layer consists of multiple wheeled and legged robots, which form a dynamically reconfigurable formation topology through a Wi-Fi communication network.

[0070] The control layer includes a top-level formation controller and a single-machine controller; the top-level formation controller calculates global control variables based on rigid graph theory, and the single-machine controller integrates gesture commands to generate drive signals.

[0071] The perception layer includes a localization perception module and a visual perception module. The localization perception module uses UWB and IMU fusion localization, and the visual perception module extracts gesture features based on an RGB-D camera and a deep learning algorithm.

[0072] The communication layer is based on the ROS2 DDS protocol stack to build a communication topology model, supporting real-time data sharing and command broadcasting among multiple robots.

[0073] Optionally, the state equation of the positioning sensing module is:

[0074]

[0075] Where x and y are the robot's position coordinates in the global coordinate system. Let v be the robot's heading angle in the global coordinate system, v and w be the robot's linear velocity and angular velocity, and b be the initial deviation of the heading angle. These variables, in order, constitute the state vector. .

[0076] Optionally, the top-level formation controller includes: a positioning node, a formation node, and a gesture node.

[0077] The positioning node receives UWB and IMU data and verifies the geometric consistency of the formation configuration.

[0078] The formation nodes calculate global control quantities based on rigid graph theory and generate synthetic control quantities by fusing gesture commands.

[0079] The gesture node receives visual perception data, classifies gesture features through a deep learning model, and maps them to the instruction space.

[0080] A gesture interaction method for a wheeled legged swarm robot system includes the following steps:

[0081] Step 1: Capture user gesture images using an RGB-D camera, and extract static gestures, finger trajectories, and palm spatial pose features;

[0082] Step 2: Use the MeidaPipe Hand Pose Landmark model to detect the hand region, and combine multiple methods to analyze the semantics of the gesture;

[0083] Step 3: Parse gesture commands through a hierarchical state machine, map the gesture semantics to a preset command space, and support standard motion control mode, switching motion control mode and triggering behavior commands.

[0084] Optionally, the hierarchical state machine further subdivides the formation control and individual control modes into motion control and behavior control sub-modes; and maps classified gestures to specific control commands according to the sub-modes to achieve stable command parsing.

[0085] Further features include: integrating the parsed control commands with the Robot Operating System 2 (ROS2) framework to coordinate multiple wheeled robots; and using an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system to provide real-time positioning for multiple wheeled robots.

[0086] The establishment of the closed-chain kinematic model includes:

[0087] Step 11: Construct the differential drive kinematic model of the tire section;

[0088] The differential drive kinematic model of the tire section is as follows:

[0089]

[0090] In the formula, d is the wheelbase and R is the wheel diameter;

[0091] Step 12: Construct a composite kinematic model of the wheel-legged robot;

[0092] 1201. Establish the geometric constraint equations for the five-bar closed-loop mechanism:

[0093]

[0094] Where 1 corresponds to Figure 1 In the diagram, rod 6 corresponds to rod 5, rod 3 corresponds to rod 3, rod 4 corresponds to rod 2, and C corresponds to wheel 4. Let be the length of the i-th rod. The joint angle corresponding to the i-th link, This represents the distance between the fuselage and wheel 4 in the ground coordinate system.

[0095] Using the target spatial coordinates of the fuselage (1) as input, the joint angle parameters of each link are solved to achieve active attitude adjustment;

[0096] 1202. Combine the differential drive kinematic model with the geometric constraint equations of the five-bar closed-loop mechanism to generate a composite kinematic model of the wheel-leg robot, and convert the linear velocity and angular velocity output by the hub motor into pose increments in the global coordinate system.

[0097] Step 13: Implement multimodal control using the main controller, including the following steps:

[0098] 1301. The robot subscribes to relevant topics through the ROS2 system to receive and publish gesture control quantities such as formation control quantities and overall group motion control quantities from other robots to the Wi-Fi communication network.

[0099] 1302. Based on the preset weighting coefficients, the formation structure control quantity and the gesture control quantity are weighted and fused to generate a composite control quantity;

[0100] ,

[0101] in, This is the final synthesized formation control quantity. The control quantity is calculated using rigid diagram theory. For gesture control quantity, These are the weighting coefficients;

[0102] 1303. The composite control quantity in the global coordinate system is converted into the control quantity in the local coordinate system through the composite kinematic model of the wheel-legged robot, such as... Figure 2As shown, different fusion strategies are selected when the global velocity vector is in different regions.

[0103] Specifically, the velocity vector is divided into three regions: the positioning error region, the standard control region, and the rotation priority region. When the global velocity vector is in the positioning error region, the heading angle is considered to be in place, and no further angle adjustment is made; only the forward speed control quantity is issued. When the global velocity vector is in the standard control region, the heading angle is adjusted proportionally, and the forward speed control quantity is calculated based on the projection. When the global velocity vector is in the rotation priority region, the heading angle error is considered to be large, and the forward speed control quantity is not issued temporarily; instead, the heading angle is adjusted proportionally.

[0104] In one embodiment, the present invention provides a control method for a wheeled legged swarm robot, comprising:

[0105] Step 1: Control multiple wheeled-legged robots. Each wheeled-legged robot includes a five-bar closed-chain mechanism. By establishing a closed-chain kinematic model, calculate the mapping relationship between joint angles and foot positions.

[0106] Step 2: Implement distributed formation control based on rigid graph theory for multiple wheeled robots, and derive the speed drive control law based on the single integral model to maintain formation stability;

[0107] Step 3: Implement human-computer interaction through a vision-based multimodal gesture recognition system, detect and classify gestures into formation control commands and individual robot control commands, and use a hierarchical state machine architecture for command parsing;

[0108] Step 4: Coordinate multiple wheeled robots in real time through an integrated communication framework and positioning module.

[0109] In one embodiment, step 2, the distributed formation control includes:

[0110] 201. The robot calculates its own position using the extended Kalman filter algorithm and broadcasts it to the communication network. At the same time, it receives position information of other robots from the communication network, compares the formation target setting with the current formation structure, abstracts the two positions into a weighted graph structure, and uses the Hungarian algorithm to solve the optimal weighted bipartite graph matching problem to calculate the robot's position assignment and achieve the optimal formation establishment.

[0111] 202. Based on rigid graph theory, a gradient control law is constructed. Each robot obtains the positions of other robots through a communication network and independently calculates its own control variables in the global common coordinate system. Then, it is converted to its own control variables through a control variable transformation rule. The calculation formula is shown below:

[0112]

[0113] in, Let be the velocity control variable of robot i in the global coordinate system. The error coefficient is defined by the user. It is the relative position vector between robot i and robot j. Let be the error value between the current relative distance of robots i and j and the preset relative distance.

[0114] In step 3, human-computer interaction is achieved through a vision-based multimodal gesture recognition system, including:

[0115] 301. Classify gestures into static gestures, dynamic finger states, hand spatial posture, and trajectory features;

[0116] For the analysis of static gestures, YOLOv10n is used for classification and judgment.

[0117] For the analysis of the other three states, the MediaPipe model is used to obtain the key point coordinates of the hand. For dynamic finger states, specifically including dynamic contact and dynamic flexion / extension states, geometric methods are used to analyze and judge the key point coordinates, and a state machine structure is used to ensure the accuracy of state transitions. For hand spatial pose, the least squares method is used to estimate the rotation matrix of the current pose relative to the initial pose. For trajectory features, it is simplified into a time series classification problem. Users can preset the order relationship of vertices and edges of a series of geometric figures as the classification basis. A Gaussian-Hidden Markov Model is used to evaluate all preset figures, and the optimal evaluation result is selected as the classification result.

[0118] 302. A hierarchical state machine architecture is used to parse and classify gestures into control commands; wherein, the architecture divides gesture semantics into two modalities: formation control and individual control, and each modality is further subdivided into motion control and behavior control submodalities;

[0119] During system operation, the system first determines the modality corresponding to the control hand in the image frame. Then, by combining the modality with hand features, it determines the parsed gesture category and thus the semantic information of the gesture. Subsequently, it issues control signals based on the correspondence between the set semantic information and control commands.

[0120] In step 4, the communication framework includes the Robot Operating System 2 (ROS2) framework, and the positioning module includes an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system.

[0121] In one embodiment, such as Figure 1As shown, the present invention provides a wheeled robot, comprising: a body 1, a leg linkage assembly including a first leg linkage 2, a second leg linkage 3, a third leg linkage 5, and a fourth leg linkage 6; a tire part 4, and body motors 7 and 8; the first to fourth leg linkages 2, 3, 5, and 6 and the body constitute a closed-loop mechanism, which can be extended to any wheeled robot.

[0122] The spatial attitude of the fuselage 1 is adjusted through multi-degree-of-freedom linkage; the tire part 4 includes symmetrically arranged left and right hub motor drive units; the fuselage motors 7 and 8 are DC servo motors, which are respectively connected to the two drive ends of the leg linkage group to control the attitude of the fuselage in torque mode; the fuselage 1 integrates a main controller, IMU sensor, Wi-Fi communication module and hub motor driver; the main controller is electrically connected to the joint encoder, IMU sensor, Wi-Fi communication module and hub motor driver of the leg linkage group to receive and parse global control commands and output drive signals of each actuator.

[0123] In one embodiment, such as Figure 3 As shown, the present invention provides a wheeled robot formation control system, which includes a four-layer structure: a physical layer (A), a control layer (B), a perception layer (C), and a communication layer (D). The layers interact and coordinate control through data interfaces.

[0124] The physical layer (A) includes a wheeled robot formation system (A1) and a user terminal (A2);

[0125] The wheeled-legged robot formation system (A1) consists of multiple such... Figure 1 The wheeled-legged robots described herein are connected in a distributed manner through a Wi-Fi communication network, forming a dynamically reconfigurable formation topology.

[0126] The user terminal (A2) is a user computer and a wireless communication module, used to input gesture commands, voice commands, or preset formation task parameters.

[0127] The control layer (B) includes a top-level formation controller (B1) and a single-robot controller (B2);

[0128] The top-level formation controller (B1) is based on rigid graph theory to construct gradient control law, which is used to calculate the global speed control quantity of all robots in the formation, ensuring the stability and consistency of the formation in the dynamic environment.

[0129] The single-robot controller (B2) is deployed inside each wheel-legged robot. It is used to receive top-level control signals and gesture commands, generate drive signals by combining its own kinematic model, and realize posture adjustment through the actuator.

[0130] The top-level formation controller (B1) and the single-robot controller (B2) achieve control quantity synchronization through a broadcast communication protocol, supporting formation mode switching and abnormal state recovery.

[0131] The perception layer (C) includes a positioning perception module (C1) and a visual perception module (C2);

[0132] The positioning and sensing module (C1) uses the EKF (Extended Kalman Filter) algorithm, which combines UWB (Ultra-Wideband) and inertial sensors, to acquire the global coordinates, heading angle, and velocity information of each robot in the formation in real time.

[0133] The positioning and sensing module (C1) performs the following steps:

[0134] 1. UWB obtains the robot's (x,y) position coordinates in real time, and IMU obtains the robot's heading angle.

[0135] 2. Establish state equations and observation functions, and update the robot's posture through the state update equations in each time period.

[0136] The state equation is as follows:

[0137]

[0138] Where x and y are the robot's position coordinates in the global coordinate system. Let v be the robot's heading angle in the global coordinate system, v and w be the robot's linear velocity and angular velocity, and b be the initial deviation of the heading angle. These variables, in order, constitute the state vector. .

[0139] Observation function The observation matrix H is:

[0140]

[0141] The visual perception module (C2) is based on an RGB-D camera and a deep learning algorithm. It is used to collect user gesture images and extract key features, including static gestures, finger trajectories and palm spatial posture information.

[0142] The gesture recognition method of the visual perception module (C2) includes:

[0143] 1. By using real-time video footage as input, the MidaPipe Poes Landmark neural network is used to achieve hand tracking, output of left and right hand movement, and output of hand key point coordinates.

[0144] 2. Determine the flexion and extension state of the finger based on the geometric relationship of key points. The formula is as follows: ,in, It is the vector from the fingertip pixel coordinates to the pixel coordinates of the middle joint of the finger. It is a vector from the pixel coordinates of the middle joint of the finger to the pixel coordinates of the base of the finger. The angle variable used to determine the flexion-extension state is based on... The numerical relationship with the preset threshold is judged.

[0145] 3. Based on different combinations of finger flexion and extension, different control modes are entered, and key point information is distributed to different analysis functions according to different control modes;

[0146] In speed control mode, the hand posture analysis function is called to map the hand posture changes to the robot posture changes, thereby realizing the robot's speed motion control; in formation setting mode, the time series information of specified key points is recorded, and the time series is classified using a Gaussian-Hidden Markov Model.

[0147] The communication layer (D) includes a data parsing unit (D1) and a broadcast transmission unit (D2);

[0148] The data parsing unit (D1) is used to receive the output data of the perception layer (C), perform format conversion and conflict detection on the positioning information and gesture features, and generate standardized control commands.

[0149] The broadcast transmission unit (D2) is based on the ROS2 DDS protocol stack to build a communication topology model, which supports real-time data sharing and command broadcasting among multiple robots, and its transmission delay meets the requirements of formation synchronization control.

[0150] The communication layer (D) and the control layer (B) communicate with each other via a CAN bus or wireless protocol to ensure low-latency control input and real-time status feedback.

[0151] The data interaction methods of the communication layer (D) include:

[0152] Step 1: The data parsing unit (D1) receives the output data (location information, gesture features) from the perception layer (C), and performs format conversion and conflict detection;

[0153] Step 2: Generate standardized control commands and transmit them to the control layer (B) via the broadcast transmission unit (D2) based on the ROS2 DDS protocol stack.

[0154] Step 3: The control layer (B) and the physical layer (A) achieve low-latency data interaction through CAN bus or wireless protocol to ensure real-time control output and status feedback.

[0155] In one embodiment, the present invention provides a cooperative control method for a wheeled legged robot formation control system, comprising:

[0156] Step 1: The perception layer (C) collects the status data (such as location information and gesture features) of the physical layer (A) in real time and transmits it to the control layer (B) through the communication layer (D);

[0157] Step 2: The top-level formation controller (B1) calculates the global control quantity based on rigid graph theory and sends it down to each individual robot controller (B2) through the broadcast transmission unit (D2).

[0158] Step 3: The standalone robot controller (B2) integrates gesture commands and formation control quantities, generates driving signals based on the composite kinematic model, and controls the physical layer (A) to perform actions;

[0159] Step 4: The user terminal (A2) displays the formation status and environmental information through the feedback interface, supporting manual intervention and dynamic adjustment of task parameters.

[0160] In one embodiment, such as Figure 3 As shown, this invention provides a control layer structure for a wheeled legged robot formation control system. The control layer (B) comprises two parts: an upper computer (B10) and a lower computer (B20), wherein:

[0161] The host computer (B10) consists of a positioning node (B11), a formation node (B12), and a gesture node (B13);

[0162] The lower-level machine (B20) includes sensor tasks (B21), calculation tasks (B22), communication tasks (B23), motor tasks (B24), and other tasks (B25);

[0163] The host computer (B10) interacts with the communication layer (D) via a positioning broadcast protocol, and communicates with the slave computer (B20) via a serial bus to issue commands and provide status feedback.

[0164] The functions of the positioning node (B11) include:

[0165] The system receives the UWB positioning data parsed by the communication layer (D) and the EKF filtering results from the inertial sensor to generate global coordinate information for each wheel-legged robot in the formation.

[0166] Based on rigid graph theory, topological verification of positioning data is performed to remove abnormal nodes and correct formation configuration deviations, ensuring the geometric stability of formation control.

[0167] The corrected positioning information is sent to the formation node (B12) of the host computer (B10) via broadcast protocol as input parameters for formation control calculation.

[0168] The functions of the formation node (B12) include:

[0169] Based on rigid graph theory, a formation gradient control law is constructed, and the global velocity control quantity of each robot is calculated using the following mathematical model:

[0170]

[0171] In the formula, This is a control factor, with a default value of 0.1. Let i be the current relative position vector of robot i and j. This represents the error between the current distance of robots i and j and the set distance, which is calculated from the set formation shape.

[0172] The gesture control commands output by the gesture node (B13) are integrated to generate a composite control quantity through preset weight coefficients, supporting formation mode switching and dynamic path planning;

[0173] The control input is sent to the lower-level computer (B20) via the serial bus for the calculation task (B22), and the execution status information fed back by the lower-level computer is received.

[0174] The functions of the gesture node (B13) include:

[0175] The visual perception data parsed by the communication layer (D) includes: static gestures, finger trajectories, and palm spatial posture features.

[0176] The task allocation of the lower-level machine (B20) is as follows:

[0177] The sensor task (B21) is used to acquire feedback signals from the fuselage IMU, encoder, and hub motor, and to calibrate sensor errors in real time.

[0178] The solution task (B22) obtains the current robot posture based on a composite kinematics model;

[0179] The communication task (B23) is responsible for serial communication with the host computer (B10) and sending and receiving Wi-Fi broadcast data, and supports priority scheduling of control commands and data packet verification;

[0180] The motor task (B24) controls the hub motor and body motor based on the drive signal output by the solution task (B22) to realize robot posture adjustment and motion execution;

[0181] Other tasks (B25) include power management, abnormal state detection, and logging functions to ensure the reliability and traceability of system operation.

[0182] The method for coordinated control between the host computer and the slave computer includes:

[0183] Step 1: The positioning node (B11) obtains the positioning data (EKF filtering result of UWB and IMU fusion) from the communication layer (D) through the positioning broadcast protocol, and verifies the geometric consistency of the formation configuration.

[0184] Step 2: The formation node (B12) generates a global control variable based on the rigid graph gradient control law, and sends it to the solution task (B22) after fusing the instructions of the gesture node (B13).

[0185] Step 3: The solution task (B22) converts the control quantity into a motor drive signal through the inverse kinematics model, and the motor task (B24) executes the action;

[0186] Step 4: The sensor task (B21) and the communication task (B23) provide real-time feedback on their execution status to the formation node (B12), forming a closed-loop control loop;

[0187] Step 5: Other tasks (B25) monitor the system's operating status and trigger the exception handling mechanism.

[0188] In one embodiment, such as Figure 4 As shown, the present invention provides a gesture control host computer for a wheeled robot, the gesture control host computer comprising a three-layer architecture of a software layer (C10), a state layer (C20), and a mode layer (C30);

[0189] The software layer (C10) is used to receive gesture data from the visual perception module (C2) and perform preprocessing;

[0190] The state layer (C20) is used to identify the user's hand state and manage state transition logic;

[0191] The mode layer (C30) is used to switch between motion control mode and behavior control mode based on hand status.

[0192] The functions of the software layer (C10) include:

[0193] Receive RGB-D images and depth data transmitted from the communication layer (D), and extract key points and pose features of the hand through a pre-trained convolutional neural network (CNN);

[0194] Image data is filtered and noise suppressed to generate standardized gesture feature vectors, which are then used as input to the state layer (C20).

[0195] It supports multi-threaded processing to ensure that the real-time performance of gesture data acquisition and feature extraction meets the requirements of formation control.

[0196] The functions of the state layer (C20) include:

[0197] The current state is determined based on the hand position and key point trajectory, including the left hand state (C21), the right hand state (C22), the single hand loss state (C23), and the idle state after loss (C24).

[0198] The gesture-controlled host computer's state transition methods include:

[0199] Step 1: The initial state is the idle state after loss (C24). When the left hand / right hand is detected, switch to the corresponding state (C21 / C22).

[0200] Step 2: In state C21 / C22, if the target hand is not detected for more than the preset timeout period, the single hand loss state is entered (C23).

[0201] Step 3: In the single-hand loss state (C23), if the original target hand is detected again, return to the original state; if no hand is detected within the timeout and another hand is detected, switch to the state of the other hand; if no hand is detected within the timeout, enter the lost idle state (C24).

[0202] Step 4: Ensure transition stability and low misjudgment rate through state machine model.

[0203] The functions of the mode layer (C30) include:

[0204] In either left-hand mode (C21) or right-hand mode (C22), two control modes can be switched:

[0205] Motion control mode (C31): Through gestures and postures, the linear velocity and angular velocity control quantities of the robot are linearly mapped to the robot with a preset scaling factor, supporting basic operations such as forward movement, turning, and acceleration;

[0206] Behavior control modality (C32): Triggers preset behavior commands through specific gestures, such as advanced functions like formation reorganization, obstacle avoidance mode switching, and emergency braking;

[0207] Switching between modes is achieved through preset gesture commands, such as spreading five fingers to switch to behavior control mode, and clenching a fist to switch to motion control mode.

[0208] The gesture-controlled mode switching method for the host computer includes:

[0209] Step 1: Support two control modes in left-hand / right-hand mode:

[0210] Motion control mode (C31): Through gestures and postures, the linear velocity and angular velocity control quantities of the robot are linearly mapped to the robot with a preset scaling factor, supporting basic operations such as forward movement, turning, and acceleration;

[0211] Behavior control modality (C32): Triggers preset behavior commands through specific gestures, such as advanced functions like formation reorganization, obstacle avoidance mode switching, and emergency braking;

[0212] Step 2: Switch modes using preset gestures; spreading your five fingers switches to behavior control mode, and clenching your fist switches to motion control mode.

[0213] In one embodiment, the present invention discloses a formation control method for a wheeled legged swarm robot system based on gesture control, taking the control of a formation of three robots as an example;

[0214] In the specific implementation of the gesture control method, the user starts the robot and runs the ROS2 node on the robot's end. The wheeled robot switches to standing mode and waits for gesture commands. For example, the user faces the camera, expresses the posture of their left hand, and points in any direction (allowing a certain range of deviation). In practice, the camera transmits video information to the gesture recognition ROS2 node. The ROS2 node parses the gesture to determine the formation motion control mode, and sends the corresponding movement direction to each robot terminal through the ROS2 communication network.

[0215] In the specific implementation of the formation control method, each robot has built-in preset formation information. The robot formation control node receives the control command sent by the ROS2 network, its own position information sent by its own positioning node, and the position information of other robots received through the ROS2 network. It calculates its own control quantity through the formation control method and sends it to the lower-level machine through the serial port. The lower-level machine uses the control method to calculate the motor control quantity to realize motion control.

[0216] This embodiment illustrates how a user, facing a fixed or robot-mounted camera, uses gestures to control a formation of three robots.

[0217] In one embodiment, the present invention discloses a wheeled legged swarm robot system based on gesture control.

[0218] 1. System Startup and Initialization

[0219] Users start the system through terminal devices (such as laptops or handheld controllers) and run the top-level control node based on ROS2. After each wheeled robot is powered on, it automatically executes a self-test program and establishes a connection with the top-level controller via Wi-Fi network. After initialization, the robot switches to "standing and waiting" mode and waits for gesture commands.

[0220] 2. Gesture Recognition and Command Parsing

[0221] Users make preset gestures facing the RGB-D camera deployed on the system.

[0222] For example, spreading the five fingers of the left hand and keeping the palm facing the camera indicates entering the "formation motion control mode";

[0223] Tilting the left hand in a certain direction (such as up, down, left, or right) indicates that the entire formation is moving in that direction.

[0224] The visual perception module (C2) extracts the coordinates of 21 key points on the hand using the MediaPipe Hand Pose model and combines them with a hierarchical state machine to determine the semantics of the gesture.

[0225] If the left hand is detected with its five fingers spread and palm facing the camera, it is determined to be in "formation movement control mode";

[0226] The direction of movement can be further determined by the angle of hand tilt (such as calculating the angle between the palm normal vector and the direction of gravity);

[0227] The gesture node (B13) publishes the parsed instructions (such as "move forward in formation") to the ROS2 topic.

[0228] 3. Formation control and motion execution

[0229] Each robot receives instructions by subscribing to topics and, combining its own location information (UWB+IMU fusion positioning) with the location information of neighboring robots, executes the following steps:

[0230] The formation node (B12) calculates the desired speed control value for each robot based on rigid graph theory;

[0231] The standalone controller (B2) weights and fuses the formation control quantity with the gesture command to generate a composite control quantity;

[0232] The global velocity is converted into motor drive signals for each joint using an inverse kinematics model;

[0233] The lower-level machine (B20) performs motor control tasks to achieve overall translation or turning of the formation.

[0234] 4. Exception handling and status feedback

[0235] If a robot loses its positioning signal or communication is interrupted, the system automatically switches to "abnormal recovery mode":

[0236] The robot pauses executing the current command and uses LED indicators to signal an abnormality.

[0237] The remaining robots readjusted their formation based on the remaining formation structure;

[0238] Users can trigger an emergency stop by making a "fist" gesture, at which point all robots will immediately cease movement.

[0239] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0240] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0241] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present invention.

Claims

1. A control system for a wheeled-legged swarm robot, characterized in that, include: The kinematics module is used to control multiple wheeled robots by establishing a closed-loop kinematic model; The distributed formation control module is used for distributed formation control based on rigid graph theory, and derives the speed-driven control law to maintain formation stability. A multimodal gesture recognition module is used to realize human-computer interaction. It parses gesture commands through a hierarchical state machine structure and sends them to the communication network. The communication and positioning module integrates the communication framework and the positioning module to coordinate the movement of multiple robots in real time.

2. The system according to claim 1, characterized in that, Preferably, the distributed formation control module performs the following steps: Step 1: Calculate the robot's position using the Extended Kalman Filter (EKF) algorithm, broadcast it to the communication network, and receive position information from other robots; Step 2: Abstract the formation target and the current formation into a weighted graph structure, and use the Hungarian algorithm to solve for the optimal position assignment; Step 3: Construct a gradient control law based on rigid graph theory, derive the velocity-driven control law through a single integral model, and calculate the robot's own control quantity in the global common coordinate system to maintain formation stability.

3. The system according to claim 1, characterized in that, The multimodal gesture recognition module performs the following steps: Step 1: Classify gestures into static gestures, dynamic finger states, hand spatial posture, and trajectory features; Step 2: Use a hierarchical state machine structure to parse gesture commands, divide gesture semantics into formation control and individual control modes, and further subdivide them into motion control and behavior control sub-modalities.

4. The system according to claim 1, characterized in that, The communication framework includes a Robot Operating System 2 (ROS2) framework, and the positioning module includes an ultra-wideband (UWB) and inertial measurement unit (IMU) fusion positioning system.

5. A swarm control system for wheeled-legged swarm robots, characterized in that, include: The physical layer consists of multiple wheeled and legged robots, which form a dynamically reconfigurable formation topology through a Wi-Fi communication network. The control layer includes a top-level formation controller and a single-machine controller; the top-level formation controller calculates global control variables based on rigid graph theory, and the single-machine controller integrates gesture commands to generate drive signals; The perception layer includes a localization perception module and a visual perception module. The localization perception module uses UWB and IMU fusion localization, and the visual perception module extracts gesture features based on an RGB-D camera and a deep learning algorithm. The communication layer is based on the ROS2 DDS protocol stack to build a communication topology model, supporting real-time data sharing and command broadcasting among multiple robots.

6. The system according to claim 5, characterized in that, The state equation of the positioning sensing module is: , Where x and y are the robot's position coordinates in the global coordinate system. Let v be the robot's heading angle in the global coordinate system, v and w be the robot's linear velocity and angular velocity, and b be the initial deviation of the heading angle. These variables, in order, constitute the state vector. .

7. The system according to claim 5, characterized in that, The top-level formation controller includes: positioning nodes, formation nodes, and gesture nodes. The positioning node receives UWB and IMU data and verifies the geometric consistency of the formation configuration. The formation nodes calculate global control quantities based on rigid graph theory and generate synthetic control quantities by fusing gesture commands. The gesture node receives visual perception data, classifies gesture features through a deep learning model, and maps them to the instruction space.

8. A gesture interaction method for a wheeled-legged swarm robot system, characterized in that, Includes the following steps: Step 1: Capture user gesture images using an RGB-D camera, and extract static gestures, finger trajectories, and palm spatial pose features; Step 2: Use the MeidaPipe Hand Pose Landmark model to detect the hand region, and combine multiple methods to analyze the semantics of the gesture; Step 3: Parse gesture commands through a hierarchical state machine, map the gesture semantics to a preset command space, and support standard motion control mode, switching motion control mode and triggering behavior commands.

9. The method according to claim 8, characterized in that, The hierarchical state machine subdivides the formation control and individual control modes into motion control and behavior control sub-modes; based on the sub-modes, it maps classified gestures to specific control commands to achieve stable command parsing.

10. The method according to claim 8, characterized in that, Further includes: The parsed control commands are integrated with the Robot Operating System 2 (ROS2) framework to coordinate multiple wheeled and legged robots; A fusion positioning system using ultra-wideband (UWB) and inertial measurement unit (IMU) provides real-time positioning for multiple wheeled robots.

Images