Method and apparatus for controlling robot based on monocular vision gesture driving

Abstract

The present disclosure relates to the technical field of computer, and particularly relates to a robot control method and device based on monocular vision gesture driving. The method comprises the following steps: acquiring a first palm image collected by a monocular camera; detecting the first palm image by using a YOLO hand region detection network to acquire a hand region image in the first palm image; identifying the hand region image by using a visual model to acquire coordinates corresponding to each hand key point in a hand key point set; identifying the coordinates corresponding to each hand key point by using a gesture recognition model to acquire gesture information corresponding to the palm; and generating a control instruction corresponding to the robot according to the gesture information, and controlling the robot by using the control instruction. The present disclosure can improve the robot control accuracy and the convenience of robot control.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a robot control method and apparatus based on monocular vision gesture drive. Background Technology

[0002] According to some embodiments, the Rehabilitation Lower Extremity Exoskeleton (RLEEX) is a core device for assisting patients with lower limb dysfunction to restore gait and improve motor function. The comfort, reliability, and real-time performance of its human-robot interaction (HRI) directly affect the rehabilitation training effect and the patient's willingness to use it. For example, contact interaction can be achieved through mechanical or physical sensors, or interaction can be based on electromyography (EMG) signals. However, this approach suffers from complex operation, bulky equipment, and complex signal processing, resulting in poor accuracy in robot control. Summary of the Invention

[0003] This disclosure provides a robot control method and apparatus based on monocular vision gesture control, which can improve both the accuracy and convenience of robot control. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a robot control method based on monocular vision gesture drive is provided, comprising: Acquire the first palm image captured by the monocular camera; The YOLO hand region detection network is used to detect the first palm image to obtain the hand region image in the first palm image; A visual model is used to identify the hand region image and obtain the coordinates of each key hand point in the key hand point set; A gesture recognition model is used to identify the coordinates of each key point on the hand to obtain the gesture information corresponding to the palm. Based on the gesture information, control commands corresponding to the robot are generated, and the robot is controlled using the control commands.

[0005] According to some embodiments, the method further includes: Based on the camera parameters of the monocular camera and the image parameters corresponding to the first palm image, the first palm image is preprocessed to obtain the processed first palm image. The preprocessing includes denoising preprocessing, sharpening preprocessing, and redundant image cropping preprocessing.

[0006] According to some embodiments, the YOLO hand region detection network includes a backbone network, a neck feature fusion network, and a coupled detection head. The step of using the YOLO hand region detection network to detect the first palm image and obtain a hand region image in the first palm image includes: The first palm image is processed using the YOLO hand region detection network to obtain a second palm image that meets the detection requirements; The Backbone network is used to extract features from the second palm image to obtain a feature set; The Neck feature fusion network is used to perform multi-scale feature fusion processing on the feature set to obtain the fused feature set. The coupled detection head is used to identify the fused feature set to obtain the hand region image in the first palm image.

[0007] According to some embodiments, the step of using the coupled detection head to perform recognition processing on the fused feature set to obtain a hand region image in the first palm image includes: The coupled detection head is used to identify the fused feature set to obtain an information set corresponding to the hand target, wherein the information set includes the confidence score, category and bounding box coordinates of the hand target; If the information set satisfies the information requirements corresponding to the hand target, the hand region image in the first palm image is obtained.

[0008] According to some embodiments, the step of using a visual model to recognize the hand region image and obtaining the coordinates corresponding to each hand key point in the hand key point set includes: Obtain key point information corresponding to the palm; Based on the key point information, a visual model is used to identify the hand region image and obtain the coordinates corresponding to each hand key point in the hand key point set; Obtain the correspondence between the identifiers of each hand key point and the coordinates of each hand key point.

[0009] According to some embodiments, the step of using a gesture recognition model to identify the coordinates corresponding to each key point of the hand and obtaining the gesture information corresponding to the palm includes: A gesture recognition model is used to identify the coordinates of each key point on the hand, and the double-layer convex hull information corresponding to the palm is obtained. A gesture recognition model is used to identify the coordinates of each key hand point to obtain a set of 2D finger joint angles. Based on the coordinates of each key hand point, obtain the set of fingertip pixel distances corresponding to the set of fingertip key points; Based on the double-layer convex hull information, the set of 2D finger joint angles, and the set of fingertip pixel distances, the gesture information corresponding to the palm is obtained.

[0010] According to some embodiments, the method further includes: The visual model is adjusted using the coordinates of each hand key point in the set of hand key points to obtain the adjusted visual model.

[0011] According to a second aspect of the present disclosure, a robot control device based on monocular vision gesture drive is provided, comprising: The image acquisition unit is used to acquire the first palm image captured by the monocular camera; The image acquisition unit is further configured to use the YOLO hand region detection network to detect the first palm image and acquire the hand region image in the first palm image. The coordinate acquisition unit is used to identify the hand region image using a visual model and acquire the coordinates corresponding to each hand key point in the hand key point set; The information acquisition unit is used to identify the coordinates corresponding to each key point of the hand using a gesture recognition model, and to acquire the gesture information corresponding to the palm. A hand control unit is used to generate control commands corresponding to the robot based on the gesture information, and to control the robot using the control commands.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the robot control method based on monocular vision gesture drive as described in any of the preceding aspects.

[0013] According to a fourth aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the robot control method based on monocular vision gesture drive as described in any of the preceding aspects.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in any one of the preceding aspects.

[0015] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: In some or related embodiments, a first palm image is acquired by a monocular camera; the first palm image is detected using a YOLO hand region detection network to obtain a hand region image in the first palm image; a visual model is used to recognize the hand region image to obtain the coordinates corresponding to each hand key point in the hand key point set; a gesture recognition model is used to recognize the coordinates corresponding to each hand key point to obtain gesture information corresponding to the palm; control commands corresponding to the robot are generated based on the gesture information, and the robot is controlled using the control commands. Therefore, robot control can be achieved through hand images captured by a monocular camera, enabling non-contact interaction. This reduces the discomfort, bulky equipment, susceptibility to interference, and complex operation associated with contact methods. Robot control via upper limb gestures can meet the needs of patients with lower limb paralysis, enhancing the initiative and comfort of rehabilitation training and improving robot control convenience. Depth estimation and gesture recognition can be achieved using a monocular camera, reducing signal processing difficulty and equipment costs. This improves the scene adaptability of the embodiments of this disclosure, reduces misjudgment problems of depth cameras in close-range and sudden posture change scenarios, and improves both the accuracy and convenience of robot control.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0018] Figure 1 This is a flowchart of the first robot control method based on monocular vision gesture drive provided in the embodiments of this disclosure; Figure 2 This is a flowchart of a second robot control method based on monocular vision gesture drive provided in the embodiments of this disclosure; Figure 3 This is an example schematic diagram of a robot control system based on monocular vision gesture drive provided in an embodiment of this disclosure; Figure 4 This is an example schematic diagram of hand key points and double convex hull recognition provided by an embodiment of this disclosure; Figure 5 This is a flowchart of a robot control method based on monocular vision gesture drive provided in an embodiment of this disclosure; Figure 6This is a schematic diagram illustrating an example of the correspondence between gestures and gait movements provided in an embodiment of this disclosure; Figure 7 This is an example diagram illustrating a comparison of easily confused gestures provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of a monocular camera pinhole imaging ranging method provided in an embodiment of this disclosure; Figure 9 This is an example schematic diagram of an FSM gait transition model provided in an embodiment of this disclosure; Figure 10 This is a block diagram illustrating a robot control device based on monocular vision gesture drive according to an exemplary embodiment; Figure 11 This is an example schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0020] This disclosure provides a robot control method, apparatus, electronic device, and storage medium based on monocular vision gesture driving. In some embodiments, the terms "robot control method based on monocular vision gesture driving" can be used interchangeably with "information processing method" and "communication method," and the terms "robot control apparatus based on monocular vision gesture driving" can be used interchangeably with "information processing apparatus" and "communication apparatus," and the terms "information processing system" and "communication system" can be used interchangeably.

[0021] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0022] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0023] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0024] In this disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression.

[0025] In the embodiments of this disclosure, "multiple" refers to two or more.

[0026] In some embodiments, the terms “at least one of,” “one or more,” “a plurality of,” and “multiple” may be used interchangeably.

[0027] The prefixes "first," "second," etc., used in the embodiments of this disclosure are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, quantity, or content of the descriptive objects. The description of the descriptive objects is found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the number of descriptive objects is not limited by ordinal numbers and can be one or more. For example, in "first device," the number of "devices" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0028] In some embodiments, "terminal" or "terminal device" may be referred to as "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," etc.

[0029] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0030] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] According to some embodiments, the HRI method for RLEEX is mainly contact-based, including two types: one type is contact-based interaction based on mechanical sensors (IMU, angle encoder, plantar pressure sensor) or physical controllers (buttons, joysticks, watches), but it has problems such as complex operation, bulky equipment, poor comfort with long-term contact, and susceptibility to interference. Moreover, as the number of gait patterns increases, more physical interfaces are needed, increasing the user's operational burden; the other type is interaction based on bioelectrical signals such as electromyography (EMG) and electroencephalography (EEG). This type of method is easily interfered with by factors such as noise, sweat, and skin impedance, and has drawbacks such as large individual differences, high implantation risk, high cost, and complex signal processing, making it difficult to apply widely.

[0032] According to some embodiments, non-contact gesture control methods based on computer vision can be used, but such methods mostly rely on deep learning neural networks, which have problems such as low recognition reliability and easy false triggering in complex scenes (lighting changes, complex backgrounds). At the same time, most methods need to rely on depth cameras to achieve depth detection, which is costly and has poor real-time performance. In addition, depth cameras are prone to false judgments of depth values ​​of 0 in close-range interaction and sudden changes in hand posture, leading to control failure and safety hazards. Furthermore, existing gesture control methods lack effective triggering mechanisms and gait transition logic, resulting in insufficient gesture recognition accuracy and easy confusion with daily gestures, which cannot meet the precise control and safety requirements of RLEEX rehabilitation training.

[0033] The technical solution of this disclosure can provide a non-contact, low-cost, highly reliable, real-time, and easy-to-operate RLEEX interactive control method that is suitable for lower limb rehabilitation scenarios, thus solving the above-mentioned defects.

[0034] Figure 1 This is a flowchart of the first robot control method based on monocular vision gesture driving provided in the embodiments of this disclosure, such as... Figure 1 As shown, this robot control method based on monocular vision gesture actuation can be used in scenarios where a lower limb rehabilitation exoskeleton robot based on monocular vision gesture actuation is controlled, and includes the following steps: In step S11, the first palm image captured by the monocular camera is obtained; In some embodiments, the execution subject of this disclosure may be, for example, an electronic device. This electronic device does not specifically refer to a particular fixed electronic device. For example, when the device identifier changes, the electronic device may also change accordingly. For example, when the structure of the electronic device changes, the electronic device may also change accordingly. The execution subject of this disclosure may be, for example, a robot, and this disclosure does not limit the type of robot. Specifically, the robot may be a lower limb rehabilitation exoskeleton robot, for example, a non-contact lower limb rehabilitation exoskeleton robot.

[0035] In some embodiments, a monocular camera may be, for example, a camera that captures images using a webcam. The monocular camera does not specifically refer to a particular fixed camera. For example, the monocular camera may change accordingly when the parameters of the webcam change. For example, the monocular camera may change accordingly when the webcam's identifier changes.

[0036] According to some embodiments, the robot may be, for example, a robot to be controlled by a hand. The robot is not specific to any particular fixed robot. For example, the robot may change accordingly when its identification changes. For example, the robot may change accordingly when its structure changes.

[0037] According to some embodiments, the hand can be anyone's hand, specifically the hand used to control the robot. The hand is not specifically defined by any particular hand. For example, the hand can be the left hand or the right hand. For example, the hand can change accordingly when the size of the hand changes. For example, the hand can change accordingly when the structure of the hand changes.

[0038] In some embodiments, the first palm image may be, for example, an image directly captured by a monocular camera of a hand capable of controlling a robot. This first palm image is not specifically a fixed image. For example, the first palm image may change when the acquisition time changes. Similarly, the first palm image may change when its content changes.

[0039] In step S12, the YOLO hand region detection network is used to detect the first palm image to obtain the hand region image in the first palm image; According to some embodiments, the YOLO hand region detection network can also be referred to as the YOLO hand region detection model. This YOLO hand region detection network can be a trained model capable of detecting hand regions from palm images. The YOLO hand region detection network does not specifically refer to a single, fixed network. For example, when the network parameters of the YOLO hand region detection network change, the YOLO hand region detection network can also change accordingly. For example, when the network identifier of the YOLO hand region detection network changes, the YOLO hand region detection network can also change accordingly.

[0040] In some embodiments, the hand region image may be an image that includes only the robot's hand region. The specific hand region image is not limited. For example, when the background of the hand region image changes, the hand region image may also change accordingly. The background of the hand region image need not affect subsequent image processing; for example, it can be the background captured by a monocular camera, or it can be a pre-set background. This disclosure does not limit this. For example, when the first palm image changes, the hand region image may also change accordingly.

[0041] According to some embodiments, the YOLO hand region detection network can be used to detect the first palm image and obtain the hand region image in the first palm image.

[0042] In step S13, a visual model is used to identify the hand region image and obtain the coordinates of each hand key point in the hand key point set; According to some embodiments, the visual model may be, for example, a trained model that can be used for multi-task computation. This visual model is not specifically defined by a single fixed model. For example, the visual model may change when the model type changes. For example, the visual model may change when the architecture of the visual model changes. This visual model may also be referred to as an edge-side real-time computer vision model. This visual model may, for example, be a MediaPipe model.

[0043] According to some embodiments, the set of hand key points may be, for example, a collection of at least one hand key point. This set of hand key points does not specifically refer to a fixed set. For example, when the number of hand key points corresponding to the set changes, the set of hand key points may also change accordingly. For example, when determining the key point information to be acquired corresponding to the set of hand key points based on palm information, if the palm information changes, the set of hand key points may also change accordingly. The palm information may include, for example, palm size, palm clarity, etc. The key point information to be acquired may include, for example, key point location information and key point quantity information, etc. This disclosure does not limit this aspect.

[0044] In some embodiments, the coordinates corresponding to each hand keypoint can be used to represent the position of each hand keypoint in the hand region image. These coordinates do not specifically refer to any fixed coordinates. For example, when the method of obtaining the coordinates or the coordinate reference point changes, the coordinates corresponding to each hand keypoint can also change accordingly.

[0045] In some embodiments, a visual model can be used to identify hand region images and obtain the coordinates of each hand key point in the hand key point set.

[0046] In step S14, a gesture recognition model is used to identify the coordinates of each key point of the hand to obtain the gesture information corresponding to the palm. According to some embodiments, gesture information can be used, for example, to indicate the gesture corresponding to the palm in the current hand region image. This gesture information is not specific to any fixed information. For example, the gesture information can change accordingly when the acquisition time of the first palm image changes.

[0047] In some embodiments, a gesture recognition model can be used to identify the coordinates of key points on the hand and obtain the gesture information corresponding to the palm.

[0048] In step S15, control commands corresponding to the robot are generated based on the gesture information, and the robot is controlled using the control commands.

[0049] In some embodiments, the control command may be, for example, a command to control the palm. The control command is not specifically a single, fixed command. For example, it could be a command to control the palm, which could adjust the gait movements of the exoskeleton joints.

[0050] According to some embodiments, control commands corresponding to the robot can be generated based on gesture information, and the robot can be controlled using these control commands. Controlling the robot can, for example, involve executing exoskeleton movements corresponding to the gestures.

[0051] In some or related embodiments, a first palm image is acquired by a monocular camera; the first palm image is detected using a YOLO hand region detection network to obtain a hand region image in the first palm image; a visual model is used to recognize the hand region image to obtain the coordinates corresponding to each hand key point in the hand key point set; a gesture recognition model is used to recognize the coordinates corresponding to each hand key point to obtain the gesture information corresponding to the palm; control commands corresponding to the robot are generated based on the gesture information, and the robot is controlled using the control commands. Therefore, robot control can be achieved using hand images captured by a monocular camera, enabling non-contact interaction. This reduces the discomfort, bulky equipment, susceptibility to interference, and operational complexity associated with contact methods. Upper limb gestures allow for robot control, meeting the needs of lower limb paralysis patients, enhancing the initiative and comfort of rehabilitation training, and improving robot control convenience. Using a monocular camera enables depth estimation and gesture recognition, reducing signal processing complexity and equipment costs. It also improves the scene adaptability of this embodiment, reducing misjudgments by depth cameras in close-range or abrupt pose changes, thus improving both robot control accuracy and convenience. Furthermore, images captured by the monocular camera can be processed promptly, reducing system response time and improving recognition efficiency.

[0052] Figure 2 This is a flowchart of the second robot control method based on monocular vision gesture driving provided in the embodiments of this disclosure, as follows: Figure 2 As shown, it includes the following steps: In step S21, the first palm image captured by the monocular camera is obtained; The relevant processes can be as described above, and will not be repeated here.

[0053] According to some embodiments, the technical solutions of this disclosure can be used, for example, in contact scenarios of human-robot interaction (HRI) in lower limb rehabilitation exoskeleton (RLEEX) robots. Specifically, it can be a method for controlling non-contact lower limb rehabilitation exoskeleton robots based on RGB monocular cameras, gesture recognition, and augmented reality triggering, suitable for rehabilitation training of patients with lower limb paralysis and aphasia due to spinal cord injury, stroke, etc.

[0054] In some embodiments, Figure 3 This is an example schematic diagram of a robot control system based on monocular vision gesture drive provided in an embodiment of this disclosure, such as... Figure 3 As shown, it can include: an image preprocessing module, a hand keypoint detection and optimization module, a gesture recognition module, a monocular depth estimation and AR triggering module, and an FSM gait control module. Therefore, these modules work together to achieve closed-loop control of "gesture input - intent recognition - distance verification - action execution," eliminating the need for contactless interaction and enabling signal acquisition and control command generation solely through an RGB monocular camera. The image preprocessing module preprocesses images; the hand keypoint detection and optimization module detects and tracks keypoints; the gesture recognition module acquires gesture information; the monocular depth estimation and AR triggering module performs depth estimation and AI virtual button triggering; and the FSM gait control module can be used, for example, to control the robot's gait.

[0055] According to some embodiments, the monocular camera has an acquisition resolution of 1280×720; the image is preprocessed with Gaussian denoising (5×5 convolution kernel), histogram equalization and sharpening to weaken illumination and noise interference, and improve the clarity of subsequent target detection and key point extraction; grayscale and binarization are used to assist in screening the foreground area and initially crop redundant background to speed up YOLO hand target detection; the improved MediaPipe Hands algorithm's palm detection module is called to locate the palm area and crop out a refined 300×300 pixel palm image to remove background interference.

[0056] In step S22, the first palm image is preprocessed according to the camera parameters of the monocular camera and the image parameters corresponding to the first palm image to obtain the processed first palm image. The preprocessing includes noise reduction preprocessing, sharpening preprocessing and redundant image cropping preprocessing. The relevant processes can be as described above, and will not be repeated here.

[0057] According to some embodiments, denoising and sharpening the first hand image acquired by the RGB monocular camera can reduce data redundancy and environmental interference, and enhance the detectability of the hand region in the image. The hand detection module can locate the gesture region, crop redundant background, and obtain a refined gesture image, providing high-quality input for subsequent key point detection and improving detection speed and accuracy.

[0058] In step S23, the YOLO hand region detection network is used to detect the first palm image to obtain the hand region image in the first palm image; The relevant processes can be as described above, and will not be repeated here.

[0059] According to some embodiments, the YOLO hand region detection network can be the YOLO v5s lightweight object detection network.

[0060] According to some embodiments, the YOLO hand region detection network includes a backbone network, a neck feature fusion network, and a coupled detection head. The YOLO hand region detection network is used to detect the hand region in a first hand image, obtaining the hand region image within the first hand image, including: The YOLO hand region detection network is used to process the first palm image to obtain a second palm image that meets the detection requirements; The Backbone network is used to extract features from the second hand image to obtain a feature set; The Neck feature fusion network is used to perform multi-scale feature fusion processing on the feature set to obtain the fused feature set. A coupled detection head is used to process the fused feature set to obtain the hand region image from the first palm image. Therefore, it reduces the susceptibility of palm detection to background interference and positioning offset, improving the accuracy of palm region image acquisition and thus the accuracy of control command acquisition.

[0061] According to some embodiments, a coupled detection head is used to perform recognition processing on the fused feature set to obtain a hand region image in the first palm image, including: A coupled detection head is used to perform recognition processing on the fused feature set to obtain the information set corresponding to the hand target. The information set includes the confidence score, category and bounding box coordinates of the hand target. If the information set meets the information requirements corresponding to the hand target, obtain the hand region image from the first palm image.

[0062] According to some implementations, a lightweight version of YOLO v5s can be used for recognition, which can adapt to the embedded deployment requirements of rehabilitation exoskeletons. The network structure balances detection accuracy and real-time performance, and the overall structure consists of four parts: Input, Backbone network, Neck feature fusion network, and Head detection output. Input side: Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling are adopted to optimize the input scale for the palm target, and the input size is fixed at 416×416 to reduce computing power consumption.

[0063] Backbone network: It adopts the lightweight CSPDarknet53 structure, and realizes feature extraction through Focus slicing operation and C3 module. It enhances the shallow feature extraction capability for hand texture and contour features, and filters out invalid background features.

[0064] Neck Feature Fusion Network: It adopts an FPN+PAN structure to achieve multi-scale feature fusion, taking into account the detection accuracy of large and small palm targets, and solving the problem of palm localization at different distances and in different poses.

[0065] Head output: Using a coupled detection head, the confidence level, category (single category only: hand), and bounding box coordinates (xmin, ymin, xmax, ymax) of the hand target are output. Hand anchor boxes with a confidence level ≥ 0.7 are filtered out to eliminate interfering targets.

[0066] According to some embodiments, (1) the preprocessed RGB image is input into the YOLO v5s network, and the Focus layer slices the image into 4 low-resolution feature maps to reduce information loss and improve inference speed; (2) the Backbone extracts shallow contour and deep texture features of the hand, and the Neck fuses multi-scale features to enhance the expression of hand target features; (3) the Head layer predicts the hand bounding box and confidence, and removes overlapping boxes by non-maximum suppression (NMS) to retain the unique optimal hand anchor box; (4) the original image is cropped according to the anchor box coordinates to obtain a pure hand region image of 300×300 pixels, which completely eliminates background interference and provides a clean input for subsequent MediaPipe key point detection.

[0067] In step S24, a visual model is used to identify the hand region image and obtain the coordinates of each hand key point in the hand key point set; The relevant processes can be as described above, and will not be repeated here.

[0068] According to some embodiments, a visual model is used to identify hand region images and obtain the coordinates corresponding to each hand key point in the set of hand key points, including: Obtain key point information corresponding to the palm; Based on the key point information, a visual model is used to identify the hand region image and obtain the coordinates of each hand key point in the hand key point set; The system obtains the correspondence between the identifiers of each hand keypoint and their coordinates. This improves the accuracy of coordinate acquisition, gesture recognition, and robot control.

[0069] According to some embodiments, the visual model may be based on the MediaPipe Hands algorithm, which can detect and track 21 hand key points. The algorithm includes two core sub-modules: palm detection and key point estimation. First, the palm region is detected using the relative rigidity of the palm, and then the finger key points are tracked. If the palm confidence is lower than the threshold, the palm detection is reactivated to improve real-time performance.

[0070] In some embodiments, according to some embodiments, Figure 4 This is an example schematic diagram illustrating the recognition of hand key points and double convex hulls provided in an embodiment of this disclosure, such as... Figure 4 The diagram shows a double-layer recognition convex hull that can solve the problem of misrecognition of gestures caused by the small palm area in some embodiments. Virtual keypoint 21 (the midpoint between keypoints 5 and 17) is introduced as a palm anchor point, simplifying palm depth calculation and keypoint coordinate calibration, and improving the stability of subsequent depth estimation and gesture recognition. Figure 4 As shown, (a) is a key point heatmap, (b) is a MediaPipe key point skeleton model, and (c) is a schematic diagram of the inner convex hull, outer convex hull, virtual key point 21, and finger joint angle.

[0071] According to some embodiments, for example, it may be to store all key point coordinate data.

[0072] In some embodiments, the MediaPipe Hands algorithm may include three parts: palm keypoint heatmap prediction, keypoint coordinate regression, and keypoint tracking. Wherein: Heatmap prediction branch: Predicts heatmaps of 21 key points of hand joints using a lightweight convolutional neural network to locate the approximate positions of the key points; Coordinate regression branch: Based on the heatmap results, the sub-pixel coordinates of key points are refined to improve the positioning accuracy (≤4 pixels). Key point tracking branch: Kalman filtering is used to track key points between frames, avoiding key point loss caused by sudden attitude changes.

[0073] In some embodiments, a pure hand image cropped by YOLO is received, its size is normalized to 256×256, and then input into the MediaPipe keypoint network; (2) Output the pixel coordinates of 21 key points of the hand joints (fingertips, finger joints, metacarpals) and establish the correspondence between the key point numbers and the hand bones; (3) To address the issue of varying hand postures in rehabilitation scenarios, the key point tracking logic is optimized. When a key point is lost in a single frame, it is filled in by interpolating the coordinates of the key points in the previous three frames to ensure the continuity of the key points. (4) Based on the double convex hull rule of this disclosure, the connection relationship of key points is reconstructed, virtual key points 21 are generated, and key point optimization is completed.

[0074] According to some embodiments, the method further includes: The visual model is adjusted using the coordinates of each hand keypoint in the hand keypoint set to obtain the adjusted visual model. Therefore, the accuracy of visual model acquisition, keypoint recognition, and robot control can be improved.

[0075] In some embodiments, the MediaPipe Hands model can be fine-tuned a second time. The dataset uses pure hand cropped images trained with YOLO as described above, with 21 key point coordinates labeled (accuracy ≤ 4 pixels). During fine-tuning, the bottom feature extraction layer is frozen, and only the key point regression head is optimized to enhance the key point extraction accuracy under close-range palm and complex poses, thus solving the problem of poor adaptability of the original model to cropped hand images.

[0076] According to some embodiments, Figure 5 This is a flowchart of a robot control method based on monocular vision gesture driving provided in an embodiment of this disclosure, as shown below. Figure 5 As shown, a serial cascaded fusion structure can be adopted, without parallel branches, consisting of four serial units: "YOLO Hand Target Detection Module Unit → Hand Region Cropping Unit → MediaPipe Keypoint Detection Module Unit → Keypoint Optimization Unit". This improves the accuracy of hand localization and keypoint extraction, with unidirectional data flow, ensuring a simple process and efficient inference. Specifically, it can include: (1) After the original RGB image is preprocessed, it is input into the YOLO hand target detection module unit and the optimal bounding box of the hand is output. Therefore, YOLO front-end positioning shields the background interference, solves the problem of easy false detection of hand, and pure hand area input improves the key point extraction accuracy and reduces the key point loss rate under complex poses. (2) The cropping unit extracts the pure hand area from the original image based on the bounding box coordinates and removes redundant background information; (3) Pure hand images are sent to the MediaPipe key point detection module unit to extract 21 original joint key points. Therefore, YOLO v5s lightweight + MediaPipe lightweight inference reduces the overall single frame processing time and improves real-time performance. (4) The key point optimization unit constructs a double-layer convex hull, generates virtual key points 21, and outputs standardized key point data; (5) Standardized data is fed into the gesture recognition module, and gesture classification is completed by combining angle and distance constraints. Therefore, fine-tuning is performed for rehabilitation scenarios to adapt to the non-standard gesture postures of the upper limbs of patients with lower limb paralysis.

[0077] In step S25, a gesture recognition model is used to identify the coordinates of each key point on the hand to obtain the gesture information corresponding to the palm. The relevant processes can be as described above, and will not be repeated here.

[0078] According to some embodiments, a gesture recognition model is used to identify the coordinates corresponding to each key point of the hand, and to obtain the gesture information corresponding to the palm, including: A gesture recognition model is used to identify the coordinates of key points on the hand and obtain the double convex hull information of the palm. A gesture recognition model is used to identify the coordinates of key points on the hand and obtain a set of 2D joint angles of the fingers. Based on the coordinates of each key hand point, obtain the set of fingertip pixel distances corresponding to the set of fingertip key points; Based on the double-layer convex hull information, the set of 2D finger joint angles, and the set of fingertip pixel distances, the gesture information corresponding to the palm is obtained. Therefore, gesture confusion can be reduced, the accuracy of gesture information acquisition can be improved, the reliability of gesture recognition can be enhanced, and the accuracy of robot control can be increased.

[0079] According to some embodiments, based on key point data from embodiments of this disclosure, accurate recognition of 12 gestures is achieved through multi-constraint fusion. The core constraints include: 1. Convex hull constraint: Determine whether a preset number of fingertip key points are located in the inner convex hull, outer convex hull, or outside the convex hull, as the basis for gesture classification; 2. Angle Constraint: The 2D angle of the finger joint is calculated by formula θ=arccos[(v1·v2) / (∥v1∥·∥v2∥)], and the 2D bending angle thresholds of different fingers are set (thumb 53°, index finger 65°, little finger 49°, and other fingers 65°) to dynamically determine the bending state of the fingers and distinguish easily confused gestures. 3. Distance Constraint: The pixel distance between key points of the fingertip is calculated using the formula Ppi j=√[(u2-u1)²+(v2-v1)²]. A distance threshold is set for specific gestures (e.g., gesture a requires Pp48>100, gesture b requires Pp420>100). Combined with distance ratio constraints (e.g., gesture c requires Pp412 / Pp812<2), the recognition accuracy is further improved.

[0080] According to some embodiments, gesture recognition model training may include: based on the MSRA hand tracking and gesture dataset, dividing the training and test sets in an 8:2 ratio, labeling the coordinates of 21 key points, and training the gesture recognition model by combining double convex hull, angle threshold, and distance constraints; after training, inputting a preprocessed palm image into the model, outputting the gesture category, and determining a valid gesture when the recognition confidence is ≥0.8. For example, the trained model can achieve an average AP value of 95.67% for recognizing 12 gestures, effectively distinguishing easily confused gestures such as "tiger claw" and "OK". Different gestures can correspond to different RLEEX gait patterns. The correspondence between gestures and gait movements can be, for example, as shown in the diagram. Figure 6 As shown in the image. For example, a comparison chart of easily confused gestures can be shown in the image below. Figure 7 As shown.

[0081] In some embodiments, (1) Dataset construction: Based on the MSRA hand tracking dataset and the MSRA hand gesture dataset, 12 target gestures (0~9 digit gestures, "love" and "rock" symbol gestures) can be selected, and 1000 samples (covering different angles and postures) can be collected for each gesture. The samples are divided into training set, validation set and test set in a ratio of 8:1:1. Only the single category "hand" is set for annotation, and the bounding box is annotated using the LabelImg tool to meet the actual detection requirements.

[0082] (2) Training parameter settings: Training is based on PyTorch 1.7.0 framework; training parameters: batch size=16, initial learning rate lr=0.01, SGD optimizer, momentum=0.937, weight decay=0.0005, training epochs=150, cosine annealing learning rate scheduling strategy is adopted to avoid overfitting.

[0083] (3) Model fine-tuning and deployment: For near hand detection optimization in rehabilitation scenarios, the first 5 layers of the Backbone are frozen for transfer learning, and only the Neck and Head layers are fine-tuned; after training, the model volume is compressed using INT8 quantization and the model is deployed to the host computer to meet real-time requirements.

[0084] In some embodiments, the dual-layer recognition convex hull can be composed of key points 0, 1, 2, 5, 9, 13, and 17 for the inner convex hull and key points 0, 1, 2, 3, 6, 10, 14, 18, and 17 for the outer convex hull, and a virtual key point 21 (which may include the midpoint between key points 5 and 17) is introduced. Combined with the finger 2D joint angle threshold and fingertip pixel distance constraint, the gesture confusion problem is effectively solved and the recognition accuracy is improved.

[0085] According to some embodiments, the 2D joint angle of the fingers is calculated using formula (1), and different finger bending angle thresholds θ0 can be set: thumb 53°, index finger 65°, little finger 49°, middle finger and ring finger 65°. Define when θ When θ0, the fingers are bent; when θ>θ0, the fingers are open.

[0086] (1) In some embodiments, gestures 6, 7, and 8 may also require calculation of pixel distances between fingertip keypoints 4, 8, 12, and 20 for constraint purposes. Assume two distinct fingertip keypoints in the image... The pixel coordinates are P pi =[u1, v1]T,P pj =[u2, v2]T, then its pixel distance in the image is calculated as shown in formula (2): (2) In some embodiments, gesture 8 needs to satisfy pixel distance constraint P p48 >100; Gesture 6 must satisfy the pixel distance constraint P. p420 >100; Gesture 7 must meet the proportional constraints of the distances between 4, 12 and 8, 12, as shown in formula (3): (3) In step S26, control commands corresponding to the robot are generated based on the gesture information, and the robot is controlled using the control commands.

[0087] The relevant processes can be as described above, and will not be repeated here.

[0088] According to some embodiments, Figure 8 This is a schematic diagram of a monocular camera pinhole imaging ranging method provided in an embodiment of this disclosure, such as... Figure 8As shown, embodiments of this disclosure may further include depth estimation. For example, the distance between keypoints 5 and 17 can be used as the true anchor point size H. By inputting the user's gender and height, the corresponding palm width can be obtained from a preset adult human body size dataset, and H can be calculated. The pixel height h of keypoints 5 and 17 in the image can be obtained in real time. Combined with the camera focal length f, the vertical distance D between the palm and the camera can be calculated using the principle of similar triangles D=(f·H) / h. For example, a virtual button (located in the center of the screen, with a diameter of 50 pixels) can be drawn on the display interface. The palm bounding box can be calculated in real time. When D≤20cm and the virtual button is within the bounding box, the button turns green (triggered state) and a control command is generated; otherwise, the button turns red (released state) and no command is generated.

[0089] According to some embodiments, this disclosure may also include AR virtual button triggering: a virtual button activation distance threshold (20cm) can be set, and a palm bounding box (composed of the maximum / minimum coordinates of 22 key points) can be constructed. When D≤20cm and the virtual button is within the bounding box, the button is triggered (turns green); otherwise, it is released (turns red). Control commands are generated only when gesture recognition is effective and the button is triggered to avoid false triggering. Compared to depth cameras, monocular cameras are lower in cost, have better real-time performance (FPS≥31), an interaction range of 20-60cm, and an error of ≤5%, meeting the interaction requirements of lower limb rehabilitation exoskeletons while avoiding the false triggering problem of depth cameras in close-range, complex background scenes.

[0090] According to some embodiments, for example, using the distance between key points 5 and 17 as anchor points, and combining the Chinese adult body size dataset with the camera focal length, the vertical distance D between the palm and the camera is calculated using the principle of similar triangles; a distance threshold of 20cm is set, and when D≤20cm and the virtual button is located within the palm bounding box, the virtual button is triggered to generate a valid control command.

[0091] According to some embodiments, Figure 9 This is an example schematic diagram of an FSM gait transition model provided in an embodiment of this disclosure, such as... Figure 9 As shown, the states in this embodiment of the disclosure can include: not powered on, standing, sitting, right leg extended forward, left leg extended forward, right leg high step, right leg low step, and right leg obstacle crossing; 12 state transition rules correspond to 12 gestures, which can form effective control commands. Safe switching of RLEEX gait modes is achieved through FSM, and a three-frame majority voting mechanism and emergency stop function are introduced to avoid misoperation and safety risks. It can adapt to different rehabilitation training scenarios, and the algorithm has good real-time performance, meeting real-time control requirements and facilitating engineering applications. The transition logic can include: 1. Initialization and Start / Stop: Gesture 0 enables switching between power off → standing and standing → power off; 2. Switching between sitting and standing postures: The "love" gesture allows you to switch from standing to sitting, and the "rock" gesture allows you to switch from sitting to standing. 3. Walking control: Hand gesture 1 (stand → right leg forward), hand gesture 2 (right leg forward → left leg forward), hand gesture 3 (left leg forward → right leg forward), hand gesture 4 (right leg forward → stand), hand gesture 5 (stand → left leg starts continuous walking), hand gesture 6 (continuous walking → stand); 4. Special Gait Control: Gesture 7 (Stand → Right leg high step → Stand, climb stairs), Gesture 8 (Stand → Right leg low step → Stand, descend stairs), Gesture 9 (Stand → Right leg over obstacle → Stand). Therefore, gesture commands can be temporarily ignored during gait tasks to avoid misoperation; a three-frame majority voting mechanism is introduced, triggering a state transition only when the same gesture is recognized in three consecutive frames; an emergency stop button is provided to immediately terminate exoskeleton movements, ensuring user safety and preventing accidental triggering. This achieves safe and orderly switching of gait modes, ensuring the safety of patient rehabilitation training.

[0092] A block diagram of a robot control device based on monocular vision gesture actuation is shown according to an exemplary embodiment. Referring to 10, the device 1000 includes: The image acquisition unit 1001 is used to acquire the first palm image captured by the monocular camera; The image acquisition unit 1001 is also used to detect the first palm image using the YOLO hand region detection network to acquire the hand region image in the first palm image; The coordinate acquisition unit 1002 is used to identify the hand region image using a visual model and obtain the coordinates corresponding to each hand key point in the hand key point set; The information acquisition unit 1003 is used to identify the coordinates of each key point of the hand using a gesture recognition model, and to acquire the gesture information corresponding to the palm. The palm control unit 1004 is used to generate control commands corresponding to the robot based on gesture information, and to control the robot using the control commands.

[0093] According to some embodiments, the image acquisition unit 1001 is further specifically used for: Based on the camera parameters of the monocular camera and the image parameters corresponding to the first palm image, the first palm image is preprocessed to obtain the processed first palm image. The preprocessing includes noise reduction preprocessing, sharpening preprocessing, and redundant image cropping preprocessing.

[0094] According to some embodiments, the YOLO hand region detection network includes a backbone network, a neck feature fusion network, and a coupled detection head. The image acquisition unit 1001 is used to detect the hand region image in the first palm image using the YOLO hand region detection network. Specifically, when acquiring the hand region image in the first palm image, it is used for: The YOLO hand region detection network is used to process the first palm image to obtain a second palm image that meets the detection requirements; The Backbone network is used to extract features from the second hand image to obtain a feature set; The Neck feature fusion network is used to perform multi-scale feature fusion processing on the feature set to obtain the fused feature set. A coupled detection head is used to identify and process the fused feature set to obtain the hand region image in the first palm image.

[0095] According to some embodiments, the image acquisition unit 1001 is used to perform recognition processing on the fused feature set using a coupled detection head, and when acquiring the hand region image in the first palm image, it is specifically used for: A coupled detection head is used to perform recognition processing on the fused feature set to obtain the information set corresponding to the hand target. The information set includes the confidence score, category and bounding box coordinates of the hand target. If the information set meets the information requirements corresponding to the hand target, obtain the hand region image from the first palm image.

[0096] According to some embodiments, the coordinate acquisition unit 1002, when using a visual model to recognize a hand region image and acquire the coordinates corresponding to each hand key point in the hand key point set, is specifically used for: Obtain key point information corresponding to the palm; Based on the key point information, a visual model is used to identify the hand region image and obtain the coordinates of each hand key point in the hand key point set; Obtain the correspondence between the identifiers of each hand key point and the coordinates of each hand key point.

[0097] According to some embodiments, when the information acquisition unit 1003 is used to identify the coordinates corresponding to each key point of the hand using a gesture recognition model and to acquire the gesture information corresponding to the palm, it is specifically used for: A gesture recognition model is used to identify the coordinates of key points on the hand and obtain the double convex hull information of the palm. A gesture recognition model is used to identify the coordinates of key points on the hand and obtain a set of 2D joint angles of the fingers. Based on the coordinates of each key hand point, obtain the set of fingertip pixel distances corresponding to the set of fingertip key points; Based on the double convex hull information, the set of 2D finger joint angles, and the set of fingertip pixel distances, the gesture information corresponding to the palm is obtained.

[0098] According to some embodiments, the information acquisition unit 1003 is further specifically used for: The visual model is adjusted by using the coordinates of each hand keypoint in the hand keypoint set to obtain the adjusted visual model.

[0099] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0100] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0101] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0102] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as a monocular vision gesture-driven robot control method. For example, in some embodiments, the monocular vision gesture-driven robot control method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the monocular vision gesture-driven robot control method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform a robot control method based on monocular vision gestures.

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0109] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A robot control method based on monocular vision gesture drive, characterized in that, include: Acquire the first palm image captured by the monocular camera; The YOLO hand region detection network is used to detect the first palm image to obtain the hand region image in the first palm image; A visual model is used to identify the hand region image and obtain the coordinates of each key hand point in the key hand point set; A gesture recognition model is used to identify the coordinates of each key point on the hand to obtain the gesture information corresponding to the palm. Based on the gesture information, control commands corresponding to the robot are generated, and the robot is controlled using the control commands.

2. The method according to claim 1, characterized in that, The method further includes: Based on the camera parameters of the monocular camera and the image parameters corresponding to the first palm image, the first palm image is preprocessed to obtain the processed first palm image. The preprocessing includes denoising preprocessing, sharpening preprocessing, and redundant image cropping preprocessing.

3. The method according to claim 1, characterized in that, The YOLO hand region detection network includes a backbone network, a neck feature fusion network, and a coupled detection head. The step of using the YOLO hand region detection network to detect the first palm image and obtain the hand region image in the first palm image includes: The first palm image is processed using the YOLO hand region detection network to obtain a second palm image that meets the detection requirements; The Backbone network is used to extract features from the second palm image to obtain a feature set; The Neck feature fusion network is used to perform multi-scale feature fusion processing on the feature set to obtain the fused feature set. The coupled detection head is used to identify the fused feature set to obtain the hand region image in the first palm image.

4. The method according to claim 3, characterized in that, The step of using the coupled detection head to identify and process the fused feature set to obtain a hand region image in the first palm image includes: The coupled detection head is used to identify the fused feature set to obtain an information set corresponding to the hand target, wherein the information set includes the confidence score, category and bounding box coordinates of the hand target; If the information set satisfies the information requirements corresponding to the hand target, the hand region image in the first palm image is obtained.

5. The method according to claim 1, characterized in that, The step of using a visual model to recognize the hand region image and obtaining the coordinates corresponding to each hand key point in the hand key point set includes: Obtain key point information corresponding to the palm; Based on the key point information, a visual model is used to identify the hand region image and obtain the coordinates corresponding to each hand key point in the hand key point set; Obtain the correspondence between the identifiers of each hand key point and the coordinates of each hand key point.

6. The method according to claim 1, characterized in that, The step of using a gesture recognition model to identify the coordinates corresponding to each key point of the hand and obtaining the gesture information corresponding to the palm includes: A gesture recognition model is used to identify the coordinates of each key point on the hand, and the double-layer convex hull information corresponding to the palm is obtained. A gesture recognition model is used to identify the coordinates of each key hand point to obtain a set of 2D finger joint angles. Based on the coordinates of each key hand point, obtain the set of fingertip pixel distances corresponding to the set of fingertip key points; Based on the double-layer convex hull information, the set of 2D finger joint angles, and the set of fingertip pixel distances, the gesture information corresponding to the palm is obtained.

7. The method according to claim 1, characterized in that, The method further includes: The visual model is adjusted using the coordinates of each hand key point in the set of hand key points to obtain the adjusted visual model.

8. A robot control device based on monocular vision gesture actuation, characterized in that, include: The image acquisition unit is used to acquire the first palm image captured by the monocular camera; The image acquisition unit is further configured to use the YOLO hand region detection network to detect the first palm image and acquire the hand region image in the first palm image. The coordinate acquisition unit is used to identify the hand region image using a visual model and acquire the coordinates corresponding to each hand key point in the hand key point set; The information acquisition unit is used to identify the coordinates corresponding to each key point of the hand using a gesture recognition model, and to acquire the gesture information corresponding to the palm. A hand control unit is used to generate control commands corresponding to the robot based on the gesture information, and to control the robot using the control commands.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the robot control method based on monocular vision gesture drive as described in any one of claims 1 to 7.

10. A storage medium storing instructions, characterized in that, When the instructions are executed on the electronic device, the electronic device performs the robot control method based on monocular vision gesture drive as described in any one of claims 1 to 7.

Images