Gesture Recognition-Based Detection Robot Debugging System and Method

Abstract

This application discloses a gesture recognition-based robot debugging system and method. The system includes: a programmable logic controller (PLC) for controlling a display module to show a prompt interface for prompting the user to make gestures; a gesture recognition module for performing gesture recognition based on the collected image data stream of the user's gestures to obtain gesture commands; the PLC for determining the target motion parameters of the robot body based on the gesture commands, so that the robot body moves under the target motion parameters; a laser rangefinder for generating warning information during the robot body's movement; and the PLC for controlling the robot body and the display module to display the warning information based on the warning information until the robot body aligns with the target detection point. Using the embodiments of this application can significantly reduce operation steps and time, significantly improve debugging efficiency, meet high-precision detection requirements, and reduce collision risks.

Description

Technical Field

[0001] This application relates to the field of engine testing technology, and in particular to a testing robot debugging system and method based on gesture recognition. Background Technology

[0002] Engine visual inspection typically requires high-precision inspection of external engine components (such as fuel connectors and sensors) to identify potential defects or assembly problems. This process takes place on a conveyor belt in an automated production line, where engines pass sequentially through inspection stations, and robots perform the visual inspection tasks.

[0003] Currently, engine appearance inspection robots mainly employ a teach pendant-based debugging method. This method uses the teach pendant to select the robot's motion mode, choose the target axis, move the robot, and record waypoints.

[0004] However, relying on teach pendant buttons for the entire operation results in low efficiency due to the need for multiple button presses to debug a single detection point. Furthermore, the frequent switching of viewpoints (teach pendant screen → robot → engine) during debugging leads to distraction, resulting in recording errors of ±2mm in scenarios with dense detection points (spacing ≤10mm), failing to meet high-precision detection requirements. Finally, current safety protection relies primarily on manual emergency stop buttons, which have significant response delays. Additionally, the robot speed requires multi-level adjustment via the teach pendant, which cannot adapt to changes in spacing in real time, leading to a high risk of collisions in close-range operation scenarios. Summary of the Invention

[0005] This application provides a gesture recognition-based detection robot debugging system and method. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0006] In a first aspect, embodiments of this application provide a gesture recognition-based detection robot debugging system, the system comprising:

[0007] The system includes a gesture recognition module, a control module, and a display module. The control module is communicatively connected to both the gesture recognition module and the display module. The control module comprises a programmable logic controller (PLC) and a robot body for engine exterior inspection, with a laser rangefinder mounted at the end effector of the robot body.

[0008] A programmable logic controller is used to control the display module to show a prompt interface for prompting the user to make gestures during the process of aligning the robot body with the target detection point of the engine.

[0009] The gesture recognition module is used to recognize gestures based on the image data stream of user gestures, obtain gesture commands, and send them to the programmable logic controller.

[0010] The programmable logic controller is also used to determine the target motion parameters of the robot body according to the gesture instructions, so that the robot body can move under the target motion parameters;

[0011] A laser rangefinder is used to generate early warning information during the robot's movement and send it to a programmable logic controller.

[0012] The programmable logic controller (PLC) is also used to control the robot body and the display module to show the warning information based on the warning information until the robot body is aligned with the target detection point.

[0013] Optionally, the gesture recognition module includes a depth camera, a composite light source auxiliary component, and an industrial control computer; the depth camera is electrically connected to the industrial control computer.

[0014] Composite light source auxiliary components are used to create a completely shadowless environment for the engines on the plate chain conveyor equipment of the production line;

[0015] A depth camera is used to capture continuous image frames of user gestures, obtain an image data stream, and send it to an industrial control computer.

[0016] An industrial control computer is used to perform gesture recognition based on image data streams, obtain gesture commands, and send them to a programmable logic controller.

[0017] Optionally, gesture recognition is performed based on the image data stream to obtain gesture commands, including:

[0018] Preprocess the image data stream to obtain a three-dimensional feature matrix of hand key points composed of multiple frames of images within a single time window;

[0019] Load a pre-trained gesture parameter recognition model; the gesture parameter recognition model includes a 3D convolutional network and a discriminative network;

[0020] The three-dimensional feature matrix of hand key points is input into a 3D convolutional network for spatial convolution to extract the hand gesture pose features and temporal features of hand movements in RGB-D image frames;

[0021] The gesture posture features and the temporal features of the gesture actions are input into the discriminative network to obtain the gesture category and the user's wrist rotation angle.

[0022] The wrist rotation angle is mapped to the robot's movement speed;

[0023] Use gesture type and movement speed as gesture commands.

[0024] Optionally, the image data stream is preprocessed to obtain a 3D feature matrix of hand key points composed of multiple frames within a single time window, including:

[0025] The RGB-D image frames with timestamps in the image data stream are segmented and removed from the background using a preset depth information threshold to obtain the user's hand foreground image;

[0026] Morphological opening operations are performed on the hand foreground image to remove image noise, resulting in a denoised hand foreground image.

[0027] The denoised hand foreground image is scaled down to a standard size of 224×224 pixels to obtain the scaled hand foreground image;

[0028] From the scaled foreground image of the hand, the three-dimensional coordinates of multiple hand key points are extracted using a preset hand key point detection algorithm, and the three-dimensional coordinates of multiple hand key points in the RGB-D image frame are obtained.

[0029] The three-dimensional coordinates of multiple hand key points in all RGB-D image frames within a single preset time window are combined to obtain a three-dimensional feature matrix of hand key points composed of multiple frames within a single time window.

[0030] Optionally, the prompt interface can be a prompt interface for selecting the motion mode or target axis, or a prompt interface for displaying robot status information; the gesture commands can be gesture type and motion speed.

[0031] Based on the gesture commands, determine the target motion parameters of the robot body, including:

[0032] When the prompt interface displays a prompt for the user to select a sports mode, the corresponding final sports mode is obtained from a pre-built mapping relationship between gesture categories and sports modes, based on the gesture category; or,

[0033] When the prompt interface displays a prompt for the user to select a target axis, the corresponding final target axis is obtained from a pre-built mapping relationship between gesture categories and target axes, based on the gesture category; or,

[0034] When the robot's status information is displayed on the prompt interface, the corresponding motion direction is obtained from the pre-built mapping relationship between the gesture category and the robot's motion direction according to the gesture category; the motion speed and motion direction are used as the final motion parameters.

[0035] The final motion pattern, final target axis, and final motion parameters are used as the target motion parameters of the robot.

[0036] Optionally, the control module may also include a robot control cabinet;

[0037] To enable the robot body to move under the final motion mode, final target axis, and final motion parameters, including:

[0038] Set the state of the final target axis to the ready-to-move state so that the robot body enters the ready-to-move state of the final target axis;

[0039] The final motion parameters are written into the designated register of the robot control cabinet so that the robot body moves according to the final motion parameters.

[0040] Optionally, during the robot's movement, warning information is generated, including:

[0041] During the robot's movement, the distance between it and the engine is acquired in real time;

[0042] When the distance is less than a preset safety threshold, a short-range signal for early warning is generated;

[0043] Encapsulate short-range signals used for early warning into early warning information.

[0044] Optionally, the display module includes a teach pendant;

[0045] Based on the warning information, control the robot body and the display module to show the warning information, including:

[0046] Upon receiving a nearby signal, the robot's movement speed is adjusted to half of its original speed to obtain the current speed.

[0047] Based on the proximity signal and current speed, a prompt message is generated and sent to the teach pendant so that the teach pendant can display the prompt message.

[0048] Optionally, the gesture recognition module is also used to generate a recording instruction and send it to the programmable logic controller when the robot body is aligned with the target detection point and the gesture instruction indicates the recording operation.

[0049] The programmable logic controller is also used to respond to the recording command, obtain the spatial coordinates and joint angle coordinates of the robot body, and store them in a preset specified storage area;

[0050] Once all engine detection points have been recorded, the trajectory waypoint dataset is obtained.

[0051] The trajectory optimization algorithm is called to process the trajectory waypoint dataset to eliminate path abrupt changes and generate an executable program that conforms to the joint acceleration constraints of the robot body.

[0052] Store executable programs so that the robot body can be used in automated inspection tasks.

[0053] Secondly, a gesture recognition-based debugging method for a detection robot, applied to a programmable logic controller, includes the following:

[0054] During the process of aligning the robot body with the target detection point of the engine, the control display module displays a prompt interface to prompt the user to make gestures;

[0055] Receive gesture commands sent by the gesture recognition module, which are obtained by recognizing gestures from the image data stream of the user's gestures.

[0056] Based on the gesture command, the target motion parameters of the robot body are determined so that the robot body can move under the target motion parameters;

[0057] Receive early warning information generated during the robot's movement from the laser rangefinder;

[0058] Based on the warning information, control the robot body and control the display module to display the warning information until the robot body is aligned with the target detection point.

[0059] In this embodiment, on the one hand, the user's gestures are directly collected and converted into commands by the gesture recognition module, eliminating the need for multiple button presses. The programmable logic controller (PLC) can receive gesture commands in real time and control the robot's movement, greatly reducing operation steps and time, and significantly improving debugging efficiency. On the other hand, by directly collecting user gestures through the gesture recognition module, the user does not need to frequently switch their gaze, allowing them to focus on gesture operations. Simultaneously, the gesture recognition module can accurately capture gesture movements and convert them into accurate motion parameters, significantly reducing recording errors in dense detection point scenarios and meeting high-precision detection requirements. Furthermore, by monitoring the distance between the robot and the target detection point, a warning message is generated and sent to the PLC when the distance is too close. The PLC can immediately adjust the robot's speed or stop its movement based on the warning message and display the warning message through the display module, reminding the user to pay attention to safety, thereby reducing the risk of collision and improving safety in close-range operation scenarios.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1This is a schematic diagram of the system structure of a gesture recognition-based detection robot debugging system provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of a plate chain conveyor for an automated production line provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram illustrating the mapping relationship between gesture categories and motion patterns provided in an embodiment of this application.

[0065] Figure 4 This application provides a mapping diagram for representing the mapping relationship between a pre-constructed gesture category and a target axis.

[0066] Figure 5 This application provides a mapping diagram for representing the mapping relationship between pre-built gesture categories and robot movement directions;

[0067] Figure 6 This is a schematic diagram illustrating the process of a laser measuring instrument performing a distance warning according to an embodiment of this application;

[0068] Figure 7 This is a system architecture diagram of a gesture recognition-based detection robot debugging system provided in an embodiment of this application;

[0069] Figure 8 This is a schematic flowchart of a gesture recognition-based robot debugging method provided in an embodiment of this application;

[0070] Figure 9 This is a schematic diagram of a control process for controlling the movement of a robot in an emergency, provided in an embodiment of this application.

[0071] Figure 10 This is a schematic block diagram illustrating the debugging process of a detection robot based on gesture recognition, as provided in an embodiment of this application.

[0072] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0073] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them.

[0074] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0075] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0076] In this embodiment, on the one hand, the user's gestures are directly collected and converted into instructions by the gesture recognition module, eliminating the need for multiple button presses. The programmable logic controller (PLC) can receive gesture instructions in real time and control the robot's movement, greatly reducing operation steps and time, and significantly improving debugging efficiency. On the other hand, by directly collecting user gestures through the gesture recognition module, the user does not need to frequently switch their gaze, allowing them to focus on gesture operations. Simultaneously, the gesture recognition module can accurately capture gesture movements and convert them into accurate motion parameters, significantly reducing recording errors in dense detection point scenarios and meeting high-precision detection requirements. Furthermore, by monitoring the distance between the robot and the target detection point, a warning message is generated and sent to the PLC when the distance is too close. The PLC can immediately adjust the robot's speed or stop its movement based on the warning message and display the warning message through the display module, reminding the user to pay attention to safety, thereby reducing the risk of collision and improving safety in close-range operation scenarios. The following describes the process in detail using exemplary embodiments.

[0077] Please see Figure 1 , Figure 1 This is a schematic diagram of the system structure of a gesture recognition-based inspection robot debugging system provided in an embodiment of this application. The system includes: a gesture recognition module, a control module, and a display module; the control module is communicatively connected to the gesture recognition module and the display module respectively; the control module includes a programmable logic controller and a robot body for engine appearance inspection, with a laser rangefinder mounted at the end of the robot body.

[0078] Specifically, the gesture recognition module includes a depth camera, a composite light source auxiliary component, and an industrial control computer, with the depth camera electrically connected to the industrial control computer. The control module also includes a robot control cabinet, which is communicatively connected to a programmable logic controller (PLC).

[0079] The depth camera is installed in a spatial location capable of fully capturing the 3D coordinates and temporal changes of the hand. The composite light source auxiliary component uses a composite light source to ensure that the error in the uniformity of illumination in the detection space is ≤5%, creating a shadowless environment throughout. The programmable logic controller (PLC) is an industrial-grade PLC, serving as the communication hub for the entire system. It interacts with the industrial control computer to receive gesture commands and communicates with the robot control cabinet, ensuring low latency in "gesture recognition + communication." The robot body is a six-axis collaborative robot. The robot control cabinet automatically switches to teach pendant control mode when the command delay exceeds a certain duration to avoid motion deviation. The laser rangefinder continuously monitors the distance between its end effector and the motor; if the distance is too close, it immediately sends a "near-distance signal" to the PLC.

[0080] The display module is a teach pendant for an industrial touchscreen, which communicates with the programmable logic controller (PLC). The teach pendant's display function can show real-time status information such as "motion mode, currently selected axis, robot coordinates, and recorded waypoints." The teach pendant's interactive function supports switching between "gesture control" and "teach pendant control." If a gesture recognition error occurs (occlusion, timeout, misoperation) or the laser detector is nearby, the teach pendant triggers an emergency pause command, which, in turn, freezes the robot's movement and triggers an audible and visual alarm via the PLC.

[0081] For example Figure 2 As shown, Figure 2 This application provides a plate chain conveyor for an automated production line. The top light source is emitted by a composite light source auxiliary component. A depth camera, a collaborative robot, and a laser rangefinder mounted on the end of the robot body are all installed in this device.

[0082] In some embodiments of this application, a programmable logic controller (PLC) is used to control a display module to display a prompt interface for prompting the user to make a gesture during the process of aligning the robot body with the target detection point of the engine; a gesture recognition module is used to perform gesture recognition based on the image data stream of the user's gesture, obtain gesture instructions, and send them to the PLC; the PLC is also used to determine the target motion parameters of the robot body according to the gesture instructions, so that the robot body moves under the target motion parameters; a laser rangefinder is used to generate warning information during the movement of the robot body and send it to the PLC; the PLC is also used to control the robot body and control the display module to display the warning information according to the warning information until the robot body aligns with the target detection point.

[0083] The prompt interface is either a prompt interface for selecting a motion mode or target axis, or a prompt interface for displaying robot status information.

[0084] It should be noted that during the robot's alignment with the engine's target detection point, the following steps are taken: first, a prompt interface for selecting the motion mode guides the user; then, a prompt interface for selecting the target axis guides the user; finally, a robot status information prompt interface guides the user. If the robot's alignment with the engine's target detection point is in its initial state, the prompt interface for selecting the motion mode should be displayed. If the user has already selected a motion mode, a prompt interface for selecting the target axis should be displayed. If the user has already selected the target axis, a robot status information prompt interface should be displayed to remind the user to control the robot's movement using gestures.

[0085] In some embodiments of this application, the gesture recognition module includes a depth camera, a composite light source auxiliary component, and an industrial control computer; the depth camera is electrically connected to the industrial control computer; the composite light source auxiliary component is used to establish a shadowless environment for the engine on the plate chain conveyor of the production line; the depth camera is used to acquire continuous image frames of user gestures, obtain an image data stream, and send it to the industrial control computer; the industrial control computer is used to perform gesture recognition based on the image data stream, obtain gesture commands, and send them to the programmable logic controller.

[0086] In some embodiments of this application, the specific process of performing gesture recognition on the image data stream to obtain gesture commands includes: preprocessing the image data stream to obtain a three-dimensional feature matrix of hand key points composed of multiple frames of images within a single time window; loading a pre-trained gesture parameter recognition model; the gesture parameter recognition model includes a 3D convolutional network and a discriminant network; inputting the three-dimensional feature matrix of hand key points into the 3D convolutional network for spatial convolution to extract the gesture posture features and temporal features of the hand in the RGB-D image frames; inputting the gesture posture features and temporal features of the gesture actions into the discriminant network to obtain the gesture category and the user's wrist rotation angle; mapping the wrist rotation angle to the movement speed of the robot body; and using the gesture category and movement speed as gesture commands.

[0087] For example, with the wrist as the origin and the palm facing as the reference, the wrist rotation angle is linearly mapped to the movement speed, and the wrist returning to center (≤5°) corresponds to a speed of 0.

[0088] Specifically, the process of preprocessing the image data stream to obtain a three-dimensional feature matrix of hand key points composed of multiple frames within a single time window includes: segmenting and removing the background of the RGB-D image frames with timestamps in the image data stream using a preset depth information threshold to obtain the user's hand foreground image; performing morphological opening operations on the hand foreground image to remove image noise, resulting in a denoised hand foreground image; scaling the denoised hand foreground image to a standard size of 224×224 pixels, resulting in a scaled hand foreground image; extracting the three-dimensional coordinates of multiple hand key points from the scaled hand foreground image using a preset hand key point detection algorithm, resulting in the three-dimensional coordinates of multiple hand key points in the RGB-D image frames; and combining the three-dimensional coordinates of multiple hand key points from all RGB-D image frames within a single preset time window to obtain a three-dimensional feature matrix of hand key points composed of multiple frames within a single time window.

[0089] Among them, the three-dimensional feature matrix of hand key points is a feature matrix of "number of frames × 21 key points × 3D coordinates".

[0090] Specifically, the pre-trained gesture parameter recognition model is trained using massive amounts of scene data. Data augmentation techniques are combined to improve the algorithm's robustness, ensuring high recognition accuracy in complex industrial environments. Based on optimizers and loss functions, multiple rounds of training are conducted, and data augmentation methods such as gesture rotation, scaling, and occlusion are used to enhance model robustness, ensuring the final model can support robot control with a single frame time ≤15ms. The specific model training process is existing technology and not protected by this application; therefore, it will not be elaborated here.

[0091] The prompt interface is for selecting a motion mode or target axis, or for displaying robot status information; the gesture commands are for gesture type and motion speed.

[0092] In some embodiments of this application, the specific process of determining the target motion parameters of the robot body according to gesture instructions includes: when a prompt interface is displayed to prompt the user to select a motion mode, obtaining the corresponding final motion mode from a pre-built mapping relationship between gesture categories and motion modes according to the gesture category; or, when a prompt interface is displayed to prompt the user to select a target axis, obtaining the corresponding final target axis from a pre-built mapping relationship between gesture categories and target axes according to the gesture category; or, when a prompt interface displays robot state information, obtaining the corresponding motion direction from a pre-built mapping relationship between gesture categories and robot motion directions according to the gesture category; using motion speed and motion direction as final motion parameters; and using the final motion mode, final target axis, and final motion parameters as the robot's target motion parameters.

[0093] Among them, the relationship diagram used to represent the mapping relationship between pre-constructed gesture categories and motion patterns is, for example... Figure 3 As shown, the user needs to select a motion mode in the robot control system. The "Motion Mode Selection" node branches into four sub-nodes, each corresponding to a specific gesture. These gestures are used to select different motion modes: "One-hand gesture 1": This gesture triggers the selection of "Joint Motion Mode". "One-hand gesture 2": This gesture triggers the selection of "Linear Motion Mode". "One-hand gesture 3": This gesture triggers the "Switch to Next Robot" operation. "Thumb horizontal": This gesture triggers the "Exit Gesture Control" operation, which may be used to exit the current gesture control mode or end the current operation process.

[0094] Among them, the relationship diagram used to characterize the mapping relationship between the pre-constructed gesture categories and the target axis is, for example... Figure 4 As shown, the user needs to select a control axis in the robot control system. The "Axis Selection" node branches into three sub-nodes, each corresponding to a specific gesture. These gestures are used to select different axes or motion modes: "One-hand gesture 1-6" (Joint Mode): This gesture triggers the selection of axes 1 through 6 of the robot, i.e., joint mode. In this mode, the user can control the independent movement of each joint of the robot. "One-hand gesture 1-3" (Linear Mode): This gesture triggers the selection of the robot's X, Y, and Z axes, i.e., linear mode. In this mode, the robot's end effector moves along a straight path, suitable for tasks requiring precise linear movement. "Thumbs up": This gesture triggers the "Return to Previous" operation, which may be used to exit the current axis selection mode or return to the previous operation sequence.

[0095] Among them, the relationship diagram used to represent the mapping relationship between the pre-constructed gesture categories and the robot's motion direction is as follows: Figure 5 As shown, users control the robot's movement through specific gestures (grasping gestures). The "Motion Control" node branches into three sub-nodes, each corresponding to a specific gesture used to control different robot movements. Right hand clockwise rotation 0-90°: This gesture triggers the robot to move in the positive direction or rotate a joint clockwise. The speed can be 0-1° / s (joint) or 0-0.3mm / s (linear). Right hand counterclockwise rotation 0-90°: This gesture triggers the robot to move in the negative direction or rotate a joint counterclockwise. The speed can also be 0-1° / s (joint) or 0-0.3mm / s (linear). Wrist return (≤5°): This gesture triggers the robot to stop moving, and the speed drops to 0.

[0096] The control module also includes a robot control cabinet.

[0097] In some embodiments of this application, the specific process of enabling the robot body to move under the final motion mode, the final target axis, and the final motion parameters includes: setting the state of the final target axis to a ready-to-move state so that the robot body enters the ready-to-move state of the final target axis; and writing the final motion parameters into a designated register of the robot control cabinet so that the robot body moves according to the final motion parameters.

[0098] In some embodiments of this application, the specific process of generating warning information during the movement of the robot body includes: acquiring the distance between the robot body and the engine in real time during the movement of the robot body; generating a near-field signal for warning when the distance is less than a preset safety threshold; and encapsulating the near-field signal for warning into warning information.

[0099] The display module includes a teach pendant.

[0100] In some embodiments of this application, the specific process of controlling the robot body and the display module to display the warning information according to the warning information includes: when a near-field signal is received, adjusting the movement speed of the robot body to half of the original movement speed to obtain the current speed; generating a prompt message based on the near-field signal and the current speed, and sending it to the teach pendant so that the teach pendant can display the prompt message.

[0101] In one possible implementation, after the programmable logic controller receives the "near-distance signal" measured by the laser rangefinder, it automatically sends a "speed × 0.5" command to the robot body. At this time, the control teach pendant simultaneously pops up a "distance too close, speed has been halved" prompt message.

[0102] For example Figure 6 As shown, the distance data measured by the laser measuring instrument is used to determine whether the distance is less than or equal to 10 centimeters (≤10cm). This is a critical safety threshold used to decide whether the robot's speed needs to be reduced. If the measured distance is less than or equal to 10 centimeters, it indicates that the distance between the robot's end effector and the motor is too close, posing a collision risk. If the measured distance is greater than 10 centimeters, it indicates that the distance between the robot's end effector and the motor is safe, and the robot can continue to move at its normal speed. When a close proximity is detected, the PLC (Programmable Logic Controller) receives a "close proximity signal" and generates an instruction to reduce the robot's speed to half its original speed (speed × 0.5) to reduce the risk of collision. If the distance is safe, the robot continues to move at its normal speed.

[0103] In some embodiments of this application, the gesture recognition module is further configured to generate a recording instruction and send it to the programmable logic controller (PLC) when the robot body is aligned with the target detection point and the gesture instruction indicates a recording operation; the PLC is further configured to, in response to the recording instruction, acquire the spatial coordinates and joint angle coordinates of the robot body and store them in a preset designated storage area; after all detection points of the engine have been recorded, a trajectory waypoint dataset is obtained; a trajectory optimization algorithm is called to process the trajectory waypoint dataset to eliminate path abrupt changes and generate an executable program that conforms to the joint acceleration constraints of the robot body; and the executable program is stored so that the robot body can use it in automatic detection tasks.

[0104] For example Figure 7 As shown, Figure 7 This application presents a system architecture diagram for a gesture recognition-based inspection robot debugging system, comprising three parts: hardware perception, algorithm recognition, and robot control. Hardware perception includes a composite light source (dual-sided + top-surface matrix) provided by a composite light source auxiliary component, a depth camera, an industrial control computer, and a laser measuring instrument (laser rangefinder) mounted on the end effector of the robot. The algorithm recognition part includes a data preprocessing module, a gesture recognition module, and a command mapping module. The robot control part includes a PLC (Programmable Logic Controller), a robot control cabinet, a six-axis collaborative robot, and a teach pendant. In this system, the RGB-D camera and the laser measuring instrument acquire image data and distance data, respectively. The industrial control computer processes the image data and extracts useful information. The algorithm recognition module analyzes the image data, recognizes the user's gestures, and maps them to control commands. The PLC receives the control commands and sends them to the six-axis collaborative robot through the robot control cabinet to control its movement. The teach pendant displays the robot's current status and control commands, allowing the user to monitor and adjust the robot's behavior.

[0105] In this embodiment, on the one hand, the user's gestures are directly collected and converted into commands by the gesture recognition module, eliminating the need for multiple button presses. The programmable logic controller (PLC) can receive gesture commands in real time and control the robot's movement, greatly reducing operation steps and time, and significantly improving debugging efficiency. On the other hand, by directly collecting user gestures through the gesture recognition module, the user does not need to frequently switch their gaze, allowing them to focus on gesture operations. Simultaneously, the gesture recognition module can accurately capture gesture movements and convert them into accurate motion parameters, significantly reducing recording errors in dense detection point scenarios and meeting high-precision detection requirements. Furthermore, by monitoring the distance between the robot and the target detection point, a warning message is generated and sent to the PLC when the distance is too close. The PLC can immediately adjust the robot's speed or stop its movement based on the warning message and display the warning message through the display module, reminding the user to pay attention to safety, thereby reducing the risk of collision and improving safety in close-range operation scenarios.

[0106] Please see Figure 8 This document provides a flowchart illustrating a gesture recognition-based robot debugging method, applicable to a programmable logic controller (PLC). Figure 8 As shown, the detection method in this application embodiment may include the following steps:

[0107] S101, During the process of aligning the robot body with the target detection point of the engine, the control display module displays a prompt interface to prompt the user to make gestures;

[0108] In some embodiments of this application, for example, if the current target detection point is a new model of engine, and a point recording operation needs to be performed on the engine, the system is started. The programmable logic controller (PLC) responds to the user's start signal and sends initialization instructions to the teach pendant, the industrial control computer deployed in the detection environment, the robot control cabinet, and the laser rangefinder. After successful initialization, the user needs to align each target detection point of the engine. Since it is a new point recording operation, the PLC controls the teach pendant to display an interface to prompt the user to select a motion mode. That is, the prompt interface for prompting the user to make gestures is the prompt interface for selecting a motion mode.

[0109] At this point, the user selects a motion mode via gestures (e.g., a "one-hand gesture" to select joint movement). The depth camera captures the user's gesture image in real time, and the image is preprocessed and subjected to spatiotemporal convolution gesture recognition by the industrial control computer. The industrial control computer encodes the recognized gesture information as gesture commands and sends them to the PLC.

[0110] S102, receive gesture instructions sent by the gesture recognition module, which are obtained by performing gesture recognition based on the image data stream of the user's gestures.

[0111] In some embodiments of this application, the PLC receives the gesture command.

[0112] S103, Based on the gesture command, determine the target motion parameters of the robot body so that the robot body moves under the target motion parameters;

[0113] In some embodiments of this application, the PLC determines whether the user has selected a joint motion mode or a linear motion mode based on the received gesture command. If a "single hand gesture of '1'" is recognized, the PLC determines it to be a joint motion mode; if other preset corresponding joint mode gestures are recognized, the PLC enters the corresponding joint motion mode processing flow and sends a command to the teach pendant, causing the teach pendant to display "Please select target axis". At this time, the teach pendant displays a prompt interface for selecting the target axis. In joint motion mode, the PLC waits to receive the user's gesture information for selecting the motion axis. The industrial control computer recognizes the user's gesture (e.g., "single hand gesture of '4'" to select axis 4) and sends the encoded gesture command to the PLC. The PLC parses the received axis selection gesture information, determines the target axis selected by the user, and sets the state of that axis to the ready-to-move state. Through the above two prompt interfaces, the final motion mode and the final target axis can be determined.

[0114] Simultaneously, the PLC sends instructions to the teach pendant, causing it to display the currently selected axis information, current speed, coordinates, and other status information, allowing the user to understand the robot's current status. At this time, the teach pendant displays a prompt interface indicating the robot's status. The user controls the robot's movement through hand gestures such as wrist rotation. The industrial control computer maps the wrist rotation angle to a movement speed and sends this speed information to the PLC. After receiving the movement speed information, the PLC determines the movement direction based on the gesture type and uses the movement speed and direction as the final movement parameters. It then uses the final movement mode, final target axis, and final movement parameters as the robot's target movement parameters, writing them into the designated register in the robot control cabinet. The robot then begins movement according to these target movement parameters. During movement, the PLC continuously receives distance data from the end effector's laser rangefinder, monitoring the distance between the robot's end effector and the engine in real time. If the laser rangefinder detects that the distance is too close (less than a preset safe distance threshold), it immediately sends a warning message carrying a "close distance signal" to the PLC.

[0115] S104, receives warning information generated during the robot's movement sent by the laser rangefinder;

[0116] S105, based on the warning information, control the robot body and control the display module to display the warning information until the robot body is aligned with the target detection point.

[0117] In some embodiments of this application, after receiving a warning message carrying a "near-distance signal," the PLC automatically adjusts the robot's movement speed to half of the current speed and sends a prompt message to the teach pendant, causing the teach pendant to display a "distance too close, speed halved" prompt message. The user can continue to adjust the wrist rotation angle to change the speed according to the actual situation, or return the wrist to center to stop the robot's movement.

[0118] For example Figure 9 The diagram illustrates two control processes for stopping movement in a robot control system. First, the user issues a stop command by returning their wrist to a position less than or equal to 5 degrees. Upon receiving the gesture command, the industrial control computer (ICC) sends an instruction to the programmable logic controller (PLC) to set the robot's speed to 0. The PLC, upon receiving the 0 speed instruction, controls the robot to stop. Alternatively, in an emergency, the user presses the emergency stop button on the teach pendant. Upon receiving the emergency stop button signal, the PLC sends an emergency stop command that includes freezing the robot's movement and triggering an audible and visual alarm. Upon receiving the emergency stop command, the robot control system immediately freezes all robot movement and activates the audible and visual alarm to alert the operator and surrounding personnel to safety.

[0119] Furthermore, after aligning with the target detection point, the programmable logic controller responds to the recording instruction, acquires the spatial coordinates and joint angle coordinates of the robot body, and stores them in a preset designated storage area; after all detection points of the engine have been recorded, a trajectory waypoint dataset is obtained; the trajectory optimization algorithm is called to process the trajectory waypoint dataset to eliminate path abrupt changes and generate an executable program that conforms to the joint acceleration constraints of the robot body; the executable program is stored so that the robot body can use it in automatic detection tasks.

[0120] In one possible implementation, once the robot moves to and precisely aligns with the target detection point, the operator triggers the recording operation via a gesture (e.g., "one-handed gesture to 5"). The industrial control computer recognizes the gesture and sends the recording instruction to the PLC. Upon receiving the instruction, the PLC sends a command to the robot control cabinet to obtain the robot's current spatial coordinates and joint angle coordinates, and synchronously stores this data in a designated storage area of ​​the PLC, forming a trajectory waypoint dataset. Simultaneously, the PLC sends a command to the teach pendant, causing it to display "Waypoint X recorded" (X being the waypoint number), informing the operator that waypoint recording was successful. If other detection points on the same engine require debugging, the operator makes a return gesture. Upon receiving this command, the PLC returns the teach pendant to the axis selection interface, allowing the operator to repeat the "select axis - move - record" process to continue debugging other detection points. Once all waypoint data for all detection points has been recorded, the operator triggers a program generation command. Upon receiving this command, the PLC calls a trajectory optimization algorithm to process the trajectory waypoint dataset, eliminating path abrupt changes and generating an executable program that conforms to the robot's joint acceleration constraints. The generated program is stored in the PLC or robot control cabinet for use by the robot in subsequent automatic detection tasks.

[0121] For example Figure 10 As shown, Figure 10 This application provides a schematic block diagram of the debugging process for a gesture recognition-based detection robot, including device initialization, robot mode and axis selection, path selection, and safety checks. Initialization operations such as device communication and teach pendant readiness are performed to ensure system readiness. The user selects a motion mode, including joint motion or linear motion. The user selects the axis to control, which can be a target axis (1-6) or a linear axis (X / Y / Z). Based on the selected mode and axis, corresponding motion control is performed. The user selects a target path, including a joint path or a linear path. The selected path is executed, controlling the robot to move along a predetermined trajectory. During movement, the user can trigger a point recording operation to record the current position and posture. After point recording, the path parameters are saved. Anomalies, such as path errors or device malfunctions, are detected. The control mode is automatically or manually adjusted based on the operation. After point recording, the motion path composed of multiple detection points is optimized and adjusted to improve efficiency and accuracy. After all operations are completed, the robot control process ends.

[0122] In this embodiment, on the one hand, the user's gestures are directly collected and converted into commands by the gesture recognition module, eliminating the need for multiple button presses. The programmable logic controller (PLC) can receive gesture commands in real time and control the robot's movement, greatly reducing operation steps and time, and significantly improving debugging efficiency. On the other hand, by directly collecting user gestures through the gesture recognition module, the user does not need to frequently switch their gaze, allowing them to focus on gesture operations. Simultaneously, the gesture recognition module can accurately capture gesture movements and convert them into accurate motion parameters, significantly reducing recording errors in dense detection point scenarios and meeting high-precision detection requirements. Furthermore, by monitoring the distance between the robot and the target detection point, a warning message is generated and sent to the PLC when the distance is too close. The PLC can immediately adjust the robot's speed or stop its movement based on the warning message and display the warning message through the display module, reminding the user to pay attention to safety, thereby reducing the risk of collision and improving safety in close-range operation scenarios.

[0123] This application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the gesture recognition-based detection robot debugging method provided in the above-described method embodiments.

[0124] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the gesture recognition-based detection robot debugging method of the above-described method embodiments.

[0125] Please see Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0126] The communication bus 1002 is used to realize the connection and communication between these components.

[0127] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0128] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0129] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.

[0130] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage system located remotely from the aforementioned processor 1001. Figure 11 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a gesture recognition-based debugging application for a detection robot.

[0131] exist Figure 11In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 1001 can be used to call the gesture recognition-based detection robot debugging application stored in the memory 1005, and specifically perform the following operations:

[0132] During the process of aligning the robot body with the target detection point of the engine, the control display module displays a prompt interface to prompt the user to make gestures;

[0133] Receive gesture commands sent by the gesture recognition module, which are obtained by recognizing gestures from the image data stream of the user's gestures.

[0134] Based on the gesture command, the target motion parameters of the robot body are determined so that the robot body can move under the target motion parameters;

[0135] Receive early warning information generated during the robot's movement from the laser rangefinder;

[0136] Based on the warning information, control the robot body and control the display module to display the warning information until the robot body is aligned with the target detection point.

[0137] In this embodiment, on the one hand, the user's gestures are directly collected and converted into commands by the gesture recognition module, eliminating the need for multiple button presses. The programmable logic controller (PLC) can receive gesture commands in real time and control the robot's movement, greatly reducing operation steps and time, and significantly improving debugging efficiency. On the other hand, by directly collecting user gestures through the gesture recognition module, the user does not need to frequently switch their gaze, allowing them to focus on gesture operations. Simultaneously, the gesture recognition module can accurately capture gesture movements and convert them into accurate motion parameters, significantly reducing recording errors in dense detection point scenarios and meeting high-precision detection requirements. Furthermore, by monitoring the distance between the robot and the target detection point, a warning message is generated and sent to the PLC when the distance is too close. The PLC can immediately adjust the robot's speed or stop its movement based on the warning message and display the warning message through the display module, reminding the user to pay attention to safety, thereby reducing the risk of collision and improving safety in close-range operation scenarios.

[0138] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for debugging the gesture recognition-based detection robot can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the gesture recognition-based detection robot debugging program can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0139] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A gesture recognition-based inspection robot debugging system, characterized in that, The system includes: The system includes a gesture recognition module, a control module, and a display module; the control module is communicatively connected to both the gesture recognition module and the display module; the control module includes a programmable logic controller and a robot body for engine appearance inspection, with a laser rangefinder mounted at the end effector of the robot body; wherein... The programmable logic controller is used to control the display module to display a prompt interface for prompting the user to make gestures during the process of aligning the robot body with the target detection point of the engine. The gesture recognition module is used to perform gesture recognition based on the image data stream of the user's gesture, obtain gesture commands, and send them to the programmable logic controller. The gesture recognition process, which involves obtaining gesture commands based on the image data stream, includes: The image data stream is preprocessed to obtain a 3D feature matrix of hand key points composed of multiple frames within a single time window; a pre-trained gesture parameter recognition model is loaded; the gesture parameter recognition model includes a 3D convolutional network and a discriminant network; the 3D feature matrix of hand key points is input into the 3D convolutional network for spatial convolution to extract the gesture posture features and temporal features of the hand in the RGB-D image frames; the gesture posture features and temporal features of the gesture are input into the discriminant network to obtain the gesture category and the user's wrist rotation angle; the wrist rotation angle is mapped to the movement speed of the robot body; the gesture category and the movement speed are used as gesture commands. The programmable logic controller is further configured to determine target motion parameters of the robot body according to the gesture command, so that the robot body moves under the target motion parameters; The prompt interface is a prompt interface for selecting a motion mode or target axis, or a prompt interface for displaying robot status information; the gesture commands are gesture type and motion speed; The step of determining the target motion parameters of the robot body based on the gesture command includes: When the prompt interface prompts the user to select a motion mode, the corresponding final motion mode is obtained from the pre-built mapping relationship between gesture categories and motion modes based on the gesture category; or, when the prompt interface prompts the user to select a target axis, the corresponding final target axis is obtained from the pre-built mapping relationship between gesture categories and target axes based on the gesture category; or, when the prompt interface displays robot status information, the corresponding motion direction is obtained from the pre-built mapping relationship between gesture categories and robot motion directions based on the gesture category; the motion speed and motion direction are used as final motion parameters; the final motion mode, the final target axis, and the final motion parameters are used as the robot's target motion parameters. The laser rangefinder is used to generate early warning information and send it to the programmable logic controller during the movement of the robot body. The programmable logic controller is further configured to control the robot body and the display module to display the warning information based on the warning information, until the robot body is aligned with the target detection point; The gesture recognition module is further configured to generate a recording instruction and send it to the programmable logic controller (PLC) when the robot body is aligned with the target detection point and the gesture instruction indicates a recording operation; the PLC is further configured to, in response to the recording instruction, acquire the spatial coordinates and joint angle coordinates of the robot body and store them in a preset designated storage area; when all detection points of the engine are recorded, a trajectory waypoint dataset is obtained; a trajectory optimization algorithm is called to process the trajectory waypoint dataset to eliminate path abrupt changes and generate an executable program that conforms to the joint acceleration constraints of the robot body; the executable program is stored so that the robot body can use it in automatic detection tasks.

2. The system according to claim 1, characterized in that, The gesture recognition module includes a depth camera, a composite light source auxiliary component, and an industrial control computer; the depth camera is electrically connected to the industrial control computer. The composite light source auxiliary component is used to create a completely shadowless environment for the engine on the plate chain conveyor equipment of the production line. The depth camera is used to capture continuous image frames of the user's gestures, obtain an image data stream, and send it to the industrial control computer; The industrial control computer is used to perform gesture recognition based on the image data stream, obtain gesture commands, and send them to the programmable logic controller.

3. The system according to claim 1, characterized in that, The preprocessing of the image data stream yields a three-dimensional feature matrix of hand key points composed of multiple frames within a single time window, including: The RGB-D image frames with timestamps in the image data stream are segmented and removed from the background using a preset depth information threshold to obtain the user's hand foreground image; A morphological opening operation is performed on the hand foreground image to remove image noise from the hand foreground image, resulting in a denoised hand foreground image; The denoised hand foreground image is scaled down to a standard size of 224×224 pixels to obtain a scaled hand foreground image; From the scaled hand foreground image, the three-dimensional coordinates of multiple hand key points are extracted using a preset hand key point detection algorithm to obtain the three-dimensional coordinates of multiple hand key points in the RGB-D image frame; The three-dimensional coordinates of multiple hand key points in all RGB-D image frames within a single preset time window are combined to obtain a three-dimensional feature matrix of hand key points composed of multiple frames within a single time window.

4. The system according to claim 1, characterized in that, The control module also includes a robot control cabinet; To enable the robot body to move under the target motion parameters, the following methods are included: Set the state of the final target axis to a ready-to-move state so that the robot body enters the ready-to-move state of the final target axis; The final motion parameters are written into a designated register of the robot control cabinet so that the robot body moves according to the final motion parameters.

5. The system according to claim 1, characterized in that, The generation of early warning information during the movement of the robot body includes: During the movement of the robot body, the distance between it and the engine is acquired in real time; When the distance is less than a preset safety threshold, a near-field signal for early warning is generated; The near-field signal used for early warning is encapsulated into early warning information.

6. The system according to claim 5, characterized in that, The display module includes a teach pendant; The step of controlling the robot body and the display module to display the warning information based on the warning information includes: Upon receiving the near-field signal, the robot body's movement speed is adjusted to half of its original movement speed to obtain the current speed; Based on the proximity signal and the current speed, a prompt message is generated and sent to the teach pendant so that the teach pendant can display the prompt message.

7. A method for debugging a detection robot based on gesture recognition, implemented using the system described in any one of claims 1-6, characterized in that, Applied to a programmable logic controller, the method includes: During the process of aligning the robot body with the target detection point of the engine, the control display module displays a prompt interface to prompt the user to make gestures; Receive gesture commands sent by the gesture recognition module, which are obtained by recognizing gestures from the image data stream of the user's gestures. Based on the gesture command, the target motion parameters of the robot body are determined so that the robot body moves under the target motion parameters; Receive early warning information generated during the movement of the robot body from the laser rangefinder; Based on the warning information, the robot body is controlled and the display module is controlled to display the warning information until the robot body is aligned with the target detection point.

Images