Collaborative robot control method, apparatus, computer device and storage medium

Abstract

The application relates to a collaborative robot control method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a gesture control event triggered for a collaborative robot, collecting hand image information in a set area; the hand image information comprises hand images in at least two time frames; gesture recognition is respectively performed on each hand image to determine respective gestures corresponding to each time frame; based on the respective gestures corresponding to each time frame and the time sequence of each time frame, motion information commonly represented by the gestures is determined; and the collaborative robot is controlled according to the motion information, thereby greatly improving work efficiency.

Description

Technical Field

[0001] This application relates to the field of collaborative robot technology, and in particular to a collaborative robot control method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, collaborative robots that support human-machine collaborative work modes have emerged. Using collaborative robots can improve production efficiency and reduce labor costs.

[0003] Traditional collaborative robot control methods rely on pre-set paths or points for repetitive tasks. If there is a deviation between the target point and the actual point, the path or point needs to be repeatedly adjusted to make the final target point gradually approach the actual target point. This consumes a lot of the robot's computing resources and has the disadvantage of low work efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a collaborative robot control method, device, computer equipment, computer-readable storage medium, and computer program product that can improve work efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a collaborative robot control method. The method includes:

[0006] In response to a gesture control event triggered by the collaborative robot, hand image information is acquired within a defined area; the hand image information includes hand images from at least two time frames;

[0007] Gesture recognition is performed on each of the hand images to determine the gesture corresponding to each time frame;

[0008] Based on the gestures corresponding to each time frame and the time sequence of each time frame, the motion information commonly represented by each gesture is determined.

[0009] The collaborative robot's movements are controlled according to the motion information.

[0010] In one embodiment, the collaborative robot control method further includes:

[0011] An initial hand image is acquired within a defined area, and gesture recognition is performed on the initial hand image to determine the initial gesture represented by the initial hand image. If the initial gesture is the control start gesture of the collaborative robot, a gesture control event for the collaborative robot is triggered.

[0012] In one embodiment, determining the motion information commonly represented by each gesture based on the gesture corresponding to each time frame and the temporal order of each time frame includes:

[0013] According to the time order of each time frame, the gestures are sorted to obtain a gesture sequence; continuous gesture pairs with gesture changes are identified in the gesture sequence; each continuous gesture pair includes a previous gesture that is earlier in the time order and a next gesture that is later in the time order; for each continuous gesture pair, the motion sub-information represented by the continuous gesture pair is determined according to the change of the next gesture relative to the previous gesture in the continuous gesture pair; the information sequence formed by each motion sub-information in time order is determined as the motion information commonly represented by each gesture.

[0014] In one embodiment, determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes:

[0015] The palm orientation of the continuous gesture pair is identified to obtain the palm orientation of the next gesture and the previous gesture in the continuous gesture pair; when the palm orientations meet the angle change conditions, the angle change information of the next gesture relative to the previous gesture is determined; based on the rotation information represented by the angle change information, the motion sub-information represented by the continuous gesture pair is determined.

[0016] In one embodiment, determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes:

[0017] The hand positions of the next and previous gestures in the continuous gesture pair are determined; when the hand positions satisfy the position change conditions, the position change information of the next gesture relative to the previous gesture is determined; based on the movement information represented by the position change information, the motion sub-information represented by the continuous gesture pair is determined.

[0018] In one embodiment, the collaborative robot control method further includes:

[0019] Determine the hand movement distance and hand movement direction represented by the position change information; determine the robot end effector movement distance corresponding to the hand movement distance and the robot end effector movement direction corresponding to the hand movement direction as the movement information represented by the position change information.

[0020] In one embodiment, the collaborative robot control method further includes:

[0021] The hand movement direction represented by the position change information and the duration of the next gesture in the gesture sequence are determined; the robot end effector movement direction corresponding to the hand movement direction and the robot end effector continuous movement time corresponding to the duration are determined as the movement information represented by the position change information.

[0022] Secondly, this application also provides a collaborative robot control device. The device includes:

[0023] The information acquisition module is used to acquire hand image information within a set area in response to a gesture control trigger event for the collaborative robot; the hand image information includes hand images from at least two time frames;

[0024] The gesture determination module is used to perform gesture recognition on each of the hand images and determine the gesture corresponding to each of the time frames.

[0025] The motion information determination module is used to determine the motion information commonly represented by each gesture based on the gesture corresponding to each time frame and the time order of each time frame.

[0026] The motion control module is used to control the actions of the collaborative robot according to the motion information.

[0027] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0028] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0029] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0030] The aforementioned collaborative robot control method, apparatus, computer equipment, storage medium, and computer program product, in response to a gesture control trigger event for the collaborative robot, collects hand image information within a set area. This hand image information includes hand images from at least two time frames, which can reflect changes in hand gestures in real time. Then, gesture recognition is performed on each hand image to determine the gesture corresponding to each time frame. Based on the gesture corresponding to each time frame and the time sequence of each time frame, the motion information commonly represented by each gesture is determined. This clearly represents the motion information covered by the hand gesture action and provides accurate motion information for the collaborative robot. Finally, the collaborative robot's actions are controlled according to this motion information, greatly improving work efficiency. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of a collaborative robot control method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a collaborative robot control method in one embodiment;

[0033] Figure 3 This is a flowchart illustrating a collaborative robot control method in another embodiment;

[0034] Figure 4 This is a flowchart illustrating a collaborative robot control method in another embodiment;

[0035] Figure 5 This is a schematic diagram of instructions in a collaborative robot control method in one embodiment;

[0036] Figure 6 This is a schematic diagram of action priority in a collaborative robot control method in one embodiment;

[0037] Figure 7 This is a schematic diagram of the effective recognition area in a collaborative robot control method in one embodiment;

[0038] Figure 8 This is a schematic diagram of the effective detection area in a collaborative robot control method in one embodiment;

[0039] Figure 9 This is a schematic diagram of the instructions in a collaborative robot control method in another embodiment;

[0040] Figure 10 This is a schematic diagram of the instructions in a collaborative robot control method in another embodiment;

[0041] Figure 11 This is a schematic diagram of the instructions in a collaborative robot control method in another embodiment;

[0042] Figure 12 This is a structural block diagram of a collaborative robot control device in one embodiment;

[0043] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The collaborative robot control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the controller 104 communicates with the collaborative robot 102 via a network and acquires hand gesture images. The collaborative robot 102 has a data receiving module and a stable standardized command response module. The data receiving module receives data transmitted by the controller, and the standardized command response module responds to commands from the controller. The controller 104 can be a hardware module containing various processing chips and their peripheral circuits, possessing logic operation functions. This processing chip can be a microcontroller, a DSP (Digital Signal Processing) chip, or an FPGA (Field Programmable Gate Array) chip. During the control of the collaborative robot 102, in response to gesture control events triggered by the collaborative robot 102, the controller 104 acquires hand image information of the hand within a designated area. This hand image information includes hand images from at least two time frames. Gesture recognition is performed on each hand image to determine the gesture corresponding to each time frame. Then, based on the gestures corresponding to each time frame and the time sequence of each time frame, the motion information commonly represented by each gesture is determined. Finally, the collaborative robot 102 is controlled according to this motion information.

[0046] In one embodiment, such as Figure 2 As shown, a collaborative robot control method is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0047] Step S202: In response to a gesture control event triggered by the collaborative robot, collect hand image information within a set area.

[0048] The collaborative robot's operating modes can include code control mode and gesture control mode. In gesture control mode, the controller can control the collaborative robot based on gestures. For example, if a hand makes a rightward lateral movement, the collaborative robot can follow suit and perform the same rightward lateral movement; similarly, if a hand makes a flipping motion, the collaborative robot can follow suit and perform the same flipping motion. Gesture control events refer to events that activate the gesture control mode. The specific way a gesture control event is triggered is not unique. For example, a user can trigger a gesture control event by clicking or pressing the gesture control mode activation control. Alternatively, the controller can acquire an initial hand image within a defined area, perform gesture recognition on the initial hand image, and determine the initial gesture represented by the initial hand image; if the initial gesture is the control activation gesture for the collaborative robot, a gesture control event for the collaborative robot is triggered.

[0049] The defined area refers to the region represented by the image range that the image acquisition device can capture. A hand image refers to the image information of a hand captured by the image acquisition device at a specific moment. Hand image information includes hand images from at least two time frames.

[0050] Specifically, the controller can respond to gesture control events triggered by the collaborative robot, controlling the image acquisition device to acquire hand image information within a designated area. For example, the controller can actively acquire trigger information for gesture control events, or it can passively receive such trigger information; this is not limited here.

[0051] Step S204: Perform gesture recognition on each hand image to determine the gesture corresponding to each time frame.

[0052] Specifically, gestures can be extended, bent, or upright, etc.

[0053] Specifically, the controller can identify the corresponding gesture based on the hand features of each hand image and determine the gesture corresponding to each hand image as the gesture of the time frame to which each hand image belongs. In a specific embodiment, the controller identifies the corresponding gesture as an extended gesture, a bent gesture, or a vertical gesture based on the hand features of each hand image. Then, the gesture of the time frame to which each hand image belongs is, in order, an extended gesture, a bent gesture, or a vertical gesture. The specific method of gesture recognition is not unique. For example, the controller can perform gesture recognition on the hand image based on a gesture recognition model to obtain the gesture represented by the hand image. Alternatively, the controller can perform similarity matching between the hand contour of the hand image and the hand contour of candidate gestures, and determine the candidate gesture with the highest contour similarity as the gesture represented by the hand image.

[0054] Step S206: Based on the gestures corresponding to each time frame and the time sequence of each time frame, determine the motion information commonly represented by each gesture.

[0055] The motion information corresponds to the hand movement information represented by each gesture. For example, if the hand movement information is "rotate 90 degrees", the motion information can be "rotate 90 degrees"; if the hand movement information is "translate 20 centimeters to the right", the motion information can be "translate 20 centimeters to the right" or "translate 30 centimeters to the right", and so on. In some specific embodiments, the motion information can include multiple motion sub-information. A motion sub-information can be determined based on changes in two adjacent gestures, or it can be determined based on changes in multiple ordered gestures.

[0056] Specifically, the controller can arrange the gestures corresponding to each time frame in chronological order to form ordered gestures, and determine the common motion information represented by each gesture based on the changing characteristics of the ordered gestures. For example, if the gestures are arranged in chronological order as an outstretched gesture and a vertical gesture, it can be said that the motion information of the hand is "rotation 90 degrees".

[0057] Step S208: Control the collaborative robot's movements according to the motion information.

[0058] Specifically, the controller can control the collaborative robot's movements based on determined motion information, and these movements can be achieved by the robot's end effector. For example, if the determined motion information is "rotate 90 degrees," the controller can control the robot's end effector to perform the same "rotate 90 degrees" movement; if the determined motion information is "translate 10 centimeters to the right," the controller can control the robot's end effector to perform the same "translate 10 centimeters to the right" movement. In some specific embodiments, technicians can also manually set the motion path. When the collaborative robot moves according to the set path, the technicians can make real-time hand gestures to adjust the actual motion path, making the control process of the collaborative robot more precise.

[0059] In the aforementioned collaborative robot control method, in response to a gesture control trigger event for the collaborative robot, hand image information within a set area is collected. This hand image information includes hand images from at least two time frames, which can reflect the changes in hand gestures in real time. Then, gesture recognition is performed on each hand image to determine the gesture corresponding to each time frame. Based on the gesture corresponding to each time frame and the time sequence of each time frame, the motion information commonly represented by each gesture is determined. This method can clearly represent the motion information covered by the hand gesture and provide accurate motion information for the collaborative robot. Finally, the collaborative robot's actions are controlled according to this motion information, greatly improving work efficiency.

[0060] In one embodiment, the collaborative robot control method further includes: acquiring an initial hand image within a set area, performing gesture recognition on the initial hand image, and determining the initial gesture represented by the initial hand image; and triggering a gesture control event for the collaborative robot if the initial gesture is a control start gesture for the collaborative robot.

[0061] The control activation gesture refers to a gesture that can initiate the gesture control mode. For example, the control activation gesture could be a clenched fist followed by an open fist, or it could be the gesture represented by the number 3, etc. It's easy to understand that the control activation gesture can be set by technical personnel; there are no restrictions here.

[0062] Specifically, the controller can acquire an initial hand image within a designated area and perform gesture recognition on that initial hand image to determine the initial gesture corresponding to the initial hand image. If the initial gesture is the control activation gesture for the collaborative robot, the collaborative robot will trigger the gesture control mode. In other words, if the initial hand image within the designated area does not contain the corresponding control activation gesture, the collaborative robot will not trigger the gesture control mode, even if other gestures exist in the initial hand image. The collaborative robot will not move until a control activation gesture is found in the initial hand image.

[0063] This embodiment uses a gesture control method that triggers the collaborative robot's gesture control by detecting and controlling the start gesture, which greatly reduces the false touch rate during the control of the collaborative robot and helps to improve control accuracy.

[0064] In one embodiment, step S206 includes: sorting each gesture according to the time order of each time frame to obtain a gesture sequence; determining continuous gesture pairs in the gesture sequence that have gesture changes; for each continuous gesture pair, determining the motion sub-information represented by the continuous gesture pair according to the change of the next gesture relative to the previous gesture in the continuous gesture pair; and determining the information sequence formed by each motion sub-information in time order as the motion information commonly represented by each gesture.

[0065] Among them, consecutive gesture pairs include the preceding gesture in time sequence and the following gesture in time sequence. A change in gesture between the preceding and following gestures can mean that the two gestures correspond to different gesture categories, such as one gesture with the palm facing up and the other with the palm facing left; or one gesture is a clenched fist with bent fingers and the other is an open palm with outstretched fingers. A change in gesture between the preceding and following gestures can also mean that the positional difference between the corresponding hand positions of the two gestures satisfies a positional change condition. The positional change condition can mean that the positional difference is greater than a difference threshold, or that the positional difference is greater than or equal to a difference threshold. In other words, when the degree of change between two consecutive gestures is small, the two gestures are considered to be the same gesture. The preceding and following gestures can be two gestures that are adjacent in time sequence, or two gestures that are not adjacent in time sequence.

[0066] Specifically, the controller can sort each gesture according to the time order of the corresponding time frame to obtain a gesture sequence, filter out continuous gestures without gesture changes from the gesture sequence, select continuous gesture pairs with gesture changes, and then for each continuous gesture pair, determine the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair. This way, the motion sub-information corresponding to each of the continuous gesture pairs with gesture changes can be obtained. Finally, these motion sub-information are arranged in time order, and the resulting information sequence serves as the motion information jointly represented by each gesture.

[0067] This embodiment breaks down the process of obtaining motion information into multiple processes of obtaining motion sub-information, thereby improving the accuracy of the collaborative robot control process.

[0068] In one embodiment, determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes: performing palm orientation recognition on the continuous gesture pair to obtain the palm orientation of the next gesture and the previous gesture in the continuous gesture pair; determining the angle change information of the next gesture relative to the previous gesture when each palm orientation satisfies the angle change condition; and determining the motion sub-information represented by the continuous gesture pair based on the rotation information represented by the angle change information.

[0069] The angle change information refers to the information corresponding to the change in the angle of the next gesture relative to the previous gesture. The angle change information can include at least one of the following: the direction of the angle change or the amount of the angle change. For example, the angle change information could be "rotate 60 degrees to the left," which means that the direction of the angle change is "to the left" and the amount of the angle change is "60 degrees."

[0070] The angle change condition can refer to the angle change needing to reach a certain threshold, such as the angle change being less than or equal to the angle threshold. Specifically, in this application, if the angle change of the palm position of the next gesture relative to the palm position of the previous gesture reaches the set threshold, it can be considered that the angle change condition is met. For example, if the angle threshold is "10 degrees," then if the angle change of the palm position of the next gesture relative to the palm position of the previous gesture is less than "10 degrees," then the angle change of each palm position is considered not to meet the angle change condition. It is easy to understand that if the angle change of the palm position of the next gesture relative to the palm position of the previous gesture is greater than or equal to "10 degrees," then the angle change of each palm position is considered to meet the angle change condition.

[0071] Rotation information may include at least one of the rotation direction represented by the direction of angle change or the rotation speed represented by the amount of angle change. For example, if the direction of angle change is "to the left", then the rotation direction represented by the direction of angle change can be "rotating to the left"; if the amount of angle change is "60 degrees", then the rotation speed represented by the amount of angle change can be the rotation speed represented by the amount of angle change, such as "60 degrees / s".

[0072] Specifically, the controller can identify the palm orientation of the next and previous gestures in a continuous gesture pair. When the palm orientation meets the angle change condition, that is, when the angle change of each palm orientation meets the threshold, the angle change information of the next gesture relative to the previous gesture can be determined. Finally, based on the rotation information represented by the angle change information, the motion sub-information represented by the continuous gesture pair can be determined.

[0073] This embodiment recognizes the angular changes of each palm position in a continuous gesture pair, which enables the control of the collaborative robot's rotational movements based on the rotational motion of the hands, thus improving the flexibility of the collaborative robot control method.

[0074] In one embodiment, determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes: determining the palm position of the next gesture and the previous gesture in the continuous gesture pair; determining the position change information of the next gesture relative to the previous gesture when each palm position satisfies the position change condition; and determining the motion sub-information represented by the continuous gesture pair based on the movement information represented by the position change information.

[0075] Among them, the position change condition can refer to the position change needing to reach a certain threshold.

[0076] In a specific embodiment, a positional change condition is considered met when the change in the palm position of the next gesture relative to the palm position of the previous gesture reaches a set threshold. For example, if the threshold is set to "10cm", then if the positional change of the palm position of the next gesture relative to the palm position of the previous gesture is less than "10cm", then the positional change of each palm position is considered not to meet the positional change condition. It is easy to understand that if the positional change of the palm position of the next gesture relative to the palm position of the previous gesture is greater than or equal to "10cm", then the positional change of each palm position is considered to meet the positional change condition.

[0077] In another specific embodiment, the defined area may include multiple sub-regions, and satisfying the position change condition means that the hand position changes from one sub-region to another. For example, assuming the defined areas are four regions a, b, c, and d, when the hand position moves from region a to region b, even if the hand position only undergoes a small distance change, it is considered that the hand position satisfies the position change condition.

[0078] The movement information may include at least one of the following: the direction of movement represented by the direction of position change, and the distance of movement represented by the amount of position change. For example, if the direction of position change is "to the left", then the direction of movement represented by the direction of position change can be "moving to the left"; if the amount of position change is "10cm", then the distance of movement corresponding to the amount of position change can be "10cm".

[0079] Specifically, the controller can first determine the palm positions of the next and previous gestures in a continuous gesture pair. When each palm position meets the position change condition, the position change information of the next gesture relative to the previous gesture is determined. Finally, based on the movement information represented by the position change information, the motion sub-information represented by the continuous gesture pair can be determined.

[0080] This embodiment identifies the positional changes of each hand in a continuous gesture pair, enabling the control of the collaborative robot's movement based on hand movements, thus improving the flexibility of the collaborative robot control method.

[0081] In one embodiment, the collaborative robot control method further includes: determining the hand movement distance and hand movement direction represented by the position change information; and determining the robot end effector movement distance corresponding to the hand movement distance and the robot end effector movement direction corresponding to the hand movement direction as the movement information represented by the position change information.

[0082] The relationship between the hand movement distance and the robot end effector movement distance can be represented by a table or function. The values ​​of the hand movement distance and the robot end effector movement distance can be equal or proportional.

[0083] Specifically, the controller can determine the hand movement distance and hand movement direction represented by the position change information, map the hand movement distance to obtain the robot end effector movement distance, and determine the robot end effector movement direction based on the hand movement direction, finally obtaining the movement information represented by the robot end effector movement distance and robot end effector movement direction.

[0084] This embodiment maps hand position change information to robot end-effector movement information, enabling the robot to track and reproduce hand movements, thus expanding the functional scope of collaborative robots and improving their work efficiency.

[0085] In one embodiment, the collaborative robot control method further includes: determining the hand movement direction represented by the position change information and the duration of the next gesture in the gesture sequence; and determining the robot end-effector movement direction corresponding to the hand movement direction and the robot end-effector continuous movement time corresponding to the duration as the movement information represented by the position change information.

[0086] The duration of the next gesture can refer to the time during which other gestures that follow in the time sequence do not change or change but do not meet the conditions for a change. For example, suppose the next gesture in a sequence is a vertical gesture, and the condition for a change is a rotation angle greater than or equal to 10 degrees. Then, the sum of the times of other gestures that follow in the time sequence do not change or change less than 10 degrees is the duration of the next gesture.

[0087] Specifically, the controller can determine the hand movement direction and the duration of the next gesture in the gesture sequence based on the position change information, and determine the robot end effector movement direction corresponding to the hand movement direction and the robot end effector continuous movement time corresponding to the duration as the movement information represented by the position change information.

[0088] This embodiment determines the robot end effector's movement direction and duration based on the hand movement direction and the duration of the next gesture in the gesture sequence. This enables the robot to achieve the origin mode function for hand movements, broadens the functional range of collaborative robots, and improves the working efficiency of collaborative robots.

[0089] In one embodiment, such as Figure 3 As shown, the collaborative robot control method includes:

[0090] Step S301: Acquire an initial hand image within a defined area, perform gesture recognition on the initial hand image, and determine the initial gesture represented by the initial hand image;

[0091] Step S302: If the initial gesture is the control start gesture for the collaborative robot, trigger a gesture control event for the collaborative robot.

[0092] Step S303: In response to a gesture control event triggered by the collaborative robot, collect hand image information within a set area;

[0093] The hand image information includes hand images from at least two time frames;

[0094] Step S304: Perform gesture recognition on each hand image to determine the gesture corresponding to each time frame;

[0095] Step S305: Sort the gestures according to the time order of each time frame to obtain the gesture sequence;

[0096] Step S306: Determine consecutive gesture pairs in the gesture sequence that have gesture changes;

[0097] Among them, a continuous gesture pair includes the previous gesture that is earlier in the time sequence and the next gesture that is later in the time sequence;

[0098] Step S307: For each consecutive gesture pair, perform palm orientation recognition on the consecutive gesture pair to obtain the palm orientation of the next gesture and the previous gesture in the consecutive gesture pair.

[0099] Step S308: Under the condition that the angle change conditions are met for each palm position, determine the angle change information of the next gesture relative to the previous gesture;

[0100] Step S309: Based on the rotation information represented by the angle change information, determine the motion sub-information represented by the continuous gesture pair;

[0101] Step S310: For each consecutive gesture pair, determine the palm position of the next gesture and the previous gesture in the consecutive gesture pair.

[0102] Step S311: If the position of each hand meets the position change condition, determine the position change information of the next gesture relative to the previous gesture;

[0103] Step S312: Determine the hand movement distance and hand movement direction represented by the position change information;

[0104] Step S313: The robot end effector movement distance corresponding to the hand movement distance and the robot end effector movement direction corresponding to the hand movement direction are determined as the movement information represented by the position change information.

[0105] Step S314: Determine the direction of hand movement represented by the position change information, and the duration of the next gesture in the gesture sequence;

[0106] Step S315: The robot end-effector movement direction corresponding to the hand movement direction and the robot end-effector continuous movement time corresponding to the duration are determined as the movement information represented by the position change information.

[0107] Step S316: Based on the movement information represented by the position change information, determine the motion sub-information represented by the continuous gesture pair;

[0108] Step S317: The information sequence formed by each motion sub-information in chronological order is determined as the motion information commonly represented by each gesture;

[0109] Step S318: Control the collaborative robot's movements according to the motion information.

[0110] In a specific embodiment, such as Figure 4 As shown, the collaborative robot control method may include the following steps:

[0111] Step S401: Begin executing the collaborative robot control process;

[0112] Step S402: The user clicks the start button on the display screen;

[0113] Step S403: Determine whether the connection with the collaborative robot is normal;

[0114] Step S404: If the connection with the collaborative robot is normal, enter the initialization state;

[0115] Step S405: Detect whether the initial hand image contains a control start gesture for the collaborative robot;

[0116] Step S406: If the connection with the collaborative robot is not normal, report an error and terminate the process;

[0117] Step S407: If the initial hand image contains a control start gesture for the collaborative robot, the collaborative robot gesture control mode is activated with the hand position where the control start gesture is detected as the origin.

[0118] In step S408, if the initial hand image does not contain a control start gesture for the collaborative robot, return to step S404.

[0119] Step S409: Recognize gestures and poses;

[0120] Step S410: Determine whether the recognition range is exceeded;

[0121] Step S411: If the recognition range is not exceeded, determine whether an end gesture has been detected;

[0122] Step S412: When the recognition range is exceeded or an end gesture is detected, the collaborative robot gesture control mode is turned off and the process ends.

[0123] Step S413: If no end gesture is detected, determine whether the difference between the gesture position in the current time frame and the origin position is greater than a threshold.

[0124] Step S414: If the difference between the gesture position and the origin position in the current time frame is greater than or equal to a threshold, control the collaborative robot to follow the gesture to move.

[0125] In step S415, if the difference between the gesture position in the current time frame and the origin position is less than the threshold, return to step S409.

[0126] In a specific embodiment, such as Figure 5 As shown, the collaborative robot control method includes:

[0127] Step S501: Enter the initialization state;

[0128] Step S502: Extract the data information of the time frame pointed to by the current ID and encapsulate it using Struct, and then perform region recognition on the hand image of the current time frame;

[0129] Step S503: Perform control start gesture judgment;

[0130] Step S504: If the hand image does not contain a control start gesture, execute the _IsOnControl=false instruction to confirm that the control system is off;

[0131] Step S505: Block the motion information corresponding to the hand gesture image and return to step S502;

[0132] Step S506: If the hand image contains a control start gesture, set the judgment origin;

[0133] Step S507: Execute the ArrangeInSpace=n instruction to obtain the region determination result n;

[0134] Step S508: Execute the _IsOnControl=true command to confirm that the control system has been turned on;

[0135] Step S509: Confirm again whether the control system is in the open state;

[0136] Step S510: With the control system in the open state, execute the case ArraryInSpace command to send the corresponding command for each area according to the number of areas n;

[0137] Step S511: Execute the movement or stop command represented by the instruction, and return to step S502.

[0138] In a specific embodiment, such as Figure 6 As shown, the collaborative robot control method includes: setting gesture action priorities, with the priority designed as Rz>Rx=Ry>x=y=z, where Rz, Rx, Ry, x, y, and z are the six degrees of freedom of the hand or robot, x, y, and z are three-dimensional spatial coordinates, and Rz, Rx, and Ry are attitude angular coordinates. There are three gesture action judgment modes, and the specific judgment process is as follows:

[0139] Step S601: Determine whether the gesture action has triggered the Rz discrimination range.

[0140] Among them, the gesture action triggers the discrimination range of a certain degree of freedom, which means that the gesture action can represent the motion information of that degree of freedom. In other words, the controller needs to control the collaborative robot to move in that degree of freedom according to the motion information.

[0141] Step S602: When the gesture triggers the Rz discrimination range, a control signal in the corresponding direction is issued.

[0142] Step S603: If the gesture does not trigger the Rz discrimination range, determine whether the gesture triggers the Rx or Ry discrimination range.

[0143] Step S604: If the gesture triggers the Rx or Ry discrimination range, a control signal in the corresponding direction is issued.

[0144] Step S605: If the gesture does not trigger the Rx or Ry discrimination range, determine whether the gesture has triggered the x, y or z discrimination range.

[0145] Step S606: When the gesture triggers the x, y, or z discrimination range, a control signal in the corresponding direction is issued.

[0146] The effective recognition area triggered by the hand gesture in the x, y, or z range is defined as the effective acquisition area of ​​the collaborative robot's image acquisition device. Pre-state judgment and protection measures are added when analyzing hand gesture images. Optionally, the range of the effective recognition area corresponding to x, y, and z is as follows: Figure 7 As shown, point 0 refers to the location of the hand. The effective recognition area is the area outside the cube ABCD-A'B'C'D'. When the hand moves beyond the area where the cube is located, it enters the effective recognition area.

[0147] For Rx or Ry, the direction of the palm normal vector represents the direction of movement, and the effective detection area is... Figure 8The shaded area is shown. For Rz, the direction the finger points represents the direction of movement. The effective detection area is the region on the xz plane with the -z axis as the reference, and the deviation to the left or right is greater than 54 degrees and less than 90 degrees.

[0148] In a specific embodiment, such as Figure 9 As shown, the collaborative robot control method includes:

[0149] Step S901: Add analysis when acquiring gesture images of time frames;

[0150] Step S902: Execute the nHand=0? instruction to determine whether the number of detected hands is 0;

[0151] In step S903, if the number of hands detected is 0, the IsOnControl=true? instruction is executed to determine whether the control system is operating normally.

[0152] Step S904: Under normal operation of the control system, execute the commands IsOnControl=false, GrpStop, and OriginPoint reset to shut down the control system, stop the motion commands, reset the origin, and return to step S902.

[0153] Step S905: If the number of hands detected is not 0 or the control system is operating normally, proceed with the normal process.

[0154] In a specific embodiment, such as Figure 10 As shown, the collaborative robot control method includes:

[0155] Step S1001: Enter the initialization state;

[0156] Step S1002: Extract the data information of the frame pointed to by the current ID and encapsulate it using Struct, and then perform region recognition on the hand image of the current frame.

[0157] Step S1003: Perform control start gesture judgment;

[0158] Step S1004: If the hand image does not contain a control start gesture, execute the _IsOnControl=false instruction to confirm that the control system is off;

[0159] Step S1005: Block the motion information corresponding to the hand gesture image and return to step S1002.

[0160] Step S1006: If the hand image contains a control start gesture, set the device origin, robot origin, and start the servo command, i.e., start the servo command.

[0161] Step S1007: Execute the _IsOnControl=true command to confirm that the control system has been turned on;

[0162] Step S1008: Execute the _IsOnControl command to determine again whether the control system is turned on;

[0163] Step S1009: With the control system in the open state, execute the Servo data processing command;

[0164] In step S1010, if the control system is not in the open state, return to step S1002.

[0165] In a specific embodiment, such as Figure 11 As shown, the collaborative robot control method includes:

[0166] Step S1101: Begin the collaborative robot control process;

[0167] Step S1102: Execute the GetFrame command to obtain the gesture image of the time frame;

[0168] Step S1103: Execute the if Greped? instruction to determine whether the gesture image contains a control start gesture;

[0169] Step S1104: Execute the OrginPoint command to set the origin;

[0170] Step S1105: Execute the passedPoint=OrginPoint instruction, then jump to step S1109;

[0171] Step S1106: If the gesture image contains a control start gesture, execute the GetFrame instruction to obtain the gesture image of the time frame;

[0172] Step S1107: Execute the NowPoint command to set the current point position of the current time frame;

[0173] Step S1108, execute NowPoint-OrginPoint The N? command determines whether the difference between the current position and the origin position is much greater than N.

[0174] Step S1109: If the difference between the current point and the origin point is not much greater than N, execute the PushServoPassedPoint command to send the data of the old point to the Servo command.

[0175] Step S1110: If the difference between the current point and the origin point is much greater than N, execute the PushServoNowPoint command to send the data of the current point to the Servo command.

[0176] Step S1111: Execute the PassedPoint=NowPoint command to assign the data of the current point to the old point;

[0177] Step S1112: Execute the GetFrame command to obtain the gesture image of the time frame;

[0178] Step S1113, execute NowPoint-OrginPoint The N? command determines whether the difference between the current position and the origin position is much greater than N.

[0179] Step S1114: If the difference between the current point and the origin point is not much greater than N, execute the PushServoPassedPoint command to send the data of the old point to the Servo command.

[0180] Step S1115: If the difference between the current point and the origin point is much greater than N, execute the PushServoNowPoint command to send the data of the current point to the Servo command.

[0181] Step S1116: Execute the PassedPoint=NowPoint instruction to assign the data of the current point to the old point, and then re-execute step S1112.

[0182] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0183] Based on the same inventive concept, this application also provides a collaborative robot control device for implementing the collaborative robot control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the collaborative robot control device provided below can be found in the limitations of the collaborative robot control method described above, and will not be repeated here.

[0184] In one embodiment, such as Figure 12 As shown, a collaborative robot control device 1200 is provided, including: an information acquisition module, a gesture determination module, a motion information determination module, and an action control module, wherein:

[0185] The information acquisition module 1202 is used to acquire hand image information within a set area in response to a gesture control trigger event for the collaborative robot; the hand image information includes hand images from at least two time frames;

[0186] The gesture determination module 1204 is used to perform gesture recognition on each hand image and determine the gesture corresponding to each time frame.

[0187] The motion information determination module 1206 is used to determine the motion information commonly represented by each gesture based on the gesture corresponding to each time frame and the time sequence of each time frame.

[0188] The motion control module 1208 is used to control the movements of the collaborative robot according to motion information.

[0189] In one embodiment, the collaborative robot control device further includes a control start gesture recognition module, which is used to: acquire an initial hand image in a set area, perform gesture recognition on the initial hand image, determine the initial gesture represented by the initial hand image, and trigger a gesture control event for the collaborative robot when the initial gesture is a control start gesture for the collaborative robot.

[0190] In one embodiment, the motion information determination module includes: a gesture sequence acquisition unit, used to sort each gesture according to the time order of each time frame to obtain a gesture sequence; a continuous gesture pair determination unit, used to determine continuous gesture pairs in the gesture sequence that have gesture changes; the continuous gesture pair includes a previous gesture that is earlier in the time order and a next gesture that is later in the time order; a motion sub-information determination unit, used to determine the motion sub-information represented by the continuous gesture pair for each continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair; and a motion information determination unit, used to determine the information sequence formed by each motion sub-information in time order as the motion information commonly represented by each gesture.

[0191] In one embodiment, the motion sub-information determination unit is specifically used to: perform palm orientation recognition on a continuous gesture pair to obtain the palm orientation of the next gesture and the previous gesture in the continuous gesture pair; determine the angle change information of the next gesture relative to the previous gesture when each palm orientation meets the angle change condition; and determine the motion sub-information represented by the continuous gesture pair based on the rotation information represented by the angle change information.

[0192] In one embodiment, the motion sub-information determination unit is specifically used to: determine the palm position of the next gesture and the previous gesture in a continuous gesture pair;

[0193] Under the condition that the position change conditions are met at each hand position, determine the position change information of the next gesture relative to the previous gesture;

[0194] Based on the movement information represented by position change information, determine the motion sub-information represented by continuous gesture pairs.

[0195] In one embodiment, the collaborative robot control device further includes a motion information determination module, used to: determine the hand movement distance and hand movement direction represented by the position change information; and determine the robot end effector movement distance corresponding to the hand movement distance and the robot end effector movement direction corresponding to the hand movement direction as the motion information represented by the position change information.

[0196] In one embodiment, the collaborative robot control device further includes a motion information determination module, which is used to: determine the hand movement direction represented by the position change information and the duration of the next gesture in the gesture sequence; and determine the robot end-effector movement direction corresponding to the hand movement direction and the robot end-effector continuous movement time corresponding to the duration as the motion information represented by the position change information.

[0197] Each module in the aforementioned collaborative robot control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0198] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a collaborative robot control method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0199] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0200] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0204] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A collaborative robot control method, characterized in that, The method includes: In response to a gesture control event triggered by the collaborative robot, hand image information is acquired within a defined area; the hand image information includes hand images from at least two time frames; Gesture recognition is performed on each of the hand images to determine the gesture corresponding to each time frame; The gestures are sorted according to the time order of each time frame to obtain a gesture sequence; Identify consecutive gesture pairs in the gesture sequence that contain gesture changes; the consecutive gesture pairs include the previous gesture that appears earlier in the time sequence and the next gesture that appears later in the time sequence; For each of the consecutive gesture pairs, the motion sub-information represented by the consecutive gesture pair is determined based on the change of the next gesture relative to the previous gesture in the consecutive gesture pair; The information sequence formed by arranging the various motion sub-informations in chronological order is determined as the motion information commonly represented by the gestures; the motion information includes multiple motion sub-informations. The collaborative robot's movements are controlled according to the motion information.

2. The method according to claim 1, characterized in that, The method further includes: Acquire an initial hand image within a defined area, perform gesture recognition on the initial hand image, and determine the initial gesture represented by the initial hand image; If the initial gesture is the control start gesture of the collaborative robot, a gesture control event for the collaborative robot is triggered.

3. The method according to claim 1, characterized in that, The step of determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes: Perform palm orientation recognition on the continuous gesture pair to obtain the palm orientation of the next and previous gestures in the continuous gesture pair; If the angle change conditions are met in each of the aforementioned hand positions, the angle change information of the next gesture relative to the previous gesture is determined; Based on the rotation information represented by the angle change information, the motion sub-information represented by the continuous gesture pair is determined.

4. The method according to claim 1, characterized in that, The step of determining the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair includes: Determine the palm position of the next and previous gestures in the continuous gesture pair; If each of the aforementioned hand positions satisfies the position change condition, determine the position change information of the next gesture relative to the previous gesture; Based on the movement information represented by the position change information, the motion sub-information represented by the continuous gesture pair is determined.

5. The method according to claim 4, characterized in that, The method further includes: Determine the hand movement distance and direction represented by the position change information; The robot end effector movement distance corresponding to the hand movement distance and the robot end effector movement direction corresponding to the hand movement direction are determined as the movement information represented by the position change information.

6. The method according to claim 4, characterized in that, The method further includes: Determine the direction of hand movement represented by the position change information, and the duration of the next gesture in the gesture sequence; The robot end effector movement direction corresponding to the hand movement direction and the robot end effector continuous movement time corresponding to the duration are determined as the movement information represented by the position change information.

7. A collaborative robot control device, characterized in that, The device includes: The information acquisition module is used to acquire hand image information within a set area in response to a gesture control trigger event for the collaborative robot; the hand image information includes hand images from at least two time frames; The gesture determination module is used to perform gesture recognition on each of the hand images and determine the gesture corresponding to each of the time frames. The motion information determination module includes: a gesture sequence acquisition unit, used to sort the gestures according to the time order of each time frame to obtain a gesture sequence; a continuous gesture pair determination unit, used to determine continuous gesture pairs with gesture changes in the gesture sequence; the continuous gesture pair includes a previous gesture that appears earlier in the time order and a next gesture that appears later in the time order; a motion sub-information determination unit, used to determine the motion sub-information represented by the continuous gesture pair based on the change of the next gesture relative to the previous gesture in the continuous gesture pair; and a motion information determination unit, used to determine the information sequence formed by the motion sub-information in time order as the motion information commonly represented by the gestures; the motion information includes multiple motion sub-information items. The motion control module is used to control the actions of the collaborative robot according to the motion information.

8. The apparatus according to claim 7, characterized in that, The device further includes a control start gesture recognition module, used for: Acquire an initial hand image within a defined area, perform gesture recognition on the initial hand image, and determine the initial gesture represented by the initial hand image; If the initial gesture is the control start gesture of the collaborative robot, a gesture control event for the collaborative robot is triggered.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images