A human-computer interaction method based on deep learning and image processing for arbitrary objects

Abstract

The present invention discloses a human-computer interaction method for arbitrary objects based on deep learning and image processing. First, the MediaPipe model is used to detect the coordinates and coordinate relationships of the interactor's body nodes to determine the interactor's state. Then, according to the different states of the interactor, a saliency and edge detection algorithm or a region growing algorithm is used to determine the potential task target area. Then, the size of the target area is calculated using OpenCV image processing methods and depth image information, and the object's posture is calculated through pixel-to-world coordinate system conversion. Then, the size and fill of the target area are used to determine whether the interactor is holding an object. The position of the interactor's hand, the size of the object, and the size of the robot arm gripper are used to determine whether interaction is possible. Finally, the robot completes the unknown object human-computer interaction task based on the obtained object position and posture. This method realizes the acquisition of the coordinates and posture of unknown objects in the scene, enabling the robot to face any object and perform human-computer interaction operations safely, stably, and accurately without training.

Description

Technical Field

[0001] The present invention discloses a human-computer interaction method based on deep learning and image processing for arbitrary objects, belonging to the technical field of human-computer interaction. Background Art

[0002] With the continuous development of intelligent robotics, human-computer interaction plays a crucial role in future technological advancements. Effective interaction between robots and their collaborators can effectively improve production efficiency and enhance product quality. Object detection is a fundamental problem in computer vision, present in every aspect of our lives, such as facial recognition, smart transportation, and industrial inspection, providing significant convenience and efficient technical support. Although object detection technology has been widely used and has achieved impressive results, it still faces many challenges that warrant further investigation. Commonly used object detection technologies, such as the YOLO family of networks, SSD, and Faster RCNN, can accurately determine the location and type of objects. However, these networks are limited in the types of objects they can recognize. Furthermore, they can become difficult to identify the same object after being rotated a certain angle, and are prone to misidentification when object characteristics (such as color and texture) are similar. In this context, researchers have applied salient object detection technology to human-computer interaction, which effectively addresses these challenges and meets the requirements for determining the spatial position and pose measurement of any unknown object in human-computer interaction.

[0003] With the continuous development of artificial intelligence technology, the use of object detection networks for human-computer interaction is being applied in more scenarios, such as smart school cafeterias and access control systems, as well as garbage detection and sorting. When facing these scenarios, robots should have a certain degree of flexibility. Sometimes they need to recognize and interact with any object in any posture during human-computer interaction. Existing network detection technology requires massive amounts of data to identify a specific type and requires an accuracy rate of over 95%. In addition, when adding new types to the existing model, modifications must be made to the original network, which is time-consuming and cumbersome. Therefore, it is very meaningful to invent a detection network that requires fewer resources and has unlimited recognition areas and objects for human-computer interaction, thereby expanding the functionality of existing object recognition technology. Summary of the Invention

[0004] The purpose of the present invention is to provide a human-computer interaction method based on deep learning and image processing for arbitrary objects, so that robots can perform safe, stable and accurate interactive operations on any object without training. The present invention uses the MediaPipe model to detect the coordinates of the interactor's body nodes and judge the interactor's status based on the coordinates. The different status information of the interactor is used to adopt saliency detection, edge detection screening algorithm or improved region growing algorithm to obtain the potential task target area. Then, the pixel size and rotation angle of the target area image are obtained through OpenCV image processing. Then, the area and fill degree of the target area are used to determine whether the interactor is holding an object. After that, the interactivity and ideal interaction position are determined using the human hand position information and the target area information, and the rotation angle of the target object area is obtained using the depth image. Finally, the robot completes the unknown object interaction task based on the obtained object position and posture, regardless of the type.

[0005] The problem to be solved by the present invention is achieved by the following technical solutions:

[0006] A human-computer interaction method based on deep learning and image processing for arbitrary objects, the specific steps of the interaction method are as follows:

[0007] Step S10, obtaining a color image that matches the depth image based on the conversion relationship between the color and depth cameras and the world coordinates, and storing the matching relationship to speed up the conversion;

[0008] Step S20, filtering and repairing the depth image;

[0009] Step S30: Using the MediaPipe human skeleton joint recognition model to obtain the pixel coordinates of the interactor's human skeleton joints, and judging different postures of the human body relative to the camera: sideways and facing the camera based on the coordinates and distance information between the detected nodes;

[0010] Step S40: When the detection result is that the camera is facing sideways, a saliency detection neural network is used to detect the human-computer interaction scene in real time, obtain a real-time salient area of ​​human-computer interaction, and filter the network output result to optimize the result;

[0011] Step S50: When the detection result is side-on, the saliency detection network output is filtered for contours and specific outer contour midpoints based on the MediaPipe model detection result to obtain the outer contour of the object area, determine the interaction intention, and collect potential objects and human hands.

[0012] Step S60: When the detection result is that the object is facing the camera, a guided region growing algorithm is used to collect object regions on the depth image;

[0013] Step S70, using OpenCV image processing methods to obtain the minimum bounding rectangle of the hand and object area, obtain the object area, and obtain the image rotation angle, pixel width, and height;

[0014] Step S80, determining an area threshold and a filling threshold based on the hand information, and calculating corresponding data to determine whether the interacting person is holding an object;

[0015] Step S90: When determining whether the interactor is holding an object, the palm area is determined using the interacting hand position to determine whether human-machine interaction is possible, the ideal gripping position of the manipulator when interaction is possible is determined, and the determined key points are converted to the world coordinate system to obtain the rotation angle of the object around the X and Y axes of the world coordinate system;

[0016] In step S100, the object's posture information and the gripping position of the mechanical claw are transmitted from the visual processing end to the robot control end to guide the robot to complete the unknown object interaction task.

[0017] Preferably, the specific process of step S10 is as follows:

[0018] Step S101: Install a Kinect v2 camera on one side of a work platform, observe the resulting image, and select a suitable placement to ensure that the camera can capture the human body;

[0019] Step S102: Use the Zhang Zhengyou calibration method to calibrate the Kinect v2 camera to obtain the camera internal and external parameter matrix K c 、

[0020] Step S103: Call the official Kinect library function to obtain the depth camera internal parameter K d According to the hardware position relationship between the depth camera and the color camera, the external parameters of the depth camera are obtained from the external parameters of the color camera.

[0021] Step S104: Obtain the conversion relationship between the color camera and the 3D world coordinate system through the internal and external parameters of the color camera, obtain the conversion relationship between the depth camera and the 3D world coordinate system through the depth camera, and perform the conversion using the 3D world coordinate system as a constant to obtain the coordinate system conversion matrix of the depth camera and the color camera;

[0022] In step S105, the multiplication result of the pixel coordinates and the transformation matrix is ​​recorded and stored, adaptive translation is performed according to the actual matching effect to repair the error, the matching is accelerated using the jit function of the Numba library, and a color image matching the depth image is obtained in a loop.

[0023] Preferably, the specific process of step S20 is as follows:

[0024] Step S201, filtering out invalid depth points in the depth image, and then performing joint bilateral filtering;

[0025] Step S202, performing median filtering on the joint bilateral filtering result;

[0026] Step S203: performing image opening operation on the median filtering result.

[0027] Preferably, the specific process of step S30 is as follows:

[0028] Step S301: Input the mapped color image into the MediaPipe model for posture detection to obtain the coordinates of the wrists and shoulders;

[0029] Step S302: Using the obtained pixel coordinates of the joint points, the distance between the two shoulders is calculated to determine the orientation of the human body relative to the camera: sideways or facing the camera.

[0030] Step S303: When the camera is facing sideways, a salient region extraction and image processing algorithm is subsequently applied; when the camera is facing straight on, an improved region growing algorithm is subsequently applied.

[0031] Preferably, the specific process of step S40 is as follows:

[0032] Step S401: Input the converted small color image and the restored depth image into a saliency detection neural network to obtain a neural network output;

[0033] Step S402: discard the areas with low confidence in the output results, select a suitable threshold and perform binarization processing on the network output results;

[0034] In step S403, an opening operation is performed on the image obtained after the binary processing to smooth and remove noise, thereby obtaining a more reliable saliency result.

[0035] Preferably, the specific process of step S50 is as follows:

[0036] Step S501, performing contour detection on the processed saliency binary image to obtain all contours and contour points of each contour;

[0037] Step S502: determine the potential interaction hand based on the wrist coordinates of both hands, and then determine whether the interaction intention exists based on the distance between the potential interaction hand and the chest;

[0038] Step S503, taking the wrist position coordinates of the interacting hand as a base point, determining the direction according to the positional relationship between the interacting person's body and the interacting person's arm, and filtering out the contour nodes on the opposite side of the base point from each contour;

[0039] Step S504 , calculating the pixel coordinates of the midpoints of the remaining contours, comparing the distances from the midpoints to the wrist of the interactive wrist, and determining the area where the object is located.

[0040] Preferably, the specific process of step S60 is as follows:

[0041] Step S601: When the detection result is that the hands are facing the camera, the distance between the wrists of the hands and the robotic arm is compared to select a potential interactive hand;

[0042] Step S602: comparing the distance between the interacting hand and the chest to analyze whether the interactor has the intention to interact;

[0043] Step S603 : For the hand with interaction intention, a base point is selected according to the pixel coordinates of the interacting hand, and a guided region growing algorithm is used to grow the depth image from the base point to collect the target object region.

[0044] Preferably, the specific process of step S70 is as follows:

[0045] Step S701: Using the previously obtained object outline or object area, use the OpenCV image processing algorithm to calculate the minimum outer rectangle of the area as the object's anchor box, and obtain the width, height, rotation angle, and center coordinates of the rectangle;

[0046] Step S702: Calculate the angle between the longer side of the rectangle and the vertical direction based on the circumscribed rectangle size information and the rotation angle of the rectangular image;

[0047] Step S703: intercept a rectangular area and store it, and set the pixel values ​​of the three channels of the pixels in the object area to 0.

[0048] Preferably, the specific process of step S80 is as follows:

[0049] Step S801, calculating the pixel area of ​​the interactor's palm according to the body proportions to establish an area threshold;

[0050] Step S802: Calculate the ratio of the number of pixels and the area of ​​the palm of the interactor facing the camera and stretched to the maximum extent for multiple times, take the average value as the fill threshold and record it;

[0051] Step S803: Calculate the pixel area and fill level of the bounding rectangle during interaction. Calculate the pixel area of ​​the rectangle based on the width and height of the bounding rectangle.

[0052] Step S804: Compare the actual area and the filling degree with corresponding thresholds to determine the possibility that the interactor is holding an object.

[0053] Preferably, the specific process of step S90 is as follows:

[0054] Step S901, using the wrist position and palm size to determine the palm area range;

[0055] Step S902: determining the upper and lower positions of the object grasped by the interacting hand based on the positional relationship between the center of the palm and the center of the rectangle;

[0056] Step S903: determining the midpoint of the boundary between the palm area and the interactive area based on the coordinates of the palm center and the coordinates of the center of the object area;

[0057] Step S904: Determine the coordinates of the boundary midpoint of the area suitable for the gripper to grasp based on the included angle and the position of the object being grasped by the hand;

[0058] Step S905 , obtaining a point close to the corresponding boundary and inside the object based on the rectangular area and the depth image information;

[0059] Step S906: convert the two points into a three-dimensional world coordinate system, calculate the distance between the two points, compare it with the size of the robot arm gripper, and determine whether the interactive task can be performed;

[0060] In step S907 , the two points converted to the world coordinate system are projected in the YOZ and XOZ directions respectively to calculate the rotation angles of the object around the X and Y directions.

[0061] Preferably, the specific process of step S100 is as follows:

[0062] Step S1001: The object pose and gripping position acquired by the visual processing end are transmitted to the robot control end via TCP / IP communication.

[0063] Step S1002: After the robot control terminal receives the information, the robot moves to the specified position with a specified posture and completes the grasping or following motion task according to the interactivity information.

[0064] The present invention has the following beneficial effects compared with the prior art:

[0065] 1. Currently, most robot human-machine interaction systems can only sort a limited number of target objects. Furthermore, recognition performance is affected to some extent by objects being obstructed by hands or rotated at an angle, and adding new identifiable categories is cumbersome. However, the method of this invention enables robots to identify, locate, and interact with any unrecognized object.

[0066] 2. The human-computer interaction method proposed in the present invention can not only locate objects in the scene, but also analyze the rotation angles of the objects around the X and Y axes in a low-cost and relatively reliable manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1It is the overall framework diagram of the robot unknown object interaction method of the present invention.

[0068] Figure 2 It is a schematic diagram of the coordinate system of the present invention. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, the first embodiment of the present invention provides a human-computer interaction method based on deep learning and image processing for arbitrary objects on the basis of the existing technology, including the following steps:

[0071] S10 obtains a color image that matches the depth image based on the conversion relationship between the color and depth cameras and the world coordinates, and stores the matching relationship to speed up the matching process. The specific steps are as follows:

[0072] Step S101: Install a Kinect v2 camera on one side of a work platform, observe the resulting image, and select a suitable placement to ensure that the camera can capture the human body.

[0073] Step S102: Use the Zhang Zhengyou calibration method to calibrate the Kinect v2 camera to obtain the camera internal and external parameter matrix K c 、

[0074] Step S103: Call the official Kinect library function to obtain the depth camera internal parameter K d According to the hardware position relationship between the depth camera and the color camera, the external parameters of the depth camera are obtained from the external parameters of the color camera. The conversion formula is:

[0075]

[0076] Step S104: Obtain the conversion relationship between the color camera and the 3D world coordinate system through the internal and external parameters of the color camera, obtain the conversion relationship between the depth camera and the 3D world coordinate system through the depth camera, and perform the conversion using the 3D world coordinate system as a constant to obtain the coordinate system conversion matrix of the depth camera and the color camera. Let the coordinates of the point P in the pixel coordinate system be p =(u,v,1) T , the point in the world coordinate system is P = (X, Y, Z, 1) T , we can get the conversion relationship:

[0077]

[0078] Since the depth camera and the color camera are on the same plane, it can be considered that d c =d d .

[0079] From equations (1) and (2), we can get the transformation matrix from depth image to color image in the same world coordinate system:

[0080]

[0081] The conversion formula from a point on the depth image to the color image is:

[0082] P p_c =T dc *P p_d (4)

[0083] In step S105, the result of multiplying the pixel coordinates by the transformation matrix is ​​recorded and stored. Based on the actual matching results, adaptive translation is performed to correct errors. The matching is accelerated using the JIT function of the Numba library. This loop generates a color image that matches the depth image. This acceleration function allows for higher frame rates and better real-time performance.

[0084] Step S20: filter and repair the depth image. The specific steps are as follows:

[0085] In step S201, invalid depth points in the depth image are filtered out, and then joint bilateral filtering is performed. A reliable depth range can be selected according to the actual application, and the depth image pixels outside the range are set to zero before joint bilateral filtering is performed.

[0086] Step S202: performing median filtering on the joint bilateral filtering result.

[0087] Step S203: perform image opening operation on the median filtering result to smooth the boundary and complete the restoration of the depth image.

[0088] Step S30: Use the MediaPipe human skeleton joint recognition model to obtain the pixel coordinates of the interactor's human skeleton joints, and determine the different postures of the human body relative to the camera: sideways, facing, etc. based on the coordinates and distance information between the detected nodes.

[0089] The specific steps are as follows:

[0090] In step S301, the mapped color image is input into the MediaPipe model for posture detection to obtain the coordinates of the wrists, shoulders, etc.

[0091] In step S302, the distance between the two shoulders is calculated using the obtained joint pixel coordinates, and the orientation of the human body relative to the camera is determined: sideways or facing the camera. The shoulder width of the interacting subject in the image is calculated using the shoulder pixel coordinates and the depth value. The height from the shoulders to the ground is calculated using formula (5), and the interacting subject's height is calculated using formula (6). The theoretical shoulder width and the actual calculated shoulder width are used to determine the interaction subject's shoulder rotation angle, and then the interaction subject's posture relative to the camera is determined.

[0092]

[0093] H=Z / 0.818 (6)

[0094] Step S303: When the camera is facing sideways, a salient region extraction and image processing algorithm is subsequently applied; when the camera is facing straight on, an improved region growing algorithm is subsequently applied.

[0095] Step S40: When the detection result is side-facing, the saliency detection neural network is used to detect the human-computer interaction scene in real time, obtain the real-time saliency area of ​​human-computer interaction, and filter the network output result to optimize the result. The specific steps are as follows:

[0096] In step S401, the converted small color image and the restored depth image are input into a saliency detection neural network to obtain a neural network output. The saliency detection neural network is used to detect the human-computer interaction scene, and the obtained network output includes human body, object areas, and other interference areas.

[0097] In step S402, regions with low confidence levels are discarded from the output, and an appropriate threshold is selected to binarize the network output. The saliency detection neural network outputs a grayscale image of the same size as the input. The grayscale value of each pixel represents the network's judgment of whether it belongs to a salient region. Therefore, an appropriate threshold is selected to discard regions with low confidence levels, excluding them from being considered salient regions.

[0098] Step S403: Perform an opening operation on the image obtained after binary processing to smooth and remove noise, and obtain a more reliable significance result. The formula is as follows:

[0099]

[0100] In step S50, when the detection result is side-on, the saliency detection network output is filtered for contours and specific outer contour midpoints based on the MediaPipe model detection results to obtain the outer contour of the object area, determine the interaction intention, and collect potential objects and human hands. The specific steps are as follows:

[0101] Step S501 : performing contour detection on the processed saliency binary image to obtain all contours and contour points of each contour.

[0102] In step S502, the potential interacting hand is determined based on the wrist coordinates of both hands, and the presence of interaction intent is determined based on the distance between the potential interacting hand and the chest. The hand coordinates are converted to the world coordinate system using the pixel coordinates and corresponding depth values ​​of both hands. The hand closest to the robotic arm in space is selected as the potential interacting hand. The midpoint of the two shoulders is approximately used as the chest position, and the distance from the interacting hand to the chest is calculated. Based on the calculated height of the interacting user, a threshold is selected to determine whether the wrist-to-chest distance is greater than the threshold. If it is greater than the threshold, the interacting user is considered to have interaction intent.

[0103] Step S503 uses the wrist coordinates of the interacting hand as the base point, determines the direction based on the positional relationship between the interacting hand's body and the interacting hand's arm, and removes any contour nodes on the opposite side of the base point from each contour. The target interaction direction is determined based on the positional relationship between the interacting hand's body and the interacting hand's arm, and all contour nodes in the direction from the interacting hand to the interacting hand's body and beyond are removed, thereby removing the interacting hand's body area and part of the surrounding area.

[0104] Step S504: Calculate the pixel coordinates of the midpoints of the remaining contours, compare the distances from each midpoint to the wrist of the interactive wrist, and determine the area where the object is located. The contour to which the midpoint with the smallest distance belongs is the contour of the object area. Therefore, the pixel coordinates of the midpoint of each closed contour need to be calculated. The distance to the nearest contour is D c =min(D1,D2,......,D n ).

[0105] Assume the coordinates of wrist base point are (u h ,v h ), the pixel distance calculation formula is as follows:

[0106]

[0107] Step S60: When the detection result is that the object is facing the camera, a guided region growing algorithm is used to collect object regions on the depth image. The specific steps are as follows:

[0108] Step S601: When the detection result is that the hands are facing the camera, the potential interaction hands are selected by comparing the distance between the wrists of the hands and the robotic arm. The positions of the hands in the world coordinate system are calculated based on the pixel coordinates and corresponding depth values ​​of the hands, and the hand closest to the robotic arm is selected as the potential interaction hand.

[0109] Step S602: Compare the distance between the interacting hand and the chest to analyze whether the interacting person has an intention to interact. Select an appropriate threshold based on the person's height to compare and determine whether there is an intention to interact.

[0110] Step S603 : For the hand with interaction intention, a base point is selected according to the pixel coordinates of the interacting hand, and a guided region growing algorithm is used to grow the depth image from the base point to collect the target object region.

[0111] Conventional region growing algorithms expand the eight adjacent points surrounding a pixel that satisfy the constraints, and repeat this process. This unrestricted expansion of depth image pixels that satisfy the depth difference can result in the final target object region including the torso of the interacting person, leading to problems such as misjudgment of the region, large size errors, distortion in rotation angle calculation, and inaccurate positioning. Therefore, based on the fact that the direction from the wrist to the object and the direction from the wrist to the palm during human interaction are approximately the same, the MediaPipe model is used to detect the coordinates of the wrist and palm, determine the pixel angle between the interacting hand's palm and wrist, and then determine the orientation of the interacting hand to determine the object's orientation relative to the wrist. The interaction's hand orientation is used to guide the depth image pixel growth method, ensuring that pixels grow only in that direction and adjacent directions, avoiding the inclusion of the torso within the object region and obtaining a more accurate object region.

[0112] Step S70: Based on the hand and object area, the OpenCV image processing method is used to obtain the minimum bounding rectangle of the area, obtain the object area, and obtain the image rotation angle, pixel width, and height. The specific steps are as follows:

[0113] Step S701: Using the previously obtained object outline or object area, use the OpenCV image processing algorithm to obtain the minimum outer rectangle of the area as the object's anchor box, and simultaneously obtain the width, height, rotation angle, and center coordinates of the rectangle.

[0114] Step S702: Calculate the angle between the longer side of the rectangle and the vertical direction based on the circumscribed rectangle's dimensions and the rectangular image's rotation angle. Determine the object's tilt direction based on the circumscribed rectangle's width (w) and height (h). Subtract the leftward tilt angle from 90 degrees to obtain the angle between the object and the vertical direction. The formula is as follows:

[0115]

[0116] Step S703: intercept a rectangular area and store it, and set the pixel values ​​of the three channels of the pixels in the object area to 0.

[0117] Step S80: Determine the area threshold and fill threshold based on the hand information, and calculate the corresponding data to determine whether the interactor is holding an object. The specific steps are as follows:

[0118] Step S801: Calculate the pixel area of ​​the interactor's palm according to the human body proportions to establish an area threshold. When the human palm is stretched to its maximum extent, its width is approximately equal to its length, and the length of the palm is approximately 60%-80% of the forearm length. At the same time, the length of the palm is approximately twice the width of the palm. Therefore, two area thresholds can be calculated. Let the pixel coordinates of the interactor's wrist and elbow be (x w ,y w ),(x e ,y e ), the pixel length of the forearm is L arm , then the area threshold calculation formula is:

[0119]

[0120] Step S802 calculates the ratio of the number of pixels to the area of ​​the person's palm facing the camera and stretched out to the maximum extent for multiple times. The average value is taken as the fill threshold and recorded. Referring to the previous step, with the person's hand stretched out to the maximum extent and facing the camera without any obstruction, a guided region growing algorithm is used to measure the ratio of the effective area of ​​the person's palm at maximum stretch to the area of ​​the bounding rectangle. The average value is taken as the fill threshold and recorded.

[0121] Step S803: Calculate the pixel area and fill factor of the bounding rectangle during interaction. Calculate the pixel area of ​​the rectangle based on the width and height of the bounding rectangle. Using the OpenCV image processing algorithm, calculate the number of pixels in the previously stored rectangle where all three channels are zero, as the effective area of ​​the object area. Calculate the ratio of this value to the rectangle area as the actual fill factor.

[0122] Step S804 compares the actual area and fill factor with corresponding thresholds to determine the likelihood that the user is holding an object. If the bounding rectangle area is greater than threshold 1, the user is deemed to be holding an object; if it is less than threshold 2, the likelihood of the user holding an object is low. If the bounding rectangle area is less than threshold 1 but greater than threshold 2, an auxiliary judgment is made based on the fill factor threshold. If the actual fill factor is greater than the fill factor threshold, the user is also deemed to be holding an object, thus avoiding omissions and misjudgments when interacting with small items.

[0123] Step S90: When determining whether the interactor is holding an object, the position of the interacting hand is used to determine the palm area, determine whether human-machine interaction can be completed, determine the ideal gripping position of the manipulator when interaction is possible, and use the determined key points to convert to the world coordinate system to calculate the rotation angle of the object around the X and Y axes of the world coordinate system. The specific steps are as follows:

[0124] Step S901 determines the palm area using wrist position and palm size. Assuming the interacting hand and the object are tilted at approximately the same angle, the palm area is determined by the angle between the image's circumscribed rectangle and the vertical. The pixel width of the palm is roughly calculated as twice the pixel length from the wrist to the palm. Within the rectangular area, the area starting at the palm and extending from half of this value from top to bottom is the palm area.

[0125] Step S902: Determine the upper and lower positions of the object being grasped by the interactive hand based on the positional relationship between the center of the palm and the center of the rectangle. The upper and lower positional relationship can be used to determine the interactive area that is more suitable for the gripper to grasp. The palm area and the interactive area are on opposite sides.

[0126] Step S903: Determine the midpoint of the boundary between the palm area and the interactive area based on the coordinates of the palm center and the center of the object area. box_c ,y box_c ), the coordinates of the palm center are (x h ,y h ), the midpoint coordinates on the adjacent boundary are (x area_c ,y area_c ), the angle between the circumscribed rectangle and the vertical direction is θ, and the pixel width of the palm is w h The length from the center of the rectangle to the palm area is h, and the length of the interactive area is h+H box / 2. Taking the case where the hand is grasping the lower part of an object and the object is tilted to the right in the image as an example, the length h from the center of the rectangle to the palm area is calculated as:

[0127] h=(y h -y box_c ) / cos(θ)-w h / 2 (11)

[0128] From this, the coordinates of the midpoints on the adjacent boundaries can be calculated:

[0129]

[0130] Step S904: Determine the coordinates of the midpoint of the boundary of the area suitable for the gripper to grasp based on the angle and the position of the object being grasped by the hand. If the hand is grasping the upper part of the object, the coordinates of the lower midpoint of the rectangle are used; otherwise, the coordinates of the upper midpoint of the rectangle are used.

[0131] In step S905, a point near the corresponding boundary and within the object is found based on the rectangular region and the depth image information. Pixels on the border are outside the object region, and their world coordinates differ significantly from those of the object. Therefore, it is necessary to iterate within the image based on the angle of the rectangular region until the depth value of the point is close to the average depth value of the object. This determines that the pixel is within the object region.

[0132] Step S906: Convert the two points to a 3D world coordinate system, calculate the distance between them, and compare it with the size of the robot's gripper to determine whether the interactive task is possible. The robot's gripper is measured, and a buffer distance is maintained. If the interactive area is greater than this size, interaction is possible; otherwise, interference will occur.

[0133] Step S907: Project the two points converted to the world coordinate system in the YOZ and XOZ directions respectively to calculate the rotation angle of the object around the X and Y directions. Convert the two points determined from pixel coordinates to the world coordinate system, and let the point with the relatively small Z coordinate be (X1, Y1, Z1) and the point with the relatively small Z coordinate be (X2, Y2, Z2). Calculate the length of the vector after projection onto the YOZ and XOZ planes using formula (13), and then calculate the rotation angle of the object around the X and Y directions using formula (14). The formula is as follows:

[0134]

[0135]

[0136] Step S100: The object's posture information and the gripping position of the mechanical claw are transmitted from the visual processing end to the robot control end to guide the robot to complete the unknown object interaction task. The specific steps are as follows:

[0137] Step S1001: The object posture and gripping position acquired by the visual processing end are transmitted to the robot control end via TCP / IP communication.

[0138] Step S1002: After the robot control terminal receives the information, the robot moves to the specified position with a specified posture and completes the grasping or following motion task according to the interactivity information.

[0139] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and exemplary embodiments. They can be applied to a variety of fields suitable for the present invention. Further modifications will be readily apparent to those skilled in the art. Therefore, the present invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims

1. A human-computer interaction method based on deep learning and image processing for arbitrary objects, characterized in that: The specific steps of the interaction method are as follows: Step S10, obtaining a color image that matches the depth image based on the conversion relationship between the color and depth cameras and the world coordinates, and storing the matching relationship to speed up the conversion; Step S20, filtering and repairing the depth image; Step S30: Using the MediaPipe human skeleton joint recognition model to obtain the pixel coordinates of the interactor's human skeleton joints, and judging different postures of the human body relative to the camera: sideways and facing the camera based on the coordinates and distance information between the detected nodes; Step S40: When the detection result is that the camera is facing sideways, a saliency detection neural network is used to detect the human-computer interaction scene in real time, obtain a real-time salient area of ​​human-computer interaction, and filter the network output result to optimize the result; Step S50: When the detection result is side-on, the saliency detection network output is filtered for contours and specific outer contour midpoints based on the MediaPipe model detection result to obtain the outer contour of the object area, determine the interaction intention, and collect potential objects and human hands. Step S60: When the detection result is that the object is facing the camera, a guided region growing algorithm is used to collect object regions on the depth image; Step S70, using OpenCV image processing methods to obtain the minimum bounding rectangle of the hand and object area, obtain the object area, and obtain the image rotation angle, pixel width, and height; Step S80, determining an area threshold and a filling threshold based on the hand information, and calculating corresponding data to determine whether the interacting person is holding an object; Step S90: When determining whether the interactor is holding an object, the palm area is determined using the interacting hand position to determine whether human-machine interaction can be completed, the ideal gripping position of the manipulator when interaction is possible is determined, and the determined key points are converted to the world coordinate system to obtain the rotation angle of the object around the X and Y axes of the world coordinate system; In step S100, the object's posture information and the gripping position of the mechanical claw are transmitted from the visual processing end to the robot control end to guide the robot to complete the unknown object interaction task.

2. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S10 is as follows: Step S101: Install a Kinect v2 camera on one side of a work platform, observe the resulting image, and select a suitable placement to ensure that the camera can capture the human body; Step S102: Use Zhang Zhengyou calibration method to calibrate the Kinect v2 camera to obtain the color camera internal and external parameter matrix Kc, Step S103: Call the official Kinect library function to obtain the depth camera intrinsic parameter K d , and obtain the depth camera extrinsic parameter from the color camera extrinsic parameter according to the hardware position relationship between the depth camera and the color camera; Step S104: Obtain the conversion relationship between the color camera and the 3D world coordinate system through the internal and external parameters of the color camera, obtain the conversion relationship between the depth camera and the 3D world coordinate system through the depth camera, and perform the conversion using the 3D world coordinate system as a constant to obtain the coordinate system conversion matrix of the depth camera and the color camera; In step S105, the multiplication result of the pixel coordinates and the transformation matrix is ​​recorded and stored, adaptive translation is performed according to the actual matching effect to repair the error, the matching is accelerated using the jit function of the Numba library, and a color image matching the depth image is obtained in a loop.

3. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S20 is as follows: Step S201, filtering out invalid depth points in the depth image, and then performing joint bilateral filtering; Step S202, performing median filtering on the joint bilateral filtering result; Step S203: performing image opening operation on the median filtering result.

4. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S30 is as follows: Step S301: Input the mapped color image into the MediaPipe model for posture detection to obtain the coordinates of the wrists and shoulders; Step S302: Using the obtained pixel coordinates of the joint points, the distance between the two shoulders is calculated to determine the orientation of the human body relative to the camera: sideways or facing the camera. Step S303: When the image is facing the camera sideways, a salient region extraction and image processing algorithm is subsequently applied; when the image is facing the camera head-on, an improved region growing algorithm is subsequently applied.

5. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S40 is as follows: Step S401: Input the converted small color image and the restored depth image into a saliency detection neural network to obtain a neural network output; Step S402: discard the areas with low confidence in the output results, select a suitable threshold and perform binarization processing on the network output results; In step S403, an opening operation is performed on the image obtained after the binary processing to smooth and remove noise, thereby obtaining a more reliable saliency result.

6. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S50 is as follows: Step S501, performing contour detection on the processed saliency binary image to obtain all contours and contour points of each contour; Step S502: determine the potential interaction hand based on the wrist coordinates of both hands, and then determine whether the interaction intention exists based on the distance between the potential interaction hand and the chest; Step S503, taking the wrist position coordinates of the interacting hand as a base point, determining the direction according to the positional relationship between the interacting person's body and the interacting person's arm, and filtering out the contour nodes on the opposite side of the base point from each contour; Step S504 , calculating the pixel coordinates of the midpoints of the remaining contours, comparing the distances from the midpoints to the wrist of the interactive wrist, and determining the area where the object is located.

7. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S60 is as follows: Step S601: When the detection result is that the hands are facing the camera, the distance between the wrists of the hands and the robotic arm is compared to select a potential interactive hand; Step S602: comparing the distance between the interacting hand and the chest to analyze whether the interactor has the intention to interact; Step S603 : For the hand with interaction intention, a base point is selected according to the pixel coordinates of the interacting hand, and a guided region growing algorithm is used to grow the depth image from the base point to collect the target object region.

8. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S70 is as follows: Step S701: Using the previously obtained object area, use the OpenCV image processing algorithm to find the minimum outer rectangle of the area as the object's anchor box, and obtain the width, height, rotation angle, and center coordinates of the rectangle; Step S702: Calculate the angle between the longer side of the rectangle and the vertical direction based on the circumscribed rectangle size information and the rotation angle of the rectangular image; Step S703: intercept a rectangular area and store it, and set the pixel values ​​of the three channels of the pixels in the object area to 0.

9. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S80 is as follows: Step S801, calculating the pixel area of ​​the interactor's palm according to the body proportions to establish an area threshold; Step S802: Calculate the ratio of the number of pixels and the area of ​​the palm of the interactor facing the camera and stretched to the maximum extent for multiple times, take the average value as the fill threshold and record it; Step S803, calculating the pixel area and fill degree of the bounding rectangle during interaction, and calculating the pixel area of ​​the rectangle according to the width and height of the bounding rectangle; Step S804: Compare the actual area and the filling degree with corresponding thresholds to determine the possibility that the interactor is holding an object.

10. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S90 is as follows: Step S901, using the wrist position and palm size to determine the palm area range; Step S902: determining the upper and lower positions of the object grasped by the interacting hand based on the positional relationship between the center of the palm and the center of the rectangle; Step S903: determining the midpoint of the boundary between the palm area and the interactive area based on the coordinates of the palm center and the coordinates of the center of the object area; Step S904: Determine the coordinates of the boundary midpoint of the area suitable for the gripper to grasp based on the included angle and the position of the object being grasped by the hand; Step S905 , obtaining a point close to the corresponding boundary and inside the object based on the rectangular area and the depth image information; Step S906: Convert the two points to a three-dimensional world coordinate system, calculate the distance between the two points, and compare it with the size of the robot arm gripper to determine whether the interactive task can be performed; In step S907 , the two points converted to the world coordinate system are projected in the YOZ and XOZ directions respectively to calculate the rotation angles of the object around the X and Y directions.

11. The method of human-computer interaction based on deep learning and image processing for arbitrary objects according to claim 1, characterized in that: The specific process of step S100 is as follows: Step S1001: Transmitting the object pose and gripping position acquired by the visual processing end to the robot control end via TCP / IP communication; Step S1002: After the robot control terminal receives the information, the robot moves to the specified position with a specified posture and completes the grasping or following motion task according to the interactivity information.

Images