A humanoid robot control method and system based on ROS

Abstract

The present invention discloses a humanoid robot control method and system based on ROS, the method comprising: real-time acquisition of motion state data of the humanoid robot; acquisition of original image data around the humanoid robot and segmentation processing to obtain independent left-eye image and right-eye image; stereo correction of the independent left-eye image and right-eye image to align the two on the same plane to obtain a corrected left-eye image and a corrected right-eye image; identification of a target object and measurement of the distance of the target object according to the corrected left-eye image and the corrected right-eye image; control of the humanoid robot to execute planned gaits and various predefined actions; output of preset voice information to the outside world through a speaker device. The system comprises: a motion state data acquisition module, a visual module, a correction module, a target recognition module, a motion control module and a voice module. The present invention can realize accurate target object recognition and ranging, and improve the robot's behavior planning and execution capabilities in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of humanoid robots, and in particular to a humanoid robot control method and system based on ROS. Background Art

[0002] As an important branch of robotics technology, humanoid robots have shown great potential and application value in the fields of industry, medicine, education, etc. in recent years. In order to achieve complex movements and behaviors, humanoid robots need a set of efficient control methods to process various sensor data, perform motion planning, and interact with the outside world. As an open source robotics software platform, the Robot Operating System (ROS) provides a wealth of tools and libraries that can greatly simplify the process of robot development.

[0003] A comprehensive analysis of the current status of research on humanoid robot software control at home and abroad shows that its development process has gone through four stages: centralized control, modular control, distributed control architecture, and the use of robot software platforms. However, the current humanoid robot control system still has the following major problems: (1) The robot software platform is highly specialized and has poor portability: The robot software platforms currently used in humanoid robot software control systems are mostly proprietary platforms developed by various research institutions for specific series of robots or certain fields. They are difficult to use in the software control systems of other humanoid robots. Therefore, the system is not very versatile and has poor portability. (2) Low code reuse rate: With the development and complexity of humanoid robot technology, different humanoid robots have a variety of different hardware, and different programs need to be designed for different hardware, resulting in a low code reuse rate. (3) Inefficient sensor data processing: Existing control methods often lack a unified data processing framework when processing data from various sensors (such as gyroscopes, accelerometers, cameras, etc.). The synchronization, fusion and real-time processing of sensor data are the key to achieving high-precision control. However, due to the lack of efficient processing methods, existing systems often find it difficult to achieve ideal real-time performance and accuracy when processing complex motion state data and image data. (4) Lack of intelligent behavior control: The behavior control of humanoid robots needs to be highly intelligent to cope with complex and changing environments and tasks. However, the existing systems are not highly intelligent in terms of behavior control and lack effective behavior planning and decision-making mechanisms, which results in poor performance of robots when performing complex tasks and makes it difficult to achieve the expected task objectives. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present invention is to provide a humanoid robot control method and system based on ROS, which can realize accurate target object recognition and ranging, improve the robot's behavior planning and execution capabilities in complex environments, realize highly intelligent behavior control, and enable the robot to complete complex tasks autonomously.

[0005] A humanoid robot control method based on ROS, comprising:

[0006] The motion state data of the humanoid robot is collected in real time, wherein the motion state data includes at least one of angular velocity data and acceleration data.

[0007] The original image data around the humanoid robot is acquired and segmented to obtain independent left-eye images and right-eye images.

[0008] The independent left-eye image and the right-eye image are stereo-corrected so that the two are aligned on the same plane to obtain a corrected left-eye image and a corrected right-eye image.

[0009] According to the corrected left-eye image and the corrected right-eye image, a target object is identified and a distance of the target object is measured.

[0010] Control the humanoid robot to perform planned gaits and various predefined actions.

[0011] The preset voice information is output to the outside world through the speaker device.

[0012] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the obtaining of raw image data around the humanoid robot and segmentation processing to obtain independent left-eye images and right-eye images includes:

[0013] A stereo camera is used to obtain left and right eye images around the humanoid robot, and image synthesis is performed to obtain original image data.

[0014] The ROS image format conversion function package is started to convert the original image data into OpenCV image data.

[0015] Write and start a camera segmentation ROS node program for image segmentation, segment the OpenCV image data, and obtain independent OpenCV left-eye image and OpenCV right-eye image.

[0016] The ROS image format conversion function package is started again to convert the OpenCV left-eye image and the OpenCV right-eye image to obtain a mutually independent ROS left-eye image and a ROS right-eye image.

[0017] Its technical effects are: based on the ROS framework, using the ROS message passing mechanism, efficiently transmitting and storing image segmentation results, making full use of the ROS node management and data transmission mechanism, each image processing step can be run as an independent ROS node to achieve multi-threaded parallel processing; using the efficient image processing algorithm provided by OpenCV, the OpenCV image data is accurately segmented, effectively separating the left and right image data, while ensuring the processing speed, accurately separating the left and right image data, reducing image noise and distortion, and thus improving the accuracy of image segmentation.

[0018] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the independent left-eye image and the right-eye image are stereo-corrected so that the two are aligned on the same plane to obtain the corrected left-eye image and the corrected right-eye image, including:

[0019] The ROS image format conversion function package is started to convert the left-eye image and the right-eye image into an OpenCV left-eye image and an OpenCV right-eye image.

[0020] The image distortion correction function in OpenCV is called, and the internal reference matrix and distortion coefficient of the camera are used to perform distortion correction on the OpenCV left-eye image and the OpenCV right-eye image to eliminate the distortion effect and generate an image after distortion elimination.

[0021] Calling the binocular stereo correction function in OpenCV, using the correction mapping matrix, performing stereo correction on the dedistorted image to generate a corrected image;

[0022] The corrected image is cropped to remove invalid areas so that the left and right images are of the same size, thereby obtaining a cropped image.

[0023] The ROS image format conversion function package is started again to convert the cropped image data to obtain a corrected ROS left-eye image and a ROS right-eye image.

[0024] The technical effect is as follows: the image distortion is corrected by using the camera's intrinsic parameter matrix and distortion coefficients, and the distorted image is converted into a geometrically accurate image by remapping the pixels in the image, thereby eliminating the geometric distortion introduced by the lens, reducing the error caused by distortion in the image processing process, ensuring that the geometric relationship in the image is consistent with the actual scene, and improving the accuracy and reliability of image processing; by generating a correction mapping matrix, the left and right eye images are remapped to align the corrected images, and the errors caused by the camera position and angle deviation are eliminated. The left and right eye images after stereo correction are aligned on the same plane, ensuring that each pixel point has an accurate correspondence in the stereo matching process, and improving the accuracy of stereo matching.

[0025] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, identifying a target object and measuring a distance of the target object according to the corrected left-eye image and the corrected right-eye image comprises:

[0026] The corrected left-eye image and the corrected right-eye image are acquired.

[0027] The corrected left-eye image and the corrected right-eye image are preprocessed, and the preprocessing includes at least one of smoothing filtering and edge detection.

[0028] The area corresponding to the target object in the preprocessed image is extracted based on color features.

[0029] Perform shape recognition on the extracted area to determine the shape of the target object.

[0030] The centroid of the target object that meets the color feature and the shape feature is calculated to obtain the center coordinates of the target object.

[0031] The technical effects are: by combining color and shape features, the accuracy and reliability of target object recognition are improved; based on the corrected image and accurate center of mass calculation, high accuracy of distance measurement is ensured.

[0032] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the region corresponding to the target object in the preprocessed image is extracted based on color features, including:

[0033] The color range of the target object is defined using the HSV color space, and a mask image for identifying the color range that matches the target object is generated.

[0034] The preprocessed image is converted from the RGB color space to the HSV color space, and the mask image is used to extract the area that meets the color range in the converted HSV color space.

[0035] The technical effect is that the use of HSV color space to define the color range can better cope with the influence of lighting changes and shadows. The hue information of HSV color space is insensitive to brightness changes, and can more stably describe color features and improve the accuracy of color recognition. By generating a mask image to identify the area that meets the color range of the target object, the interference information in the background and environment is effectively removed to ensure that only the area related to the target object is extracted, reducing the possibility of misidentification and improving the reliability of the system.

[0036] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the step of performing shape recognition on the extracted area and determining the shape of the target object comprises:

[0037] Use OpenCV's contour detection function to perform shape recognition in the extracted area and find the contour of the target object.

[0038] The contour is fitted by a polygon fitting method to obtain the shape characteristics of the target object and determine the shape of the target object.

[0039] Its technical effect is: using OpenCV's contour detection function, the boundary of the target object can be accurately found, ensuring that shape recognition is based on clear contour information. Combined with the polygon fitting method, background noise and interference information can be effectively filtered out, the accuracy of target object detection can be enhanced, the geometric features of the target object can be accurately described, and the accuracy of shape recognition can be improved.

[0040] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the calculating the center of mass of the target object that meets the color characteristics and shape characteristics to obtain the center coordinates of the target object includes:

[0041] Get the coordinates of all points of the target object's outline.

[0042] Use OpenCV's moments function to calculate the geometric moments of the target object's outline.

[0043] The coordinates of the center of mass are calculated using the geometric moment to obtain the center coordinates of the target object.

[0044] The technical effect is: by calling the moments function of OpenCV, the geometric moment of the target object's contour is calculated. The geometric moment is a mathematical tool that describes the shape and centroid distribution of the image area, including the mass distribution of the image area, the centroid position and other information. According to the calculated geometric moment, the centroid coordinates of the target object are calculated using the corresponding formula, the center point of the target object is determined, and the precise center coordinates are provided.

[0045] In a preferred embodiment of the present invention, in the above ROS-based humanoid robot control method, controlling the humanoid robot to execute the planned gait and various predefined actions includes:

[0046] The ROS node is started, the control system of the humanoid robot is initialized, the robot model and action library are loaded, and the sensor data subscription is initialized.

[0047] According to the current task, plan the actions and gaits that the robot needs to perform, use the gait generation algorithm, and select the corresponding action sequence from the predefined action library.

[0048] The motion state data is fused using a sensor fusion algorithm to estimate the posture of the humanoid robot.

[0049] According to the result of posture estimation, the humanoid robot is adjusted and re-controlled in real time to execute the planned movements and gaits.

[0050] After the action is completed, adjust the robot joints, reset the posture, and stop the real-time collection of sensor data.

[0051] Its technical effects are: applying sensor fusion algorithms to fuse motion state data, effectively estimating the robot's posture, more accurately reflecting the robot's true state, and reducing the inaccuracy caused by sensor noise and errors; real-time adjustment and re-control of the robot to execute planned movements and gaits, so that the robot can cope with dynamically changing environments and task requirements, ensuring that the robot can maintain an efficient working state in a changing environment.

[0052] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the outputting of preset voice information to the outside world through a speaker device includes:

[0053] Subscribe or listen for messages or commands that trigger voice output.

[0054] Receive the message or instruction that triggers the voice output, analyze the content, and determine the specific voice information that needs to be output.

[0055] According to the voice information, a corresponding voice text is selected from a preset voice library, and the voice text is converted into a voice file using a speech synthesis function package in ROS.

[0056] Use the voice playback function package in ROS to play the voice file through the speaker device.

[0057] Its technical effects are: the mechanism of subscribing to or listening to trigger messages ensures that the system can respond to external events or instructions in real time and realize automated voice output, so that the robot can respond quickly to various instructions; the speech synthesis function package in ROS is used for text-to-speech conversion, which ensures that the generated voice files are of high quality and clear pronunciation, making the voice output more understandable and natural; the preset voice information is output through the speaker device, which improves the quality of interaction between the robot and the user.

[0058] A humanoid robot control system based on ROS, comprising:

[0059] The motion state data acquisition module is used to acquire the motion state data of the humanoid robot in real time, wherein the motion state data includes at least one of angular velocity data and acceleration data.

[0060] The visual module is used to obtain the original image data around the humanoid robot and perform segmentation processing to obtain independent left eye images and right eye images.

[0061] The correction module is used to perform stereo correction on the independent left-eye image and the right-eye image so that the two are aligned on the same plane to obtain a corrected left-eye image and a corrected right-eye image.

[0062] The target recognition module is used to recognize the target object and measure the distance of the target object according to the corrected left-eye image and the corrected right-eye image.

[0063] The motion control module is used to control the humanoid robot to perform planned gaits and various predefined actions.

[0064] The voice module is used to output preset voice information to the outside world through the speaker device.

[0065] The beneficial effects of the present invention are:

[0066] The present invention is based on the ROS framework and utilizes the message transmission mechanism of ROS to efficiently transmit and store image segmentation results, making full use of the node management and data transmission mechanism of ROS. Each image processing step can be run as an independent ROS node to realize multi-threaded parallel processing. The efficient image processing algorithm provided by OpenCV is utilized to accurately segment the OpenCV image data, effectively separate the left and right image data, accurately separate the left and right image data while ensuring the processing speed, reduce image noise and distortion, and thus improve the accuracy of image segmentation.

[0067] The present invention realizes accurate image correction and target recognition. By using the image distortion correction function and binocular stereo correction function of OpenCV, high-precision correction of left and right eye images is achieved. Distortion correction eliminates geometric distortion introduced by the lens. Stereo correction ensures image alignment and matching, improves the accuracy of image processing, effectively reduces errors caused by lens distortion and angle deviation, and ensures the accuracy of the stereo matching process. Combined with color features and shape recognition technology, through HSV color space processing and contour detection, the target object can be accurately extracted and its center of mass calculated, which improves the accuracy of target recognition and ensures the high accuracy of target object distance measurement.

[0068] The present invention realizes dynamic and flexible motion control. By combining the gait generation algorithm and the sensor fusion algorithm, it realizes the dynamic adjustment of the robot's gait and movement. By estimating the robot's posture in real time and adjusting the movement according to the changes in the environment and tasks, the system can maintain efficient operating performance in complex and dynamic environments, enhancing the robot's ability to adapt to environmental changes.

[0069] The present invention comprehensively utilizes the node management and data transmission mechanism of ROS and the image processing algorithm of OpenCV to optimize the performance in each link of data acquisition, processing, control and output. Accurate image processing and target recognition, dynamic motion control and natural voice interaction jointly improve the comprehensive performance of the entire robot system, making it more suitable for complex tasks in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 The figure is a flow chart of the humanoid robot control method based on ROS of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0073] Please refer to Figure 1The first embodiment of the present invention provides a humanoid robot control method based on ROS, which includes: collecting motion state data of the humanoid robot in real time, the motion state data including at least one of angular velocity data and acceleration data; acquiring original image data around the humanoid robot and performing segmentation processing to obtain independent left-eye image and right-eye image; performing stereo correction on the independent left-eye image and right-eye image to align the two on the same plane to obtain a corrected left-eye image and a corrected right-eye image; identifying a target object and measuring the distance of the target object according to the corrected left-eye image and the corrected right-eye image; controlling the humanoid robot to execute a planned gait and various predefined actions; and outputting preset voice information to the outside world through a speaker device.

[0074] Specifically, a gyroscope is used to collect angular velocity data of the humanoid robot; and an accelerometer sensor is used to collect acceleration data of the humanoid robot.

[0075] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the obtaining of raw image data around the humanoid robot and performing segmentation processing to obtain independent left-eye images and right-eye images includes: using a stereo camera to obtain left-eye and right-eye images around the humanoid robot, performing image synthesis, and obtaining raw image data; starting a ROS image format conversion function package to convert the raw image data into OpenCV image data; writing and starting a camera segmentation ROS node program for image segmentation, segmenting the OpenCV image data, and obtaining mutually independent OpenCV left-eye images and OpenCV right-eye images; and starting the ROS image format conversion function package again to convert the OpenCV left-eye image and the OpenCV right-eye image to obtain mutually independent ROS left-eye images and ROS right-eye images.

[0076] Since ROS has its own format and specifications for all data, and images in OpenCV are stored in the form of Mat matrices, there are certain differences between the image formats of the two. Therefore, when processing image segmentation, it is necessary to convert the original ROS image into an image in OpenCV format. ROS provides a function package cv_bridge with an OpenCV interface, which allows us to convert ROS images into OpenCV images so that we can use the OpenCV library to perform various processing on them. After the processing is completed, the processed image data is converted into a ROS image again, and then published through a topic to achieve image transmission between different ROS nodes.

[0077] Specifically, the method of starting the camera segmentation ROS node and segmenting the OpenCV image data includes:

[0078]

[0079]

[0080] The callback function processes the received image data and uses CvBridge to convert the ROS image message into an image in OpenCV format; it divides the composite image data into left and right eye images; it converts the segmented images back into ROS image messages and publishes them to the topics / camera / left_image and / camera / right_image; the listener function initializes the ROS node, subscribes to the original image data, and sets the publisher of the segmented image data.

[0081] Its technical effects are: based on the ROS framework, using the ROS message passing mechanism, efficiently transmitting and storing image segmentation results, making full use of the ROS node management and data transmission mechanism, each image processing step can be run as an independent ROS node to achieve multi-threaded parallel processing; using the efficient image processing algorithm provided by OpenCV, the OpenCV image data is accurately segmented, effectively separating the left and right image data, while ensuring the processing speed, accurately separating the left and right image data, reducing image noise and distortion, and thus improving the accuracy of image segmentation.

[0082] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the independent left-eye image and the right-eye image are stereo-corrected so that the two are aligned on the same plane to obtain the corrected left-eye image and the corrected right-eye image, including: starting the ROS image format conversion function package to convert the left-eye image and the right-eye image into an OpenCV left-eye image and an OpenCV right-eye image; calling the image distortion correction function in OpenCV, using the camera's intrinsic parameter matrix and distortion coefficient to perform distortion correction on the OpenCV left-eye image and the OpenCV right-eye image, eliminate the distortion effect, and generate an image after distortion elimination; calling the binocular stereo correction function in OpenCV, using the correction mapping matrix, stereo-correcting the image after distortion elimination, and generating a corrected image; cropping the corrected image, removing the invalid area, so that the left and right images are of the same size, and obtaining a cropped image; starting the ROS image format conversion function package again, converting the cropped image data, and obtaining a corrected ROS left-eye image and a ROS right-eye image.

[0083] Specifically, in image distortion correction, the obtained OpenCV left-eye image and OpenCV right-eye image are distorted to eliminate the distortion effect of the lens and generate a dedistorted image. The camera's intrinsic matrix and distortion coefficient are used as input, and the OpenCV undistort function is called to process the image. The specific methods include:

[0084]

[0085]

[0086] In binocular stereo correction, use cv2.stereoRectify to calculate the correction matrix, and align the corrected image to the same plane through cv2.initUndistortRectifyMap and cv2.remap to generate a stereo corrected image. The specific methods include:

[0087] #Assume the following are the calculated correction parameters

[0088] R1,R2,P1,P2,Q,valid_roi1,valid_roi2=cv2.stereoRectify(K1,dist_coeffs1,

[0089] K2,dist_coeffs2,

[0090] image_size,R,T)

[0091] #Calculate the mapping matrix for dedistortion and correction

[0092] map1x,map1y=cv2.initUndistortRectifyMap(K1,dist_coeffs1,R1,P1,image_size,cv2.CV_16SC2)

[0093] map2x,map2y=cv2.initUndistortRectifyMap(K2,dist_coeffs2,R2,P2,image_size,cv2.CV_16SC2)

[0094] # Apply the mapping matrix for stereo correction

[0095] rectified_image_left=cv2.remap(undistorted_image_left,map1x,map1y,cv2.INTER_LINEAR)

[0096] rectified_image_right=cv2.remap(undistorted_image_right,map2x,map2y,cv2.INTER_LINEAR)

[0097] #Save or display the corrected image

[0098] cv2.imwrite('rectified_left.jpg',rectified_image_left)

[0099] cv2.imwrite('rectified_right.jpg',rectified_image_right).

[0100] In image cropping, the corrected image is cropped to remove invalid areas and make the left and right images the same size for subsequent processing. Specific methods include:

[0101] #Define the cropping area

[0102] x,y,w,h=valid_roi1

[0103] cropped_image_left=rectified_image_left[y:y+h,x:x+w]

[0104] x,y,w,h=valid_roi2

[0105] cropped_image_right=rectified_image_right[y:y+h,x:x+w]

[0106] #Save or display the cropped image

[0107] cv2.imwrite('cropped_left.jpg',cropped_image_left)

[0108] cv2.imwrite('cropped_right.jpg', cropped_image_right).

[0109] The technical effect is as follows: the image distortion is corrected by using the camera's intrinsic parameter matrix and distortion coefficients, and the distorted image is converted into a geometrically accurate image by remapping the pixels in the image, thereby eliminating the geometric distortion introduced by the lens, reducing the error caused by distortion in the image processing process, ensuring that the geometric relationship in the image is consistent with the actual scene, and improving the accuracy and reliability of image processing; by generating a correction mapping matrix, the left and right eye images are remapped to align the corrected images, and the errors caused by the camera position and angle deviation are eliminated. The left and right eye images after stereo correction are aligned on the same plane, ensuring that each pixel point has an accurate correspondence in the stereo matching process, and improving the accuracy of stereo matching.

[0110] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, identifying the target object and measuring the distance of the target object based on the corrected left-eye image and the corrected right-eye image includes: acquiring the corrected left-eye image and the corrected right-eye image; preprocessing the corrected left-eye image and the corrected right-eye image, and the preprocessing includes at least one of smoothing filtering and edge detection; extracting the area corresponding to the target object in the preprocessed image based on color features; performing shape recognition on the extracted area to determine the shape of the target object; calculating the center of mass of the target object that meets the color features and shape features, and obtaining the center coordinates of the target object.

[0111] The technical effects are: by combining color and shape features, the accuracy and reliability of target object recognition are improved; based on the corrected image and accurate center of mass calculation, high accuracy of distance measurement is ensured.

[0112] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the method of extracting the area corresponding to the target object in the preprocessed image based on color features includes: using the HSV color space to define the color range of the target object, and generating a mask image for identifying the color range that conforms to the target object; converting the preprocessed image from the RGB color space to the HSV color space, and using the mask image to extract the area that conforms to the color range in the converted HSV color space.

[0113] Specifically, appropriate upper and lower limits are selected according to the color of the target object. Taking the color range of a red object as an example, the specific method includes:

[0114] cv::Scalar lowerBound1(0,100,50); / / Hue lower limit 1: 0, saturation lower limit: 100, brightness lower limit: 50

[0115] cv::Scalar upperBound1(10,255,255); / / Hue upper limit 1:10, saturation upper limit: 255, brightness upper limit: 255

[0116] cv::Scalar lowerBound2(160,100,50); / / Hue lower limit 2: 160, saturation lower limit: 100, brightness lower limit: 50

[0117] cv::Scalar upperBound2(180,255,255); / / Upper limit of hue 2: 180, upper limit of saturation: 255, upper limit of brightness: 255.

[0118] Generate a mask image in the HSV image using the defined color range. White pixels in the mask image represent areas that match the color range, and black pixels represent areas that do not match the color range. Specific methods include:

[0119] Use OpenCV's cv::inRange function to generate the mask:

[0120] cv::Mat mask1,mask2,finalMask;

[0121] cv::inRange(hsvImage,lowerBound1,upperBound1,mask1); / / The first part of the mask

[0122] cv::inRange(hsvImage,lowerBound2,upperBound2,mask2); / / The second part of the mask

[0123] cv::bitwise_or(mask1,mask2,finalMask); / / Merge masks.

[0124] Convert the corrected image from RGB color space to HSV color space. The hue, saturation, and brightness in HSV color space are more suitable for color feature segmentation and recognition. Specific methods include:

[0125] Use OpenCV's cv::cvtColor function:

[0126] cv::Mat hsvImage;

[0127] cv::cvtColor(correctedImage,hsvImage,cv::COLOR_BGR2HSV).

[0128] Use the generated mask image to extract the area in the corrected image that matches the color range. The specific methods include:

[0129] Using OpenCV's cv::bitwise_and function:

[0130] cv::Mat result;

[0131] cv::bitwise_and(correctedImage,correctedImage,result,finalMask).

[0132] The technical effect is that the use of HSV color space to define the color range can better cope with the influence of lighting changes and shadows. The hue information of HSV color space is insensitive to brightness changes, and can more stably describe color features and improve the accuracy of color recognition. By generating a mask image to identify the area that meets the color range of the target object, the interference information in the background and environment is effectively removed to ensure that only the area related to the target object is extracted, reducing the possibility of misidentification and improving the reliability of the system.

[0133] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the shape recognition of the extracted area and the judgment of the shape of the target object include: using the contour detection function of OpenCV to perform shape recognition in the extracted area to find the contour of the target object; fitting the contour by a polygon fitting method to obtain the shape characteristics of the target object and judge the shape of the target object.

[0134] Specific methods include:

[0135] Convert the mask image to a grayscale image. If the mask image is already a binary image, it can be used directly.

[0136]

[0137]

[0138] Use edge detection methods such as Canny edge detection to assist contour detection.

[0139] cv::Mat edges;

[0140] cv::Canny(grayImage,edges,100,200);

[0141] Use OpenCV's findContours function to find contours. This function returns a point set of all contours in the image.

[0142] std::vector<std::vector <cv::point>>contours;

[0143] std::vector <cv::vec4i>hierarchy;

[0144] cv::findContours(edges,contours,hierarchy,cv::RETR_TREE,cv::CHAIN_APPROX_SIMPLE);

[0145] Filter out some unnecessary contours, such as those with too small an area.

[0146]

[0147] Use OpenCV's approxPolyDP function to perform polygon fitting on each filtered contour, and determine the shape of the object by the number of vertices of the fitted polygon.

[0148]

[0149]

[0150] The shape is determined by the number of polygon vertices: three vertices are a triangle, four vertices are a rectangle or square, and five or more vertices are a polygon or a circle (by further determining the circle).

[0151]

[0152] Its technical effect is: using OpenCV's contour detection function, the boundary of the target object can be accurately found, ensuring that shape recognition is based on clear contour information. Combined with the polygon fitting method, background noise and interference information can be effectively filtered out, the accuracy of target object detection can be enhanced, the geometric features of the target object can be accurately described, and the accuracy of shape recognition can be improved.

[0153] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the calculation of the center of mass of the target object that meets the color characteristics and shape characteristics to obtain the center coordinates of the target object includes: obtaining the coordinates of all points of the contour of the target object; using the moments function of OpenCV to calculate the geometric moments of the contour of the target object; using the geometric moments to calculate the coordinates of the center of mass to obtain the center coordinates of the target object.

[0154] Specific methods include:

[0155] Use the findContours function to find all contours in the image and get a list containing all contours

[0156] std::vector <std::vector <cv::point>>contours;

[0157] cv::findContours(masked_image,contours,cv::RETR_EXTERNAL,cv::CHAIN_APPROX_SIMPLE);

[0158] Geometric moments include zero-order moments, first-order moments, and other high-order moments, which can be used to calculate the center of mass. For each contour, use OpenCV's moments function to calculate the geometric moments of the contour.

[0159] cv::Moments moments=cv::moments(contour);

[0160] Use geometric moments to calculate the coordinates of the center of mass. The calculation formula is: Among them, m 10 and m 01 are the first-order moment, m 00 is the zero-order moment.

[0161] All calculated center of mass coordinates are stored as the final center coordinates of the target object.

[0162] The technical effect is: by calling the moments function of OpenCV, the geometric moment of the target object's contour is calculated. The geometric moment is a mathematical tool that describes the shape and centroid distribution of the image area, including the mass distribution of the image area, the centroid position and other information. According to the calculated geometric moment, the centroid coordinates of the target object are calculated using the corresponding formula, the center point of the target object is determined, and the precise center coordinates are provided.

[0163] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the control of the humanoid robot to execute the planned gait and various predefined actions includes: starting the ROS node, initializing the control system of the humanoid robot, loading the robot model and action library, and initializing sensor data subscription; according to the current task, planning the actions and gaits that the robot needs to execute, using the gait generation algorithm, selecting the corresponding action sequence from the predefined action library, sending control instructions to each joint of the robot, and adjusting the joint angle and movement speed; using a sensor fusion algorithm, such as a Kalman filter, to fuse the motion state data and estimate the posture of the humanoid robot; according to the result of the posture estimation, adjusting and re-controlling the humanoid robot to execute the planned actions and gaits in real time; after the action is executed, adjusting the robot joints, resetting the posture, and stopping the real-time collection of sensor data.

[0164] Its technical effects are: applying sensor fusion algorithms to fuse motion state data, effectively estimating the robot's posture, more accurately reflecting the robot's true state, and reducing the inaccuracy caused by sensor noise and errors; real-time adjustment and re-control of the robot to execute planned movements and gaits, so that the robot can cope with dynamically changing environments and task requirements, ensuring that the robot can maintain an efficient working state in a changing environment.

[0165] In a preferred embodiment of the present invention, in the above-mentioned ROS-based humanoid robot control method, the output of preset voice information to the outside world through a speaker device includes: subscribing to or monitoring messages or instructions that trigger voice output; receiving the message or instruction that triggers voice output, parsing the content, and determining the specific voice information that needs to be output; according to the voice information, selecting a corresponding voice text from a preset voice library, and using a voice synthesis function package in ROS, such as the sound_play package, to convert the voice text into a voice file; using the voice playback function package in ROS to play the voice file through a speaker device.

[0166] Its technical effects are: the mechanism of subscribing to or listening to trigger messages ensures that the system can respond to external events or instructions in real time and realize automated voice output, so that the robot can respond quickly to various instructions; the speech synthesis function package in ROS is used for text-to-speech conversion, which ensures that the generated voice files are of high quality and clear pronunciation, making the voice output more understandable and natural; the preset voice information is output through the speaker device, which improves the quality of interaction between the robot and the user.

[0167] The second embodiment of the present invention provides a humanoid robot control system based on ROS, which includes: a motion state data acquisition module, which is used to collect the motion state data of the humanoid robot in real time, and the motion state data includes at least one of angular velocity data and acceleration data; a visual module, which is used to obtain the original image data around the humanoid robot and perform segmentation processing to obtain independent left-eye images and right-eye images; a correction module, which is used to perform stereo correction on the independent left-eye image and the right-eye image so that the two are aligned on the same plane to obtain a corrected left-eye image and a corrected right-eye image; a target recognition module, which is used to identify the target object and measure the distance of the target object according to the corrected left-eye image and the corrected right-eye image; a motion control module, which is used to control the humanoid robot to execute a planned gait and various predefined actions; a voice module, which is used to output preset voice information to the outside world through a speaker device.

[0168] The computer program product of the ROS-based humanoid robot control method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0169] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned ROS-based humanoid robot control method can be executed, thereby achieving accurate target object recognition and ranging, improving the robot's behavior planning and execution capabilities in complex environments, achieving highly intelligent behavior control, and enabling the robot to autonomously complete complex tasks.

[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0171] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.< / cv::point> < / cv::point>

Claims

1. A humanoid robot control method based on ROS, characterized in that: include: Collecting motion state data of the humanoid robot in real time, wherein the motion state data includes at least one of angular velocity data and acceleration data; The original image data around the humanoid robot is acquired and segmented to obtain independent left-eye images and right-eye images; Performing stereoscopic correction on the independent left-eye image and the right-eye image so that the two are aligned on the same plane to obtain a corrected left-eye image and a corrected right-eye image; According to the corrected left-eye image and the corrected right-eye image, identifying a target object and measuring a distance of the target object; Control the humanoid robot to perform planned gaits and various predefined actions; Output preset voice information to the outside world through a speaker device; The step of identifying a target object and measuring a distance of the target object according to the corrected left-eye image and the corrected right-eye image includes: Acquire the corrected left-eye image and the corrected right-eye image; Preprocessing the corrected left-eye image and the corrected right-eye image, wherein the preprocessing includes at least one of smoothing filtering and edge detection; Extract the area corresponding to the target object in the preprocessed image based on the color feature; Performing shape recognition on the extracted area to determine the shape of the target object; Calculating the center of mass of the target object that meets the color feature and the shape feature to obtain the center coordinates of the target object; The region corresponding to the target object extracted in the preprocessed image based on the color feature includes: Using the HSV color space to define the color range of the target object, generating a mask image for identifying the color range that matches the target object; Convert the preprocessed image from the RGB color space to the HSV color space, and use the mask image to extract the area that meets the color range in the converted HSV color space; The performing shape recognition on the extracted area to determine the shape of the target object comprises: Use OpenCV's contour detection function to perform shape recognition in the extracted area and find the contour of the target object; Fitting the contour by a polygon fitting method to obtain the shape characteristics of the target object and determine the shape of the target object; The calculating the centroid of the target object that meets the color feature and the shape feature to obtain the center coordinates of the target object includes: Obtaining coordinates of all points of the contour of the target object; Use OpenCV's moments function to calculate the geometric moments of the target object's contour; The coordinates of the center of mass are calculated using the geometric moment to obtain the center coordinates of the target object.

2. The humanoid robot control method based on ROS according to claim 1, characterized in that: The obtaining of the original image data around the humanoid robot and performing segmentation processing to obtain independent left-eye images and right-eye images includes: Use a stereo camera to obtain left and right eye images around the humanoid robot, perform image synthesis, and obtain original image data; Start the ROS image format conversion function package to convert the original image data into OpenCV image data; Write and start a camera segmentation ROS node program for image segmentation, segment the OpenCV image data, and obtain independent OpenCV left-eye image and OpenCV right-eye image; The ROS image format conversion function package is started again to convert the OpenCV left-eye image and the OpenCV right-eye image to obtain a mutually independent ROS left-eye image and a ROS right-eye image.

3. The humanoid robot control method based on ROS according to claim 1, characterized in that: The performing stereoscopic correction on the independent left-eye image and the right-eye image so that the two are aligned on the same plane to obtain the corrected left-eye image and the corrected right-eye image comprises: Start the ROS image format conversion function package to convert the left eye image and the right eye image into an OpenCV left eye image and an OpenCV right eye image; Calling the image distortion correction function in OpenCV, using the intrinsic parameter matrix and distortion coefficient of the camera, to perform distortion correction on the OpenCV left-eye image and the OpenCV right-eye image, eliminate the distortion effect, and generate an image after distortion elimination; Calling the binocular stereo correction function in OpenCV, using the correction mapping matrix, performing stereo correction on the dedistorted image to generate a corrected image; Cropping the corrected image to remove invalid areas so that the left and right images are of the same size, thereby obtaining a cropped image; The ROS image format conversion function package is started again to convert the cropped image data to obtain a corrected ROS left-eye image and a ROS right-eye image.

4. The humanoid robot control method based on ROS according to claim 1, characterized in that: The controlling of the humanoid robot to execute the planned gait and various predefined actions comprises: Start the ROS node, initialize the control system of the humanoid robot, load the robot model and action library, and initialize sensor data subscription; According to the current task, plan the actions and gaits that the robot needs to perform, use the gait generation algorithm, and select the corresponding action sequence from the predefined action library; fusing the motion state data using a sensor fusion algorithm to estimate the posture of the humanoid robot; According to the result of posture estimation, the humanoid robot is adjusted and re-controlled in real time to execute the planned movements and gaits; After the action is completed, adjust the robot joints, reset the posture, and stop the real-time collection of sensor data.

5. The humanoid robot control method based on ROS according to claim 1, characterized in that: Outputting the preset voice information to the outside world through the speaker device includes: Subscribe to or listen for messages or commands that trigger voice output; Receiving the message or instruction that triggers voice output, parsing the content, and determining the specific voice information that needs to be output; According to the voice information, a corresponding voice text is selected from a preset voice library, and the voice text is converted into a voice file using a speech synthesis function package in ROS; Use the voice playback function package in ROS to play the voice file through the speaker device.

6. A humanoid robot control system based on ROS, characterized in that: include: A motion state data acquisition module, used for acquiring the motion state data of the humanoid robot in real time, wherein the motion state data includes at least one of angular velocity data and acceleration data; The visual module is used to obtain the original image data around the humanoid robot and perform segmentation processing to obtain independent left-eye images and right-eye images; A correction module, used for performing stereo correction on the independent left-eye image and the right-eye image so that the two are aligned on the same plane to obtain a corrected left-eye image and a corrected right-eye image; A target recognition module, used to recognize a target object and measure a distance of the target object according to the corrected left-eye image and the corrected right-eye image; The motion control module is used to control the humanoid robot to perform planned gaits and various predefined actions; A voice module is used to output preset voice information to the outside world through a speaker device; The operations performed by the target recognition module include: acquiring the corrected left-eye image and the corrected right-eye image; preprocessing the corrected left-eye image and the corrected right-eye image, wherein the preprocessing includes at least one of smoothing filtering and edge detection; extracting the area corresponding to the target object in the preprocessed image based on color features; performing shape recognition on the extracted area to determine the shape of the target object; calculating the center of mass of the target object that meets the color features and shape features to obtain the center coordinates of the target object; The method of extracting the area corresponding to the target object in the preprocessed image based on the color feature comprises: defining the color range of the target object using the HSV color space, generating a mask image for identifying the color range that matches the target object; converting the preprocessed image from the RGB color space to the HSV color space, and extracting the area that matches the color range in the converted HSV color space using the mask image; The performing shape recognition on the extracted area to determine the shape of the target object comprises: using the contour detection function of OpenCV to perform shape recognition in the extracted area to find the contour of the target object; fitting the contour by a polygon fitting method to obtain the shape characteristics of the target object and determine the shape of the target object; The calculation of the center of mass of the target object that meets the color characteristics and shape characteristics to obtain the center coordinates of the target object includes: obtaining the coordinates of all points of the outline of the target object; using the moments function of OpenCV to calculate the geometric moments of the outline of the target object; using the geometric moments to calculate the coordinates of the center of mass to obtain the center coordinates of the target object.

Images