Method, system and terminal for aligning multiple sensing timestamps of humanoid robot

Abstract

The invention relates to a humanoid robot multi-sensing timestamp alignment method, system and terminal, and relates to the technical field of robots, and the method comprises the steps: obtaining image records of images captured by a plurality of preset cameras; determining a main record and a comparison record according to the image record; determining a main timestamp based on the main record, and determining a comparison timestamp according to the main timestamp and the comparison record; determining an accumulated time error according to the main timestamp and the comparison timestamp; determining an optimal timestamp according to the accumulated time error; and synchronizing the comparison timestamps of the plurality of preset cameras according to the optimal timestamp. The method and the device have the effects of improving the data processing efficiency and speed and timely aligning data from different sensors.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and more particularly to a method, system and terminal for aligning multi-sensor timestamps of a humanoid robot. Background Art

[0002] A humanoid robot is a robot with a body structure similar to that of a human and capable of performing tasks such as walking, grasping, and manipulating objects.

[0003] In the prior art, a humanoid robot is generally equipped with multiple cameras and various sensors, and the state of the humanoid robot is judged by analyzing the images and data obtained by the cameras or sensors, so as to improve the accuracy of controlling the humanoid robot.

[0004] When the images and data obtained by the cameras or sensors are at different times, it is easy to cause errors when analyzing the images and data at a certain moment, thereby reducing the accuracy of controlling the humanoid robot. Summary of the Invention

[0005] In order to improve the efficiency and speed of data processing and be able to align data from different sensors in a timely manner, the present invention provides a method, system and terminal for aligning multi-sensor timestamps of a humanoid robot.

[0006] In a first aspect, the present invention provides a method for aligning multi-sensor timestamps of a humanoid robot, adopting the following technical solution:

[0007] A method for aligning multi-sensor timestamps of a humanoid robot includes:

[0008] Obtaining image records of images captured by multiple preset cameras;

[0009] Determining a main record and a comparison record according to the image records;

[0010] Determining a main timestamp based on the main record, and determining a comparison timestamp according to the main timestamp and the comparison record;

[0011] Determining a cumulative time error according to the main timestamp and the comparison timestamp;

[0012] Determining an optimal timestamp according to the cumulative time error;

[0013] Synchronizing the comparison timestamps of multiple preset cameras according to the optimal timestamp.

[0014] By adopting the above technical solution, the cumulative error of the time points of images captured by different cameras and the other cameras is calculated, so as to select the camera with the smallest cumulative error as the standard to synchronize the time points of images captured by the other cameras, thereby aligning the time of images captured by the cameras and improving the efficiency and speed of data processing.

[0015] Optionally, it further includes:

[0016] Obtain the data record of the data acquired by a preset sensor;

[0017] Determine the left state timestamp and the right state timestamp according to the data record and the optimal timestamp;

[0018] Determine the interpolation ratio according to the left state timestamp, the right state timestamp and the optimal timestamp;

[0019] Determine the left state data from the data record according to the left state timestamp, and determine the right state data from the data record according to the right state timestamp;

[0020] Determine the interpolated data according to the left state data, the right state data and the interpolation ratio;

[0021] Package and publish the interpolated data based on the optimal timestamp.

[0022] By adopting the above technical solution, timestamp alignment can ensure the temporal consistency of data from different sensors, thus significantly improving the accuracy of data synchronization, helping to reduce errors caused by time asynchrony, and enabling direct fusion and comparison of data at the same time point without complex time conversion or interpolation calculations, thereby improving the efficiency and speed of data processing, and further enabling the robot to respond within the shortest time, improving the overall performance and response speed.

[0023] Optionally, it further includes a data anomaly handling method, and the data anomaly handling method includes:

[0024] Judge whether the data is continuous according to the data record;

[0025] When the data is not continuous, determine the abnormal data points according to the data record;

[0026] Determine the abnormal timestamp based on the abnormal data points;

[0027] Determine the image timestamp and the previous timestamp from the image record according to the abnormal timestamp;

[0028] Retrieve the abnormal image based on the image timestamp, and retrieve the previous image based on the previous timestamp;

[0029] Determine the abnormal features based on the abnormal data points;

[0030] Judge whether there are abnormal features according to the abnormal image and the previous image;

[0031] When there are abnormal features in the abnormal image or the previous image, delete the abnormal feature points in the data record and update the data record.

[0032] By adopting the above technical solution, when the robotic arm or gripper of the robot is blocked by external objects, it is likely to cause the robotic arm or gripper to be unable to move at a predetermined speed, resulting in abnormal data points with data errors in the data obtained by the sensor. It is determined whether there is an obstruction based on the images before and after the abnormal data points appear, and when there is an obstruction, it is determined that the sensor is operating normally, and the abnormal data points are deleted to reduce the situation where the control system is misled by incorrect data.

[0033] Optionally, the data anomaly processing method further includes:

[0034] When there are no abnormal features in the abnormal image or the previous image, determine the abnormal position based on the abnormal data points;

[0035] Determine the maintenance path according to the abnormal position;

[0036] Control a preset unmanned aerial vehicle to reach the abnormal position according to the maintenance path and obtain maintenance images;

[0037] Determine whether there is a foreign object at the abnormal position according to the maintenance images;

[0038] When there is a foreign object at the abnormal position, determine the grasping position according to the maintenance images;

[0039] Determine the gripper number according to the grasping position;

[0040] Determine the gripper position according to the gripper number;

[0041] Determine the grasping path according to the gripper position and the grasping position;

[0042] Control a preset mechanical gripper to grasp the foreign object according to the grasping path.

[0043] By adopting the above technical solution, when the robotic arm or gripper of the robot is entangled by a foreign object, it is likely to cause the robotic arm or gripper to be difficult to move. The unmanned aerial vehicle is used to check whether there is a foreign object at the abnormal position, and when there is a foreign object entanglement at the abnormal position, the mechanical gripper on the robot is used to grasp the foreign object to remove the foreign object.

[0044] Optionally, the data anomaly processing method further includes:

[0045] After the mechanical gripper grasps the foreign object, obtain the post-grasping image of the abnormal position;

[0046] Determine whether there is a foreign object at the abnormal position according to the post-grasping image;

[0047] When there is a foreign object at the abnormal position, determine the foreign object material according to the post-grasping image;

[0048] Determine the tearing force according to the foreign object material;

[0049] Determine the driving threshold according to the abnormal position;

[0050] When the tearing force is lower than the driving threshold, determine the driving number according to the abnormal position;

[0051] Control the preset driving device according to the driving number to tear the foreign object according to the tearing force.

[0052] By adopting the above technical solution, when the foreign object is tightly wound at the abnormal position, it is difficult to directly grab and remove the foreign object by the mechanical gripper. At this time, the strength of the foreign object is judged by the appearance of the foreign object, and when the strength of the foreign object is high, the driving device at the abnormal position provides power to tear the foreign object.

[0053] Optionally, the data anomaly handling method further includes:

[0054] When the driving device tears the foreign object according to the tearing force, obtain the tearing image of the abnormal position;

[0055] Determine the crack area according to the tearing image;

[0056] Determine the tearing coefficient according to the crack area;

[0057] Determine the buffering force according to the tearing coefficient and the tearing force;

[0058] Control the preset driving device according to the driving number to tear the foreign object according to the buffering force.

[0059] By adopting the above technical solution, when the driving device at the abnormal position provides power to tear the foreign object, the force provided by the driving device is adjusted in real time according to the area of the crack on the foreign object, so as to reduce the force provided by the driving device as the crack increases, and further reduce the situation that the huge force provided by the driving device tears the foreign object and then drives the gripper or the robotic arm to move quickly, resulting in equipment damage or personal injury.

[0060] Optionally, it further includes a freezing handling method, and the freezing handling method includes:

[0061] When there is no foreign object at the abnormal position, obtain the environmental temperature where the robot is located;

[0062] When the environmental temperature is lower than the preset freezing threshold, obtain the abnormal humidity at the abnormal position;

[0063] When the abnormal humidity is higher than the preset condensation threshold, judge whether there is a freezing situation at the abnormal position according to the inspection image;

[0064] When there is a freezing situation at the abnormal position, control the preset alarm device to issue a freezing alarm according to the abnormal position.

[0065] By adopting the above technical solution, when the temperature is relatively low, it is easy to cause icing at the joints of the robot's robotic arm or gripper, resulting in sluggish movement of the robotic arm or gripper. Determine whether there is icing by observing the appearance of the robot, and issue an alarm in a timely manner when icing occurs to facilitate timely handling by the staff.

[0066] Optionally, the freezing treatment method further includes:

[0067] When freezing occurs at the abnormal position, determine the activity range of the driving device according to the driving number;

[0068] Determine the heating path according to the activity range and the abnormal position;

[0069] Determine the heating temperature according to the ambient temperature and abnormal humidity;

[0070] Control the preset drone to heat the abnormal position according to the heating temperature along the heating path.

[0071] By adopting the above technical solution, when icing occurs, the drone carrying the heating device performs circumferential heating around the joints of the robot's robotic arm or gripper, so as to thaw in time and reduce the sluggish movement of the robotic arm or gripper.

[0072] In a second aspect, the present application provides a system for aligning multi-sensor timestamps of a humanoid robot, adopting the following technical solution:

[0073] A system for aligning multi-sensor timestamps of a humanoid robot includes:

[0074] An acquisition module for acquiring image records, data records, maintenance images, post-grasp images, torn images, ambient temperature, and abnormal humidity;

[0075] A memory for storing the program of any of the above methods for aligning multi-sensor timestamps of a humanoid robot;

[0076] A processor, and the program in the memory can be loaded and executed by the processor.

[0077] In a third aspect, the present application provides an intelligent terminal, adopting the following technical solution:

[0078] An intelligent terminal includes a memory and a processor, and a computer program capable of being loaded and executed by the processor for any of the above methods for aligning multi-sensor timestamps of a humanoid robot is stored on the memory.

[0079] By adopting the above technical solution, the cumulative error of the time points when different cameras capture images with the remaining cameras is calculated, and then the camera with the smallest cumulative error is selected as the standard to synchronize the time points of the images captured by the remaining cameras, so as to align the time of the images captured by the cameras, improving the efficiency and speed of data processing.

[0080] In summary, the present application includes at least one of the following beneficial technical effects:

[0081] 1. Calculate the cumulative error of the time points when different cameras capture images with the remaining cameras, and then select the camera with the smallest cumulative error as the standard to synchronize the time points of the images captured by the remaining cameras, so as to align the time of the images captured by the cameras, improving the efficiency and speed of data processing;

[0082] 2. Timestamp alignment can ensure the temporal consistency of data from different sensors, thus significantly improving the accuracy of data synchronization, helping to reduce errors caused by time asynchrony, and enabling direct fusion and comparison of data at the same time point without complex time conversion or interpolation calculations, thereby improving the efficiency and speed of data processing, and further enabling the robot to respond in the shortest time, improving the overall performance and response speed;

[0083] 3. When the robotic arm or gripper of the robot is blocked by an external object, it is likely to cause the robotic arm or gripper to fail to move at the predetermined speed, resulting in abnormal data points with data errors in the data obtained by the sensor. Determine whether there is an obstruction based on the images before and after the abnormal data points appear, and when there is an obstruction, determine that the sensor is operating normally and delete the abnormal data points to reduce the situation where incorrect data misleads the control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is the flow of a method for multi-sensor timestamp alignment of a humanoid robot Figure 1 ;

[0085] Figure 2 is the flow of a method for multi-sensor timestamp alignment of a humanoid robot Figure 2 ;

[0086] Figure 3 is the flow of a data anomaly handling method Figure 1 ;

[0087] Figure 4 is the flow of a data anomaly handling method Figure 2 ;

[0088] Figure 5 is the flow of a data anomaly handling method Figure 3 ;

[0089] Figure 6 is the process of the data anomaly handling method Figure 4 ;

[0090] Figure 7 is the process of the freezing handling method Figure 1 ;

[0091] Figure 8 is the process of the freezing handling method Figure 2 . Detailed implementation manners

[0092] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0093] The embodiments of the present application disclose a method for aligning multi-sensor timestamps of a humanoid robot. Timestamp alignment can ensure the temporal consistency of data from different sensors, thereby significantly improving the accuracy of data synchronization, helping to reduce errors caused by time asynchrony, and enabling direct fusion and comparison of data at the same time point without complex time conversion or interpolation calculations, thereby improving the efficiency and speed of data processing, and further enabling the robot to respond in the shortest time, improving the overall performance and response speed.

[0094] Referring to Figure 1 , a method for aligning multi-sensor timestamps of a humanoid robot includes:

[0095] Step 100: Obtain image records captured by a plurality of preset cameras.

[0096] The camera refers to a device on the humanoid robot for capturing images. The image record refers to a data set of images captured by different cameras and their capture times. The image record can be extracted from the camera, and the method for obtaining the image record is selected by the staff according to the actual situation and will not be elaborated here.

[0097] Step 101: Determine the main record and the comparison record according to the image record.

[0098] The main record refers to the image record of the camera to be evaluated, where the camera to be evaluated refers to the camera sequentially selected from all cameras. The comparison record refers to the image records of the other cameras except the camera corresponding to the main record. The method for determining the comparison record of the main record is selected by the staff according to the actual situation and will not be elaborated here.

[0099] Step 102: Determine the main timestamp based on the main record, and determine the comparison timestamp according to the main timestamp and the comparison record.

[0100] The main timestamp refers to the time point of the image of the last frame captured in the main record, and the comparison timestamp refers to the first time point in the comparison record that is not higher than the main timestamp. The method for determining the main timestamp and the comparison timestamp is selected by the staff according to the actual situation and will not be elaborated here.

[0101] Step 103: Determine the cumulative time error based on the main timestamp and the comparison timestamp.

[0102] The cumulative time error refers to the sum of the differences between the main timestamp and each comparison timestamp. The deviation degree between the main timestamp and the other cameras is judged through the cumulative time error. The calculation method of the cumulative time error is selected by the staff according to the actual situation and will not be elaborated here.

[0103] Step 104: Determine the optimal timestamp based on the cumulative time error.

[0104] The optimal timestamp refers to the main timestamp with the smallest cumulative time error. The method for determining the optimal timestamp is selected by the staff according to the actual situation and will not be elaborated here.

[0105] Step 105: Synchronize the comparison timestamps of multiple preset cameras according to the optimal timestamp.

[0106] Calculate the cumulative error of the time points of the images captured by different cameras and the other cameras, and thus select the camera with the smallest cumulative error as the standard to synchronize the time points of the images captured by the other cameras, so as to align the time points of the images captured by the cameras and improve the efficiency and speed of data processing.

[0107] Refer to Figure 2 , a method for aligning multi-sensor timestamps of a humanoid robot, further includes:

[0108] Step 106: Obtain the data record of the data acquired by the preset sensor.

[0109] The sensor refers to the sensor device on the robot used to acquire data such as the joint angles of the robotic arm, the position of the gripper, and the position of the chassis. The data record refers to the data acquired by the sensor and its corresponding time point. The data record can be retrieved from the corresponding sensor. The method for obtaining the data record is selected by the staff according to the actual situation and will not be elaborated here.

[0110] Step 107: Determine the left state timestamp and the right state timestamp based on the data record and the optimal timestamp.

[0111] The left state timestamp refers to the first time point in the data record that is not higher than the optimal timestamp, and the right state timestamp refers to the first time point in the data record that is higher than the optimal timestamp. The method for determining the left state timestamp and the right state timestamp is selected by the staff according to the actual situation and will not be elaborated here.

[0112] Step 108: Determine the interpolation ratio based on the left status timestamp, the right status timestamp, and the optimal timestamp.

[0113] The interpolation ratio refers to the proportional factor used to determine the relative weight of the optimal timestamp between the left status timestamp and the right status timestamp. The calculation method of the interpolation ratio is common knowledge to those skilled in the art and will not be elaborated here.

[0114] Step 109: Determine the left status data from the data record according to the left status timestamp, and determine the right status data from the data record according to the right status timestamp.

[0115] The left status data refers to the data value obtained at the left status timestamp in the data record, and the right status data refers to the data value obtained at the right status timestamp in the data record. The determination methods of the left status data and the right status data are common knowledge to those skilled in the art and will not be elaborated here.

[0116] Step 110: Determine the interpolated data according to the left status data, the right status data, and the interpolation ratio.

[0117] The interpolated data is the data value corresponding to the optimal timestamp calculated according to the interpolation algorithm. The calculation method of the interpolated data is common knowledge to those skilled in the art and will not be elaborated here.

[0118] Step 111: Encapsulate and publish the interpolated data based on the optimal timestamp.

[0119] Package the synchronization status of each sensor of the robot together and publish the synchronization status of the humanoid robot at a moment externally, so as to ensure the temporal consistency of data from different sensors, thereby significantly improving the accuracy of data synchronization, helping to reduce errors caused by time asynchrony, and enabling direct fusion and comparison of data at the same time point without complex time conversion or interpolation calculations, thus improving the efficiency and speed of data processing, and further enabling the robot to respond in the shortest time, improving the overall performance and response speed.

[0120] Refer to Figure 3 , the data exception handling method includes:

[0121] Step 200: Determine whether the data is continuous according to the data record.

[0122] Continuous means that there is no obvious gap between adjacent values. The determination method of continuity is common knowledge to those skilled in the art and will not be elaborated here.

[0123] Step 201: When the data is not continuous, determine the abnormal data points according to the data record.

[0124] When the data is discontinuous, it means that there is at least one obvious gap between adjacent numerical values in the data record. The abnormal data points are the data points with gaps, and the method for determining abnormal data points is common knowledge to those skilled in the art and will not be elaborated here.

[0125] Step 202: Determine the abnormal timestamp based on the abnormal data points.

[0126] The abnormal timestamp is the time point when the abnormal data point is obtained, and the abnormal timestamp can be retrieved from the data record. The method for retrieving the abnormal timestamp is common knowledge to those skilled in the art and will not be elaborated here.

[0127] Step 203: Determine the image timestamp and the previous timestamp from the image record according to the abnormal timestamp.

[0128] The image timestamp refers to the first time point not higher than the abnormal timestamp, and the previous timestamp refers to the second time point not higher than the abnormal timestamp. The method for determining the image timestamp and the previous timestamp is common knowledge to those skilled in the art and will not be elaborated here.

[0129] Step 204: Retrieve the abnormal image based on the image timestamp and retrieve the previous image based on the previous timestamp.

[0130] The abnormal image is the image captured by the camera at the image timestamp, and the previous image is the image captured by the camera at the previous timestamp. The abnormal image and the previous image can be retrieved from the image record. The method for retrieving the abnormal image and the previous image is common knowledge to those skilled in the art and will not be elaborated here.

[0131] Step 205: Determine the abnormal features based on the abnormal data points.

[0132] The abnormal features refer to the features that are likely to cause changes in the data of the sensor. For example, when the abnormal data points are within the data record of detecting the joint angle of the robotic arm, the abnormal features generally include obstacles such as stones and boxes. The abnormal features can be queried from the feature data table, which refers to the data table recording the feature data of the abnormal features corresponding to different data records. The method for determining the abnormal features is selected by the staff according to the actual situation and will not be elaborated here.

[0133] Step 206: Judge whether there are abnormal features according to the abnormal image and the previous image.

[0134] It can be judged whether there are foreign object features in the abnormal image or the previous image through image recognition technology. The method for judging the abnormal features is common knowledge to those skilled in the art and will not be elaborated here.

[0135] Step 207: When there are abnormal features in the abnormal image or the previous image, delete the abnormal feature points in the data record and update the data record.

[0136] The presence of abnormal features in the abnormal image or the previous image indicates that the sensor is operating normally. At this time, the abnormal data points are deleted to reduce the situation of misleading the control system with incorrect data and improve the stability of the humanoid robot's operation.

[0137] Refer to Figure 4 , the data anomaly processing method further includes:

[0138] Step 208: When there are no abnormal features in the abnormal image or the previous image, determine the abnormal position based on the abnormal data points.

[0139] The absence of abnormal features in the abnormal image or the previous image indicates that there is a problem with the operation of the humanoid robot itself and further inspection is required. The abnormal position refers to the position information corresponding to the data record where the abnormal data point is located. For example, when the abnormal data point is within the data record for detecting the joint angle of the robotic arm, the abnormal position is the position of the robotic arm joint, and the abnormal position can be obtained by querying from the position data table. The position data table refers to a data table that records different data records and their corresponding abnormal positions.

[0140] Step 209: Determine the maintenance path according to the abnormal position.

[0141] The drone is a device used to detect the appearance of the humanoid robot. The drone is also equipped with a heating device for heating and a camera device for capturing images. The drone is selected by the staff according to the actual situation and will not be elaborated here. The maintenance path refers to the route for the drone to reach the abnormal position. The maintenance path can be automatically generated by the control program supporting the drone according to the abnormal position. The method for determining the maintenance path is common knowledge for those skilled in the art and will not be elaborated here.

[0142] Step 210: Control the preset drone to reach the abnormal position according to the maintenance path and obtain the maintenance image.

[0143] The maintenance image is the picture of the humanoid robot captured after the drone reaches the abnormal position. The method for obtaining the maintenance image is selected by the staff according to the actual situation and will not be elaborated here.

[0144] Step 211: Judge whether there is a foreign object at the abnormal position according to the maintenance image.

[0145] Foreign objects refer to objects such as cloth strips and plastics that are easily wound around the humanoid robot. It can be judged whether there is a foreign object at the abnormal position through image recognition technology. The method for identifying foreign objects is common knowledge for those skilled in the art and will not be elaborated here.

[0146] Step 212: When there is a foreign object at the abnormal position, determine the grasping position according to the maintenance image.

[0147] The presence of foreign objects at the abnormal position indicates that the entanglement of foreign objects causes the manipulator or gripper to be unable to extend and retract normally, resulting in the sensor detecting abnormal data. The grasping position refers to the position used to grasp the foreign object. Generally, the center of the foreign object is adopted, and the grasping position can be identified through image recognition technology. The method for determining the grasping position is common knowledge for those skilled in the art and will not be elaborated here.

[0148] Step 213: Determine the gripper number according to the grasping position.

[0149] A mechanical gripper refers to a device used to grasp objects on a humanoid robot. Mechanical grippers are generally symmetrically arranged on both sides of the robot. The gripper number refers to the number used to distinguish mechanical grippers. Each gripper corresponds to a gripper number one by one. The movement range of the mechanical gripper is limited, and it is generally difficult to handle objects too close to its own position. Select the gripper number of the mechanical gripper far from the grasping position. The method for selecting the gripper number is common knowledge for those skilled in the art and will not be elaborated here.

[0150] Step 214: Determine the gripper position according to the gripper number.

[0151] The gripper position is the position of the mechanical gripper corresponding to the gripper number. The method for determining the gripper position is common knowledge for those skilled in the art and will not be elaborated here.

[0152] Step 215: Determine the grasping path according to the gripper position and the grasping position.

[0153] The grasping path refers to the route for the mechanical gripper to reach the grasping position from the gripper position. The grasping path is automatically generated by the supporting control program of the mechanical gripper. The method for determining the grasping path is common knowledge for those skilled in the art and will not be elaborated here.

[0154] Step 216: Control the preset mechanical gripper to grasp the foreign object according to the grasping path.

[0155] When the manipulator or gripper of the robot is entangled by foreign objects, it is easy to cause the manipulator or gripper to be difficult to move. Check whether there are foreign objects at the abnormal position through the drone, and when there is foreign object entanglement at the abnormal position, grasp the foreign object through the mechanical gripper on the robot to remove the foreign object.

[0156] Refer to Figure 5 , the data anomaly handling method further includes:

[0157] Step 217: After the mechanical gripper grasps the foreign object, obtain the post-grasping image of the abnormal position.

[0158] The post-grasping image refers to the picture of the humanoid robot after the mechanical gripper grasps the foreign object. The post-grasping image can be obtained through the camera device on the drone. The method for obtaining the post-grasping image is selected by the staff according to the actual situation and will not be elaborated here.

[0159] Step 218: Determine whether there is a foreign object at the abnormal position according to the post-grabbing image.

[0160] It is possible to determine whether there is a foreign object in the post-grabbing image through image recognition technology. The method for identifying foreign objects is common knowledge for those skilled in the art and will not be elaborated here.

[0161] Step 219: When there is a foreign object at the abnormal position, determine the foreign object material according to the post-grabbing image.

[0162] The foreign object material refers to the type of substance that constitutes the foreign object, such as plastic, fabric, etc. The foreign object material can be determined through image recognition technology. The method for determining the foreign object material is common knowledge for those skilled in the art and will not be elaborated here.

[0163] Step 220: Determine the tearing force according to the foreign object material.

[0164] The tearing force refers to the minimum force required to tear the foreign object. The tearing force can be obtained by querying from the force relationship table, and the force relationship table is a data table that records different foreign object materials and their corresponding tearing forces.

[0165] Step 221: Determine the driving threshold according to the abnormal position.

[0166] The driving device refers to the power device on the robot used to drive the robotic arm or robotic gripper to move. The driving threshold refers to the maximum power value that the power device driving the abnormal position can provide. The driving threshold can be obtained by querying from the driving relationship table, and the driving relationship table is a data table that records different abnormal positions and their corresponding driving thresholds.

[0167] Step 222: When the tearing force is lower than the driving threshold, determine the driving number according to the abnormal position.

[0168] The tearing force being lower than the driving threshold means that the power value that the driving device can provide is higher than the force required to tear the foreign object, that is, the foreign object can be directly torn by the driving device. The driving number is the number used to distinguish the driving device, and each driving number corresponds to a driving device one by one. The driving number can be obtained by querying the number relationship table, and the number relationship table is a data table that records the driving numbers corresponding to different abnormal positions.

[0169] Step 223: Control the preset driving device to tear the foreign object according to the tearing force according to the driving number.

[0170] When the foreign object is wound tightly at the abnormal position, it is difficult to directly grab and remove the foreign object with the robotic gripper. At this time, the strength of the foreign object is judged by the appearance of the foreign object, and when the strength of the foreign object is high, the driving device at the abnormal position provides power to tear the foreign object.

[0171] Refer to Figure 6 , the data anomaly handling method further includes:

[0172] Step 224: When the driving device tears the foreign object according to the tearing force, obtain the tearing image of the abnormal position.

[0173] The tearing image refers to the picture of the humanoid robot obtained when the driving device tears the foreign object. The tearing image can be obtained through the camera device on the drone. The method for obtaining the tearing image is selected by the staff according to the actual situation and will not be elaborated here.

[0174] Step 225: Determine the crack area according to the tearing image.

[0175] The crack area refers to the total area value of the cracks on the surface of the foreign object in the tearing image. The crack area can be determined by image recognition technology. The method for determining the crack area is common knowledge in the field and will not be elaborated here.

[0176] Step 226: Determine the tearing coefficient according to the crack area.

[0177] The tearing coefficient is a value used to correct the tearing force. When the cracks on the surface of the foreign object increase, the material strength of the foreign object is likely to decrease. At this time, a lower tearing force can be used for tearing. The tearing coefficient can be obtained by querying from the coefficient relationship table. The coefficient relationship table is a data table that records different crack area ranges and their corresponding tearing coefficients.

[0178] Step 227: Determine the buffering force according to the tearing coefficient and the tearing force.

[0179] The buffering force refers to the tearing force adjusted according to the tearing coefficient. Generally, the product of the tearing coefficient and the tearing force is calculated as the buffering force. The method for determining the buffering force is common knowledge in the field and will not be elaborated here.

[0180] Step 228: Control the preset driving device to tear the foreign object according to the buffering force according to the driving number.

[0181] When power is provided by the driving device at the abnormal position to tear the foreign object, the force provided by the driving device is adjusted in real time according to the area of the cracks on the foreign object, so as to reduce the force provided by the driving device as the cracks increase, and further reduce the situation that the huge force provided by the driving device tears the foreign object and then drives the gripper or the robotic arm to move quickly, resulting in equipment damage or personal injury.

[0182] Refer to Figure 7 , the freezing treatment method includes:

[0183] Step 300: When there is no foreign object at the abnormal position, obtain the environmental temperature where the robot is located.

[0184] The ambient temperature refers to the temperature value of the space where the robot is located. The ambient temperature can be obtained through a temperature sensor. The method for obtaining the ambient temperature is selected by the staff according to the actual situation and will not be elaborated here.

[0185] Step 301: When the ambient temperature is lower than the preset freezing threshold, obtain the abnormal humidity at the abnormal location.

[0186] The freezing threshold refers to the highest temperature value at which the driving device is prone to freezing. The freezing threshold generally depends on the temperature value at which the lubricating oil used in the driving device freezes. The freezing threshold is selected by the staff according to the actual situation and will not be elaborated here. When the ambient temperature is lower than the freezing threshold, it means that the lubricating oil on the driving device is prone to condensation. The abnormal humidity refers to the relative humidity value at the abnormal location. The abnormal humidity can be obtained using a humidity sensor. The method for obtaining the abnormal humidity is selected by the staff according to the actual situation and will not be elaborated here.

[0187] Step 302: When the abnormal humidity is higher than the preset condensation threshold, determine whether there is a freezing situation at the abnormal location based on the inspection image.

[0188] The condensation threshold refers to the minimum humidity value at which water droplets are likely to precipitate from the water vapor in the air. The condensation threshold is selected by the staff according to the actual situation and will not be elaborated here. When the abnormal humidity is higher than the condensation threshold, it means that water vapor in the air is likely to precipitate. At this time, the oil is prone to freezing, causing the water vapor to be adsorbed and precipitated by the frozen oil, resulting in an expansion of the freezing range. The freezing situation at the abnormal location can be judged through image recognition technology. The method for judging freezing is common knowledge in the field and will not be elaborated here.

[0189] Step 303: When there is a freezing situation at the abnormal location, control a preset alarm device to issue a freezing alarm according to the abnormal location.

[0190] The existence of a freezing situation at the abnormal location means that the joints of the robot's robotic arm or gripper are frozen, resulting in sluggish movement of the robotic arm or gripper. The alarm device refers to a device used to alert the staff. The freezing alarm is a warning used to notify the staff of the freezing phenomenon at the abnormal location. By issuing a freezing warning through the alarm device, the staff can be notified of the freezing situation in a timely manner, so as to thaw in time and reduce the damage to the robot components caused by freezing.

[0191] Refer to Figure 8 , the freezing treatment method further includes:

[0192] Step 304: When there is a freezing situation at the abnormal location, determine the activity range of the driving device according to the drive number.

[0193] The operating range refers to the range within which a robotic arm or robotic gripper driven by a driving device can move. The operating range can be obtained by querying the range relationship table, which is a data table that records the operating ranges corresponding to driving devices with different driving numbers.

[0194] Step 305: Determine the heating path based on the operating range and the abnormal position.

[0195] The heating path refers to the route for the drone to uniformly heat the abnormal position. After excluding the operating range, the drone is controlled to uniformly heat around the abnormal position in the remaining space, thereby reducing the interference of the drone when the humanoid robot moves. The method for determining the heating position is common knowledge in the art and will not be elaborated here.

[0196] Step 306: Determine the heating temperature based on the ambient temperature and the abnormal humidity.

[0197] The heating temperature refers to the temperature value for the heating device on the drone to heat the abnormal position to melt the frozen oil and water droplets. The lower the ambient temperature and the higher the abnormal humidity, the higher the heating temperature used. The heating temperature can be obtained by querying the heating relationship table, which is a data table that records the heating temperatures corresponding to different ambient temperatures and abnormal humidities.

[0198] Step 307: Control the preset drone to heat the abnormal position according to the heating path at the heating temperature.

[0199] When there is icing, the drone equipped with a heating device is used to perform circumferential heating around the joint parts of the robotic arm or gripper of the robot, so as to thaw in time and reduce the situation of sluggish movement of the robotic arm or gripper.

[0200] Based on the same inventive concept, an embodiment of the present invention provides a system for aligning multi-sensor timestamps of a humanoid robot, including:

[0201] An acquisition module, configured to acquire image records, data records, inspection images, post-grasp images, tearing images, ambient temperature, and abnormal humidity;

[0202] A memory, configured to store the program of any one of the above methods for aligning multi-sensor timestamps of a humanoid robot;

[0203] A processor, and the program in the memory can be loaded and executed by the processor.

[0204] Based on the same inventive concept, an embodiment of the present invention provides an intelligent terminal, including a memory and a processor, and a computer program capable of being loaded and executed by the processor is stored on the memory, which is any one of the above methods for aligning multi-sensor timestamps of a humanoid robot.

[0205] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the system, device, and unit described above, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated here.

[0206] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for multi-sensor timestamp alignment of a humanoid robot, characterized in that, including: Obtaining image records of multiple preset captured images by cameras; Determining a main record and a comparison record according to the image records; Determining a main timestamp based on the main record, and determining a comparison timestamp according to the main timestamp and the comparison record; Determining an accumulated time error according to the main timestamp and the comparison timestamp; Determining an optimal timestamp according to the accumulated time error; Synchronizing the comparison timestamps of multiple preset cameras according to the optimal timestamp.

2. The method for multi-sensor timestamp alignment of a humanoid robot according to claim 1, wherein, It further includes: Obtaining data records of data acquired by a preset sensor; Determining a left status timestamp and a right status timestamp according to the data record and the optimal timestamp; Determining an interpolation ratio according to the left status timestamp, the right status timestamp and the optimal timestamp; Determining left status data from the data record according to the left status timestamp, and determining right status data from the data record according to the right status timestamp; Determining interpolation data according to the left status data, the right status data and the interpolation ratio; Encapsulating and publishing the interpolation data based on the optimal timestamp.

3. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 2, characterized in that, It further includes a data anomaly handling method, and the data anomaly handling method includes: Judging whether the data is continuous according to the data record; When the data is not continuous, determining abnormal data points according to the data record; Determining an abnormal timestamp based on the abnormal data points; Determining an image timestamp and a previous timestamp from the image records according to the abnormal timestamp; Retrieving an abnormal image based on the image timestamp, and retrieving a previous image based on the previous timestamp; Determining abnormal features based on the abnormal data points; Judging whether there are abnormal features according to the abnormal image and the previous image; When there are abnormal features in the abnormal image or the previous image, deleting the abnormal feature points in the data record and updating the data record.

4. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 3, characterized in that, The data anomaly handling method further includes: When there are no abnormal features in the abnormal image or the previous image, determining an abnormal position based on the abnormal data points; Determining a maintenance path according to the abnormal position; Controlling a preset drone to reach the abnormal position according to the maintenance path, and acquiring a maintenance image; Judging whether there is a foreign object at the abnormal position according to the maintenance image; When there is a foreign object at the abnormal position, determining a grasping position according to the maintenance image; Determining a gripper number according to the grasping position; Determining a gripper position according to the gripper number; Determining a grasping path according to the gripper position and the grasping position; Controlling a preset mechanical gripper to grasp the foreign object according to the grasping path.

5. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 4, characterized in that, The data anomaly handling method further includes: After the mechanical gripper grasps the foreign object, acquiring a post-grasping image of the abnormal position; Judging whether there is a foreign object at the abnormal position according to the post-grasping image; When there is a foreign object at the abnormal position, determining the foreign object material according to the post-grasping image; Determining a tearing force according to the foreign object material; Determining a driving threshold according to the abnormal position; When the tearing force is lower than the driving threshold, determining a driving number according to the abnormal position; Controlling a preset driving device to tear the foreign object according to the tearing force according to the driving number.

6. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 5, characterized in that, The data anomaly handling method further includes: When the driving device tears the foreign object according to the tearing force, acquiring a tearing image of the abnormal position; Determining a crack area according to the tearing image; Determining a tearing coefficient according to the crack area; Determining a buffering force according to the tearing coefficient and the tearing force; Controlling a preset driving device to tear the foreign object according to the buffering force according to the driving number.

7. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 6, characterized in that, It also includes a freezing processing method, and the freezing processing method includes: When there is no foreign object at the abnormal position, obtain the environmental temperature where the robot is located; When the environmental temperature is lower than the preset freezing threshold, obtain the abnormal humidity at the abnormal position; When the abnormal humidity is higher than the preset condensation threshold, judge whether there is a freezing situation at the abnormal position according to the inspection image; When there is a freezing situation at the abnormal position, control a preset alarm device to issue a freezing alarm according to the abnormal position.

8. A method for multi-sensor timestamp alignment of a humanoid robot according to claim 7, characterized in that, The freezing processing method further includes: When there is a freezing situation at the abnormal position, determine the activity range of the driving device according to the driving number; Determine the heating path according to the activity range and the abnormal position; Determine the heating temperature according to the environmental temperature and the abnormal humidity; Control a preset drone to heat the abnormal position according to the heating temperature according to the heating path.

9. A system for multi-sensor timestamp alignment of a humanoid robot, characterized in that, It includes: An acquisition module, configured to acquire image records, data records, inspection images, post-gripping images, tearing images, environmental temperature, and abnormal humidity; A memory, configured to store a program of a method for multi-sensor timestamp alignment of a humanoid robot as described in any one of claims 1 to 8; A processor, and the program in the memory can be loaded and executed by the processor.

10. An intelligent terminal, characterized in that, It includes a memory and a processor, and a computer program capable of being loaded and executed by the processor and being a method for multi-sensor timestamp alignment of a humanoid robot as described in any one of claims 1 to 8 is stored on the memory.

Images