Robot sensor calibration method and device and robot

Abstract

The invention relates to a robot sensor calibration method and device and a robot. The robot is provided with a first visual sensor and a second visual sensor. The method comprises the steps that the feature point matching condition of a first collected image and a second collected image of the robot under at least one same pose is displayed; the first acquired image is an image acquired by a first visual sensor in a calibration mode, and the second acquired image is an image acquired by a second visual sensor in the same pose; in response to a pose selection operation, obtaining a target feature point meeting a calibration requirement according to a feature point matching condition of the selected pose; and calibrating the relative poses of the first visual sensor and the second visual sensor based on the target feature point. According to the method, the feature points of the images collected by the two vision sensors autonomously detected by the machine are displayed, the target feature points meeting the calibration requirements are manually selected, the robot is helped to find the target feature points meeting the calibration requirements through participation of the user, and the calibration accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a calibration method and device for a robot sensor, and a robot. Background Art

[0002] With the development of robot technology, robots have greatly brought convenience to people's lives. Among them, cleaning robots are becoming an important component of homes and other entertainment venues, playing an increasingly important role in cleaning services.

[0003] Maps are the basis for the robot to walk autonomously. In order to build a map, the robot can be equipped with a first visual sensor and a second visual sensor, through which a three-dimensional map of the robot can be built. For example, the first visual sensor can be a camera, and the second visual sensor can be a 3D-TOF. In the case of two visual sensors, the relative position of the two visual sensors needs to be calibrated. However, the traditional calibration method has the problem of inaccurate calibration. Summary of the invention

[0004] Based on this, it is necessary to provide a calibration method, device, robot, computer-readable storage medium and computer program product for a robot sensor that can improve the calibration accuracy in order to address the above technical problems.

[0005] In a first aspect, the present application provides a robot sensor calibration method, wherein the robot is provided with a first visual sensor and a second visual sensor, and the method comprises:

[0006] Demonstrate the matching of feature points between the first acquired image and the second acquired image of the robot in at least one same posture; the first acquired image is an image acquired by the first visual sensor in the calibration mode, and the second acquired image is an image acquired by the second visual sensor in the same posture;

[0007] In response to the selection operation of the posture, obtaining target feature points that meet the calibration requirements according to the matching conditions of the feature points of the selected posture;

[0008] The relative position and posture of the first visual sensor and the second visual sensor are calibrated based on the target feature points.

[0009] In a second aspect, the present application provides a robot sensor calibration device, the device comprising:

[0010] A display module, used to display the matching of feature points between a first acquired image and a second acquired image of the robot in at least one same posture; the first acquired image is an image acquired by the first visual sensor in a calibration mode, and the second acquired image is an image acquired by the second visual sensor in the same posture;

[0011] A selection module, for obtaining target feature points that meet calibration requirements according to feature point matching conditions of the selected posture in response to a selection operation of the posture;

[0012] A calibration module is used to calibrate the relative position and posture of the first visual sensor and the second visual sensor based on the target feature points.

[0013] In a third aspect, the present application provides a robot, wherein the robot is provided with a first visual sensor and a second visual sensor, and the robot is also provided with a controller, wherein the controller is electrically connected to the first visual sensor and the second visual sensor, respectively, and the controller has a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the above embodiments are implemented.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the processor implements the steps of the method described in the above embodiments when executing the computer program.

[0015] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the methods described in the above embodiments when executed by a processor.

[0016] The robot sensor calibration method, device, robot, computer-readable storage medium and computer program product can automatically detect the matching feature points of the captured images of the two sensors in at least one same posture by displaying the matching of feature points of the first captured image and the second captured image of the robot in at least one same posture, so as to facilitate the user to select the target feature points that meet the calibration requirements based on the feature point matching, and then calibrate the relative posture of the first visual sensor and the second visual sensor based on the target feature points. This method can help the robot find the target feature points that meet the calibration requirements through the participation of the user by displaying the feature points of the captured images of the two visual sensors detected autonomously by the machine, combined with the method of manually selecting the target feature points that meet the calibration requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the structure of the sensing and control part of a robot;

[0018] Figure 2is a schematic flow chart of a calibration method for a robot sensor in one embodiment;

[0019] Figure 3 This is a schematic diagram of a display interface for feature point matching in an embodiment;

[0020] Figure 4 A schematic diagram of a display interface for feature point matching in another embodiment;

[0021] Figure 5 is a schematic flow chart of a calibration method for a robot sensor in yet another embodiment;

[0022] Figure 6 is a structural block diagram of a calibration device for a robot sensor in one embodiment;

[0023] Figure 7 FIG. 4 is a diagram showing the internal structure of a robot controller in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] like Figure 1 As shown, the sensing and control part 10 of the robot includes: a controller 101, a driver 102, multiple types of sensors 103, and a communication module 104. The multiple types of sensors may include a first visual sensor and a second visual sensor. Among them, each sensor in the multiple types of sensors 103 and the driver 102 are connected to the controller 101 signal, and the controller 101 processes the sensor data collected by the multiple types of sensors, makes a decision, generates a control instruction based on the decision result, and sends the control instruction to the driver 102. The driver 102 may include a driver for the robot driving wheel, a driver for the module adjustment device, etc. The decision may include navigation, obstacle avoidance, and control of various components of the robot. Based on the communication module 104, it can communicate with the server 20, and the server 20 can be connected to the user terminal 30 for communication, and the control instruction can be sent to the robot through the user terminal 30. For example, robot operation instructions, robot charging instructions, robot map building instructions, etc. It should be understood that the user terminal 30 is installed with robot-related applications and can send relevant instructions to the robot.

[0026] The user terminal 30 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0027] The robot refers to a self-moving device equipped with a sensor device that can move autonomously to perform tasks. In one embodiment, the robot includes but is not limited to a cleaning robot, a food delivery robot, and a logistics robot. The cleaning robot includes robots used for cleaning home and non-home scenes, such as sweepers, mops, automatic floor scrubbers, entertainment venues (such as swimming pool cleaners), and other cleaning robots.

[0028] In one embodiment, Figure 2 As shown, a robot sensor calibration method is provided, wherein the robot is provided with a first visual sensor and a second visual sensor. Figure 1 The controller or user terminal in the example is used to illustrate, including the following steps:

[0029] Step 202: Display the matching of feature points between the first collected image and the second collected image in at least one same posture of the robot.

[0030] Among them, the first acquired image is an image acquired by the first visual sensor in the calibration mode. The second acquired image is an image acquired by the second visual sensor in the calibration mode at the same posture as the first visual sensor. The first acquired image and the second acquired image in the same posture are specifically images acquired by the first visual sensor and the second visual sensor of the robot at the same time. The first visual sensor is a camera, the second visual sensor is a 3D-TOF, or the first visual sensor is a camera, and the second visual sensor is a radar. The first visual sensor and the second visual sensor can make a three-dimensional map. It is understandable that when the robot is equipped with more than two visual sensors, the feature point matching of the images acquired by at least two different visual sensors of the robot in the same posture can also be displayed to calibrate at least two different visual sensors.

[0031] The calibration mode refers to the calibration operation of the robot. The robot can enter the calibration mode in response to the user's calibration command, or it can enter the calibration mode autonomously when a calibration anomaly is detected. During the robot operation, calibration anomalies may occur due to the influence of the robot's movement, which may affect the business. Through the robot's calibration operation, the relative position between the first visual sensor and the second visual sensor is calibrated, thereby improving the accuracy of the robot's data collection.

[0032] After entering the calibration mode, the robot is controlled to walk within the working range, and the first visual sensor of the robot is controlled to acquire a first acquired image, and the second visual sensor of the robot is controlled to acquire a second acquired image.

[0033] Further, the first captured image and the second captured image in the same posture are identified, and based on the calibration requirements, feature points in postures that meet the calibration requirements are selected. Based on the calibration requirements, obvious and easily identifiable feature points in machine postures are required. Machine postures with obvious feature points need to meet the following requirements:

[0034] 1. Number of feature points: There are multiple obvious feature points near the machine posture, which can be detected by computer vision algorithms. They can be more angular or have areas with sharp color contrast.

[0035] 2. Feature point distribution: The feature points near the machine posture are evenly distributed without obvious aggregation or sparseness.

[0036] 3. Feature point distinguishability: Feature points near the machine posture can be accurately matched and identified without confusion or mismatching, that is, these points are not very similar in shape or have very similar color change patterns.

[0037] 4. Relationship between feature points and the environment: Feature points near the machine posture that have an obvious relationship with the environment are given priority, such as corners, door frames, furniture, etc.

[0038] Based on the above-mentioned feature point requirements, categories such as wall corners, door frames, and furniture can be identified, and a feature point area with evenly distributed feature points can be determined, and the pose corresponding to the feature area can be determined as a candidate pose.

[0039] Based on the above principles, feature points on the first acquired image and the second acquired image are identified respectively, and then based on the image features, matching feature points of the first acquired image and the second acquired image at the same position are determined, so as to obtain the matching of feature points of the first acquired image and the second acquired image at at least one same position. The matching situation includes at least one of the number of matching feature points of the first acquired image and the second acquired image, the matching degree of feature points, and the number of matching feature points.

[0040] Step 204 , in response to the posture selection operation, obtaining target feature points that meet the calibration requirements according to the feature point matching of the selected posture.

[0041] In some positions with obvious feature points, the robot can calibrate the two visual sensors based on the feature points in the position. However, in some scenarios, the robot's working environment is relatively clean and it is difficult to find suitable feature points. In this case, the automatic calibration method is likely to result in low calibration accuracy, which in turn affects the robot's subsequent operations. For example, if the working environment is a pure white wall or a symmetrical wall, it is difficult for the robot to find the feature points that meet the requirements for calibration.

[0042] In this embodiment, by showing the matching of feature points of the first collected image and the second collected image in the same posture of the robot, the user can intuitively understand the matching of feature points. On this basis, the target feature points for calibration are further selected.

[0043] Specifically, based on the matching of feature points of each displayed posture, the user triggers a selection operation. In response to the selection operation of the posture, the target feature points that meet the calibration requirements are obtained according to the matching of feature points of the selected posture. For example, the matching feature points of the first acquired image and the second acquired image under the selected posture are used as target feature points. Through the participation of the user, this method can help the robot find the target feature points that meet the calibration requirements, thereby improving the accuracy of calibration.

[0044] Step 206: calibrate the relative position and posture of the first vision sensor and the second vision sensor based on the target feature points.

[0045] Specifically, the relative position of the first visual sensor and the second visual sensor is solved according to the different positions of the target feature point in the first visual sensor coordinates and the second visual sensor coordinates. The specific calibration process can be implemented using a calibration tool provided by an open source library such as OpenCV.

[0046] It should be noted that the relative posture of the first visual sensor and the second visual sensor is calibrated based on the target feature points, which is used to calibrate the relative posture between the two visual sensors of the robot. In addition to calibrating the posture between different sensors of the robot, a set of parameters, namely the factory parameters, will be calibrated for the robot using professional fixtures before leaving the factory, and the factory parameters will be permanently retained and users are not allowed to change them.

[0047] The robot sensor calibration method described above can automatically detect the matching feature points of the captured images of the two sensors in at least one same posture by displaying the matching of the feature points of the first captured image and the second captured image of the robot in at least one same posture, so that the user can select the target feature points that meet the calibration requirements based on the feature point matching, and then calibrate the relative posture of the first visual sensor and the second visual sensor based on the target feature points. This method can help the robot find the target feature points that meet the calibration requirements through the participation of the user by displaying the feature points of the captured images of the two visual sensors detected autonomously by the machine, combined with the method of manually selecting the target feature points that meet the calibration requirements.

[0048] In another embodiment, the matching of feature points between a first acquired image and a second acquired image of the robot in at least one same posture is displayed, including: in response to a switching operation, displaying the first acquired image and the second acquired image in a candidate posture on the same interface; the matching degree of feature points between the first acquired image and the second acquired image in the candidate posture is greater than a threshold; and identifying matching feature points in the first acquired image and the second acquired image.

[0049] The switching operation includes an operation that enables human-computer interaction to switch pages or page contents, such as a touch sliding operation and a touchless sliding operation. For example, sliding to the left switches backward, and sliding to the right switches forward. In this embodiment, the first captured image and the second captured image under multiple candidate postures can be displayed through the switching operation.

[0050] Among them, the candidate pose is the pose with the matching degree of the feature points of the first acquisition image and the second acquisition image. In this embodiment, the matching degree of the feature points of the first acquisition image and the second acquisition image of each pose is first calculated, and the pose with the matching degree of the feature points of the first acquisition image and the second acquisition image greater than the threshold is determined as the candidate pose. Therefore, the poses presented to the user for selection are those that have been autonomously screened by the machine and have a matching degree greater than the threshold, which reduces the number of poses that the user needs to view and improves the calibration efficiency.

[0051] In another embodiment, the candidate postures may be displayed in descending order according to the matching degree of the candidate postures in response to the switching operation. That is, the candidate posture with the highest matching degree is displayed first, and in response to the backward switching operation, the candidate postures ranked later are displayed in sequence. The advantage of this is that the candidate postures with high matching degree can be seen by the user first, and the user may be able to select the target posture after several switches, which reduces the number of postures that the user needs to view and improves the calibration efficiency.

[0052] The matching degree of the feature points of the first acquired image and the second acquired image in the candidate posture is greater than the threshold. That is, after acquiring the first acquired image and the second acquired image in each posture, the feature points of the first acquired image and the second acquired image are first identified. The feature points are points with obvious features that meet the calibration requirements (such as table corners, wall corners, etc.). Based on the image features, the feature points of the first acquired image and the second acquired image are matched. By matching the feature points of the first acquired image and the second acquired image, the feature points of the same target in the first acquired image and the feature points of the second acquired image can be determined.

[0053] On this basis, the feature points on the first acquired image and the feature points on the second acquired image that match are identified. In one embodiment, a schematic diagram of a display interface for matching feature points is provided, where the first acquired image is displayed on the left side of the interface, and the first acquired image is displayed on the right side of the interface, and the feature points on the first acquired image and the feature points on the second acquired image that match are simultaneously identified on the same display interface, so that the user can intuitively understand the matching of the feature points of the first acquired image and the second acquired image in the same posture with the naked eye, thereby helping the user to select target feature points that meet the calibration requirements.

[0054] In another embodiment, identifying the matching feature points in the first acquired image and the second acquired image may include: connecting the matching feature points in the first acquired image and the second acquired image with lines. That is, on the same display interface, the matching feature points on the first acquired image and the second acquired image are connected with lines, and the user can intuitively understand the matching of the feature points of the first acquired image and the second acquired image in the same posture through the connection.

[0055] Furthermore, the first collected image and the second collected image are preprocessed to the same size and symmetrically arranged. Then the connecting line of the matched feature points should be a straight line, and the connecting lines of each matched feature point should be parallel. Figure 3 As shown, the first acquired image is the image displayed on the left, and the second acquired image is the image displayed on the right. The matching feature points in the first acquired image and the second acquired image are connected by lines, so that through the parallelism of the connecting lines of the matching feature points, the user can very intuitively understand the matching of the feature points of the first acquired image and the second acquired image in the same posture. For example, if the connecting lines of the matching feature points of the first acquired image and the second acquired image in this posture are all parallel straight lines, it means that the matching degree of the feature points in this posture is high.

[0056] In another embodiment, identifying matching feature points in the first acquired image and the second acquired image may include: determining a first area on the first acquired image where the matching feature points are located, and determining a second area on the second acquired image where the matching feature points are located; identifying a first area on the first acquired image, matching feature points on the first area, the second area on the second acquired image, and matching feature points on the second area.

[0057] Specifically, the feature points of each acquired image can be clustered, and the feature points within a certain range of feature point distances are determined to belong to the same area, and the area is identified to obtain the area where the feature points are located. It can be understood that, according to clustering, there may be one or more areas where feature points are located on each acquired image. Then, the first area of ​​the first acquired image, the matched feature points on the first area, the second area on the second acquired image, and the matched feature points of the second area are identified.

[0058] Specifically, the first region of the first collected image and the second region of the second collected image with the same image features can be marked in the same marking manner, so that the regions with the same image features can be intuitively identified in the two collected images. Figure 4 As shown, the first acquired image is the image displayed on the left, and the second acquired image is the image displayed on the right. By identifying the first area of ​​the first acquired image, identifying the matching feature points of the first area, identifying the second area of ​​the second acquired image, and identifying the matching feature points of the second area, the user can intuitively understand the feature point matching status of the first acquired image and the second acquired image in the same posture through the number of areas on each identified acquired image, the shape of the area, and the number of matching feature points in the area.

[0059] In another embodiment, in response to the selection operation of the posture, target feature points that meet the calibration requirements are obtained according to the feature point matching situation of the selected posture, including: in response to the posture selection operation, the target posture selected based on the feature point matching situation is obtained; if at least one of the selection number of the target posture and the feature point matching degree between the first acquired image and the second acquired image under the same target posture meets the requirements, the matching feature points of the first acquired image and the second acquired image under the target posture are used as target feature points.

[0060] In this embodiment, the switching operation can present the matching of feature points between the first acquired image and the second acquired image in each position, which can be selected by the user. In one embodiment, the switching operation can present the matching of feature points between the first acquired image and the second acquired image in each candidate position, which can be selected by the user.

[0061] Experienced users know that the target posture should be selected based on the number of matched feature points and shape matching. If at least one of the number of selected target postures and the degree of matching of feature points of the first acquired image and the second acquired image under the same target posture meets the requirements, the matched feature points of the first acquired image and the second acquired image under the target posture are used as target feature points.

[0062] The number of target postures selected meets the requirements, which may mean that the number of target postures selected is greater than a threshold. For example, the number of selected target postures is greater than or equal to three. It may also mean that the matching degree of feature points between the first acquired image and the second acquired image meets the requirements. For example, when the matching degree of feature points between the first acquired image and the second acquired image is greater than a threshold, it means that the matching degree of feature points under this posture is high, and the calibration requirements can be met based on the feature points under this posture. It may also mean that the number of target postures selected and the matching degree of feature points between the first acquired image and the second acquired image under the same target posture both meet their respective requirements.

[0063] In this embodiment, the target posture is selected through the user's selection operation. When at least one of the selected number of target postures or the matching degree of the feature points of the first acquired image and the second acquired image meets the requirements, the target feature points are determined and calibration is triggered. For example, after three target postures are selected, the calibration button height on the interface can be calibrated based on the target feature points after the user selects it. This human-computer interaction method provides guidance for the user to perform calibration operations and improves the accuracy of the target feature point selection.

[0064] In one embodiment, if the number of choices for the target posture and the degree of feature point matching between the first acquired image and the second acquired image under the same target posture do not meet the requirements, a switching prompt is output to instruct the user to switch posture and reselect a posture.

[0065] Specifically, if the matching degree of the feature points of the first captured image and the second captured image of the robot in each posture is less than the threshold, in order to improve the matching accuracy, a prompt can be output to remind the user to find a real environment with obvious feature points and place the robot in the environment. The calibration accuracy can be improved by manually helping to select postures with obvious feature points. For example, the prompt "The current environment is relatively monotonous and not complex enough. Please place the robot in a more complex space" can be output, and a picture prompt of a complex space can be output to help people quickly locate the complex space.

[0066] In another embodiment, a target posture selection prompt may be displayed to guide the user on how to select a suitable target posture during the process of selecting the target posture. For example, when switching each target posture, a text prompt is displayed, "The more feature points of the target posture, the better, and the closer the shape, the better", or another example, a correct target posture picture and an incorrect target posture picture are displayed. By displaying the target posture selection prompt, it is possible to help inexperienced users choose the appropriate target posture.

[0067] In another embodiment, it can also be determined whether the user is an experienced user. If the user is an inexperienced user, the target posture selection prompt is displayed. If the user is an experienced user, the target posture selection prompt is closed. Among them, whether the user is an experienced user. Whether the user is an experienced user can be determined based on whether the user has selected the target posture to participate in the calibration, and the number of times the target posture is selected to participate in the calibration. In this embodiment, different prompt processing is performed on experienced users and inexperienced users to improve the user experience.

[0068] In another embodiment, the robot sensor calibration method further includes: obtaining a calibration instruction for the visual sensor based on human-computer interaction; in response to the calibration instruction, controlling the robot to run, and collecting data from the first visual sensor and the second visual sensor during the movement.

[0069] Human-machine interaction is the interaction between humans and machines. Specifically in this application, human-machine interaction can be the interaction between humans and robots, or the interaction between humans and the robot's control terminal. The control terminal is a terminal with a robot management application installed, that is, Figure 1 The user terminal in.

[0070] One way of human-machine interaction is that a person issues a calibration command by voice, and the robot collects and recognizes the voice signal to obtain the calibration command. Another way of human-machine interaction is that a person touches a calibration button on the robot body, and the robot obtains the calibration command.

[0071] A human-machine interaction method is implemented through a human-machine interaction interface of a robot control terminal. The application of the control terminal is provided with a calibration interface, and when a calibration button of the calibration interface is detected to be triggered, a calibration instruction for the visual sensor is obtained.

[0072] When a calibration instruction for the visual sensor is obtained based on human-machine interaction, the machine is controlled to move in the working area, and the first visual sensor and the second visual sensor collect data during the movement.

[0073] In this embodiment, the calibration of the visual sensor is triggered by human-computer interaction, which increases the convenience of calibration.

[0074] In another embodiment, the robot sensor calibration method further includes: when an abnormality in the robot's visual perception is detected, outputting a calibration abnormality prompt, wherein the calibration abnormality prompt is used to instruct the user whether to recalibrate the visual sensor.

[0075] The field of view of the traditional robot's visual sensor is fixed. The field of view refers to the angle between the outer contour of the module where the sensor can obtain data and the center of the module, which is an angle range. The field of view includes the horizontal field of view and the vertical field of view, which is analogous to the range of viewing angles that the human eye can see when the eyeball is not moving. In some application scenarios, in order to increase the field of view of the visual sensor and increase the viewing range of the robot, the visual sensor is fixed on an adjustment device that can move in at least one direction to adjust the perception range of the visual sensor in that direction. In this application scenario, due to the change in the position of the robot's sensor, the robot's visual perception is easily abnormal, and there is a need for recalibration.

[0076] When the robot's visual perception is abnormal, it is usually caused by calibration abnormality. In this embodiment, during the operation of the robot, when the robot's visual perception is detected to be abnormal, a calibration abnormality prompt is output. Among them, the visual perception abnormality can be determined from the image recognition result or the mapping quality. If the image recognition result is abnormal, or the mapping quality is low, it can be determined that the robot's visual perception is abnormal.

[0077] The calibration abnormality prompt is used to instruct the user whether to recalibrate the visual sensor. Therefore, the method of outputting the calibration abnormality prompt should be based on the method in which the user can obtain relevant reminders in a timely manner. In one embodiment, the output method of the calibration abnormality prompt includes: controlling the robot to output a voice prompt of the calibration abnormality. In one embodiment, the output method of the calibration abnormality prompt includes: outputting an information prompt of the calibration abnormality to the user terminal, and the information prompt may include a voice prompt and a text prompt. For example, the calibration abnormality prompt may be output to the user terminal in the form of a pop-up window or a voice prompt.

[0078] In one embodiment, if the user is detected near the robot, the robot is controlled to output a voice prompt of calibration abnormality. If the user is detected not near the robot, an information prompt of calibration abnormality is output to the user terminal.

[0079] After the calibration abnormality prompt is output, the user can obtain the calibration instruction for the visual sensor through human-computer interaction. For example, the robot's visual sensor can be controlled to collect data through voice interaction or interaction with the APP on the user terminal.

[0080] In this embodiment, when an abnormality in the robot's visual perception is detected, a calibration abnormality prompt is output, which enables the user to promptly understand the fact of the calibration abnormality and facilitates the user to understand the working dynamics of the robot.

[0081] In another embodiment, the robot self-detects whether visual perception is abnormal. Detecting whether the robot's visual perception is abnormal includes at least one of the following three methods.

[0082] The first method is to perform target recognition based on the collected data of the robot's visual sensor. If the confidence of the recognized target is less than a threshold, it is determined that the robot's visual perception is abnormal.

[0083] Specifically, if the confidence of various targets identified based on the data collected by the robot's visual sensor is generally very low, it can be considered that the position of the visual sensor is significantly deviated and needs to be recalibrated.

[0084] The second type: obtaining multiple frames of real-time three-dimensional point cloud data based on the collected data of the robot's visual sensor; if the matching degree between the multiple frames of historical three-dimensional point cloud and the real-time point cloud data is less than a threshold, determining that the robot's visual perception is abnormal.

[0085] The real-time 3D point cloud can be obtained by fusing the image data collected by the image sensor and the depth data collected by the depth sensor. If the matching degree between the multi-frame historical 3D point cloud and the real-time point cloud data is less than the threshold, it can be considered that the position of the depth sensor has a large deviation and needs to be recalibrated.

[0086] The third type: if the matching degree of the feature points of the image collected by the first visual sensor and the feature points of the image collected by the second visual sensor at the same viewing angle is less than a threshold, it is determined that the visual perception of the robot is abnormal.

[0087] In this embodiment, the feature points collected by the image sensor within the repetitive range of the field of view are matched with the feature points collected by the depth sensor. If the matching value is very small, it can be considered that there is a large deviation in the relative positions of the two and recalibration is required.

[0088] Any abnormality in any of the above three data indicates that the robot's vision is abnormal and triggers the recalibration of the visual sensor.

[0089] In this embodiment, the robot actively detects the data collected by the visual sensor, and triggers the visual sensor to recalibrate in time when an abnormality is detected.

[0090] In another embodiment, the robot sensor calibration method further includes: displaying a visual perception result obtained based on the relative posture of the first visual sensor and the second visual sensor after calibration; the visual perception result is used to represent the calibration effect.

[0091] Specifically, after the calibration result of the visual sensor parameters, the visual perception result is obtained based on the calibrated visual sensor parameters. The visual perception result obtained after calibration can be displayed through the relevant interface of the user terminal, and the user can judge the calibration effect from the visual effect based on the displayed visual perception result.

[0092] In one embodiment, after calibration, a three-dimensional map is generated based on the collected data of the first visual sensor and the second visual sensor, and the three-dimensional map is displayed to the user, so that the user can intuitively judge the calibration effect based on the displayed three-dimensional map in combination with the real three-dimensional map.

[0093] In one implementation, if the user confirms that the calibration effect is not ideal, the user can continue to trigger the calibration through human interaction.

[0094] In one embodiment, before the visual perception result is displayed, the machine can also perform a self-check to determine whether the robot's visual perception is abnormal. If the self-check passes, the visual perception result obtained based on the relative position of the first visual sensor and the second visual sensor after calibration is displayed.

[0095] Specifically, the robot detects whether the visual perception of the robot is abnormal, including at least one of the following methods: the first method: performing target recognition based on the collected data of the robot's visual sensor, if the confidence of the recognized target is less than a threshold value, determining that the robot's visual perception is abnormal; the second method: acquiring multiple frames of real-time three-dimensional point cloud data based on the collected data of the robot's visual sensor; if the matching degree between the multiple frames of historical three-dimensional point clouds and the real-time point cloud data is less than a threshold value, determining that the robot's visual perception is abnormal; the third method: if the matching degree between the feature points of the image collected by the first visual sensor and the feature points of the image collected by the second visual sensor of the same perspective is less than a threshold value, determining that the robot's visual perception is abnormal.

[0096] In another embodiment, after displaying the visual perception result, an option may be further output for the user to confirm whether to use the relative posture of the currently calibrated first visual sensor and the second visual sensor as subsequent parameters, and the user may be prompted whether to use the parameter as the historical optimal parameter.

[0097] The above-mentioned robot sensor calibration method obtains calibration instructions for the visual sensor through human-computer interaction, and displays the visual perception results obtained based on the parameters of the calibrated visual sensor after the calibration is completed. The user can judge the calibration effect from the visual effect based on the displayed visual perception results.

[0098] In one embodiment, Figure 5 As shown, a robot sensor calibration method comprises:

[0099] Step 502: when an abnormality in the robot's visual perception is detected, a calibration abnormality prompt is output, wherein the calibration abnormality prompt is used to instruct the user whether to recalibrate the visual sensor.

[0100] Wherein, detecting whether the visual perception of the robot is abnormal includes at least one of the following methods:

[0101] The first method is to perform target recognition based on the collected data of the robot's visual sensor. If the confidence of the recognized target is less than a threshold, it is determined that the robot's visual perception is abnormal.

[0102] The second type: obtaining multiple frames of real-time three-dimensional point cloud data based on the collected data of the robot's visual sensor; if the matching degree between the multiple frames of historical three-dimensional point cloud and the real-time point cloud data is less than a threshold, determining that the robot's visual perception is abnormal.

[0103] The third type: if the matching degree of the feature points of the image collected by the first visual sensor and the feature points of the image collected by the second visual sensor at the same viewing angle is less than a threshold, it is determined that the visual perception of the robot is abnormal.

[0104] Step 504: obtain calibration instructions for the visual sensor based on human-computer interaction.

[0105] Step 506, in response to the calibration instruction, controlling the robot to run, and collecting data by the first visual sensor and the second visual sensor during the movement.

[0106] Step 508: In response to the switching operation, the first captured image and the second captured image are displayed in the same interface at the candidate posture; the first captured image is an image captured by the first visual sensor in the calibration mode, and the second captured image is an image captured by the second visual sensor at the same posture. The matching degree of the feature points of the first captured image and the second captured image at the candidate posture is greater than a threshold.

[0107] Step 510: Identify matching feature points in the first collected image and the second collected image.

[0108] The identifying of the matching feature points in the first acquired image and the second acquired image includes any one of the following methods: the first method: connecting the matching feature points in the first acquired image and the second acquired image with lines. The second method: determining the first area where the matching feature points on the first acquired image are located, and determining the second area where the matching feature points on the second acquired image are located; identifying the first area of ​​the first acquired image, the matching feature points on the first area, the second area on the second acquired image, and the matching feature points on the second area.

[0109] Step 512: In response to the posture selection operation, a target posture selected based on the feature point matching situation is obtained.

[0110] Step 514: If at least one of the number of selections of the target posture and the degree of matching of feature points between the first acquired image and the second acquired image at the same target posture meets the requirements, the matching feature points between the first acquired image and the second acquired image at the target posture are used as target feature points.

[0111] Step 516: calibrate the relative position and posture of the first vision sensor and the second vision sensor based on the target feature points.

[0112] Step 518, displaying the visual perception result obtained based on the relative posture of the first visual sensor and the second visual sensor after calibration; the visual perception result is used to represent the calibration effect.

[0113] In this embodiment, the method triggers the calibration of the visual sensor through human-computer interaction, which increases the convenience of calibration. At the same time, the user's participation in the calibration process of the visual sensor is increased, and the interaction between the robot and the user is increased, so that the user can intuitively experience the calibration process and calibration effect of the robot's visual sensor, thereby increasing the stickiness between the user and the product.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a robot sensor calibration device for implementing the robot sensor calibration method involved above. The implementation solutions provided by each device to solve the problem are similar to the implementation solutions recorded in the above method, so the specific limitations in the embodiments of each device provided below can refer to the limitations of the method above, and will not be repeated here.

[0116] In one embodiment, a calibration device for a robot sensor is provided, such as Figure 6 As shown, including:

[0117] The display module 602 is used to display the feature point matching between the first collected image and the second collected image of the robot in at least one same posture; the first collected image is an image collected by the first visual sensor in the calibration mode, and the second collected image is an image collected by the second visual sensor in the same posture.

[0118] The selection module 604 is used to obtain target feature points that meet the calibration requirements according to the feature point matching conditions of the selected posture in response to the selection operation of the posture.

[0119] The calibration module 606 is used to calibrate the relative position and posture of the first visual sensor and the second visual sensor based on the target feature points.

[0120] In another embodiment, presenting comprises:

[0121] The switching module is used to display the first collected image and the second collected image in the candidate posture on the same interface in response to the switching operation; the matching degree of the feature points of the first collected image and the second collected image in the candidate posture is greater than a threshold.

[0122] An identification module is used to identify matching feature points in the first collected image and the second collected image.

[0123] In another embodiment, the identification module is used to connect the matched feature points in the first acquired image and the second acquired image with lines. Or, the identification module is used to determine the first area where the matched feature points on the first acquired image are located, and determine the second area where the matched feature points on the second acquired image are located; identify the first area of ​​the first acquired image, the matched feature points on the first area, the second area on the second acquired image, and the matched feature points on the second area.

[0124] In another embodiment, the selection module comprises:

[0125] The selection operation module is used to obtain the target posture selected based on the matching situation of the feature points in response to the posture selection operation.

[0126] The target determination module is used to use the matching feature points of the first acquired image and the second acquired image under the target posture as target feature points if at least one of the selection number of the target posture and the matching degree of feature points of the first acquired image and the second acquired image under the same target posture meets the requirements.

[0127] In another embodiment, the selection module further includes a prompt module for displaying a target posture selection prompt.

[0128] In another embodiment, the calibration device for the robot sensor further includes:

[0129] Specify an acquisition module for obtaining calibration instructions for a visual sensor based on human-computer interaction.

[0130] The control module is used to control the operation of the robot in response to the calibration instruction, and the first visual sensor and the second visual sensor collect data during the movement.

[0131] In another embodiment, the calibration device for the robot sensor further includes:

[0132] The detection module is used to output a calibration abnormality prompt when an abnormality in the robot's visual perception is detected. The calibration abnormality prompt is used to instruct the user whether to recalibrate the visual sensor.

[0133] In another embodiment, the detection module is used to perform target recognition based on the collected data of the robot's visual sensor, and if the confidence of the recognized target is less than a threshold, the robot's visual perception is determined to be abnormal. Or, the detection module is used to obtain multiple frames of real-time three-dimensional point cloud data based on the collected data of the robot's visual sensor; if the matching degree of the multiple frames of historical three-dimensional point cloud and the real-time point cloud data is less than a threshold, the robot's visual perception is determined to be abnormal. Or, the detection module is used to determine that the robot's visual perception is abnormal if the matching degree of the feature points of the image collected by the first visual sensor and the feature points of the image collected by the second visual sensor at the same viewing angle is less than a threshold.

[0134] In another embodiment, the calibration device for the robot sensor further includes:

[0135] An effect display module is used to display a visual perception result obtained based on the relative posture of the first visual sensor and the second visual sensor after calibration; the visual perception result is used to represent the calibration effect.

[0136] Each module in the above-mentioned robot sensor calibration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0137] In one embodiment, a robot is provided, and the internal structure diagram of its controller can be as follows: Figure 7As shown. The computer device includes a processor and a memory connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a calibration method for a robot sensor is implemented.

[0138] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods of the above embodiments are implemented.

[0140] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the methods of the above embodiments when executed by a processor.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot sensor calibration method, wherein the robot is provided with a first visual sensor and a second visual sensor, It is characterized in that The method comprises: Demonstrate the matching of feature points between the first acquired image and the second acquired image of the robot in at least one same posture; the first acquired image is an image acquired by the first visual sensor in the calibration mode, and the second acquired image is an image acquired by the second visual sensor in the same posture; In response to the selection operation of the posture, obtaining target feature points that meet the calibration requirements according to the matching conditions of the feature points of the selected posture; The relative position and posture of the first visual sensor and the second visual sensor are calibrated based on the target feature points.

2. The method according to claim 1, It is characterized in that The matching of feature points of the first collected image and the second collected image of the display robot in at least one same posture includes: In response to the switching operation, the first collected image and the second collected image are displayed in the same interface at the candidate posture; the matching degree of the feature points of the first collected image and the second collected image at the candidate posture is greater than a threshold; Matching feature points in the first acquired image and the second acquired image are identified.

3. The method according to claim 2, It is characterized in that The identifying the matching feature points in the first collected image and the second collected image includes any one of the following methods: The first method: connecting the matching feature points in the first collected image and the second collected image with lines; The second method: determining a first area where the matching feature points on the first acquired image are located, and determining a second area where the matching feature points on the second acquired image are located; A first area of ​​the first acquired image, matching feature points on the first area, the second area of ​​the second acquired image, and matching feature points on the second area are identified.

4. The method according to claim 1, It is characterized in that In response to the selection operation of the posture, obtaining target feature points that meet the calibration requirements according to the feature point matching of the selected posture includes: In response to the posture selection operation, obtaining a target posture selected based on the matching situation of the feature points; If at least one of the number of selections of the target posture and the degree of matching of feature points between the first acquired image and the second acquired image at the same target posture meets the requirements, the matching feature points between the first acquired image and the second acquired image at the target posture are used as target feature points.

5. The method according to claim 4, It is characterized in that The method further comprises: A target posture selection prompt is displayed, where the target posture selection prompt is used to guide the user on how to select the target posture, and the prompt method includes at least one of a text prompt and a picture display.

6. The method according to claim 1, It is characterized in that The method further comprises: Obtain calibration instructions for visual sensors based on human-computer interaction; In response to the calibration instruction, the robot is controlled to run, and the first visual sensor and the second visual sensor collect data during the movement.

7. The method according to claim 6, It is characterized in that The method further comprises: When an abnormality in the robot's visual perception is detected, a calibration abnormality prompt is output, and the calibration abnormality prompt is used to instruct the user whether to recalibrate the visual sensor.

8. The method according to claim 7, It is characterized in that Detecting whether the visual perception of the robot is abnormal includes at least one of the following methods: The first method is to identify a target based on the collected data of the robot's visual sensor, and if the confidence of the identified target is less than a threshold, it is determined that the robot's visual perception is abnormal; The second type: multiple frames of real-time three-dimensional point cloud data are obtained based on the collected data of the robot's visual sensor; if the matching degree between the multiple frames of historical three-dimensional point cloud and the real-time point cloud data is less than a threshold, it is determined that the robot's visual perception is abnormal; The third type: if the matching degree of the feature points of the image collected by the first visual sensor and the feature points of the image collected by the second visual sensor at the same viewing angle is less than a threshold, it is determined that the visual perception of the robot is abnormal.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: Display the visual perception result obtained based on the relative posture of the first visual sensor and the second visual sensor after calibration; the visual perception result is used to represent the calibration effect.

10. A robot sensor calibration device, wherein the robot is provided with a first visual sensor and a second visual sensor, It is characterized in that The device comprises: A display module, used to display the matching of feature points between a first acquired image and a second acquired image of the robot in at least one same posture; the first acquired image is an image acquired by the first visual sensor in a calibration mode, and the second acquired image is an image acquired by the second visual sensor in the same posture; A selection module, for obtaining target feature points that meet calibration requirements according to feature point matching conditions of the selected posture in response to a selection operation of the posture; A calibration module is used to calibrate the relative position and posture of the first visual sensor and the second visual sensor based on the target feature points.

11. A robot, It is characterized in that The robot is provided with a cleaning part, a cleaning drive device, a first visual sensor and a second visual sensor, and the cleaning drive device is connected to the cleaning part for driving the cleaning part to perform cleaning operations; the robot is also provided with a controller, and the controller is electrically connected to the first visual sensor, the second visual sensor and the cleaning drive device, respectively, and the controller has a memory and a processor, and the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of claims 1 to 9 are implemented.

Images