Robot work object positioning method and device, robot, and storage medium

Abstract

The application provides a work object positioning method and device of a robot, the robot, and a storage medium. The robot comprises a shooting device and a movable mechanism carrying the shooting device. The method comprises continuously adjusting the shooting pose of the shooting device through the movable mechanism, and controlling the shooting device to shoot a work object at different shooting poses. First depth information of the work object corresponding to each image information is obtained based on the image information obtained by the shooting device at different shooting poses. A plurality of first depth information is fused to obtain second depth information of the fused work object. The work object is positioned according to the second depth information of the fused work object. The application can improve the positioning accuracy of the robot for the work object.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, robot, and storage medium for locating work objects for a robot. Background Technology

[0002] With the continuous development of robotics technology, robots have been widely used in various fields such as industrial manufacturing, logistics and transportation, and medical assistance. For example, robots can use their onboard imaging equipment to locate objects and perform various operations on them.

[0003] However, for some metal, highly reflective, or curved objects, the robot's positioning of the object is prone to deviation due to its complex surface material and irregular structural shape, resulting in low positioning accuracy and affecting the robot's subsequent operations. Summary of the Invention

[0004] This application provides a robot positioning method, apparatus, robot, and storage medium, aiming to improve the positioning accuracy of the robot for the work object.

[0005] In a first aspect, this application provides a method for locating a work object using a robot, the robot including a camera and a movable mechanism supporting the camera, the method comprising: The shooting position of the shooting device is continuously adjusted by the active mechanism, and the shooting device is controlled to shoot the object to be worked under different shooting positions. Based on the image information captured by the shooting device under different shooting poses, the first depth information of the object to be worked corresponding to each of the image information is obtained; Multiple pieces of the first depth information are fused to obtain the fused second depth information of the object to be worked on. The object to be worked on is located based on the second depth information of the fused object.

[0006] Secondly, this application also provides a control device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements any of the robot's object localization methods provided in the embodiments of this application.

[0007] Thirdly, this application also provides a robot, comprising: The robot itself; At least one movable mechanism is provided on the robot body, and one of the movable mechanisms carries a camera device, the camera device changing its pose as the movable mechanism is adjusted; An end effector, disposed on the robot body and / or the movable mechanism, is used to perform a target operation; As provided in the embodiments of this application, the control device is used to control the active mechanism, the shooting device, and the end effector.

[0008] Fourthly, this application also provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, which can be executed by one or more processors to implement any of the robot object localization methods provided in the embodiments of this application.

[0009] This application discloses a robot object localization method, device, robot, and storage medium. By continuously adjusting the shooting posture of the shooting device through a moving mechanism, the shooting device can acquire image information of the object to be worked and the corresponding first depth information from different perspectives. Then, by fusing the first depth information corresponding to each shooting posture representing different perspectives, a second depth information with higher accuracy and completeness corresponding to the object to be worked is obtained. This avoids the problem of missing depth information due to the material characteristics of the object itself or the influence of external environmental factors under a single perspective. As a result, the robot can accurately locate the object to be worked based on the second depth information, thereby greatly improving the positioning accuracy of the robot for the object.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic flowchart illustrating the steps of a robot's object localization method according to an embodiment of this application; Figure 2 A schematic diagram of a scene where a robot takes pictures of a work object, as provided in one embodiment of this application; Figure 3 for Figure 1 A flowchart illustrating the sub-steps of the robot's task object localization method; Figure 4 for Figure 1 A flowchart illustrating the sub-steps of the robot's task object localization method; Figure 5 A schematic block diagram of a control device provided in an embodiment of this application; Figure 6 This is a schematic block diagram of a robot provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0015] This application provides a robot-based object localization method, apparatus, robot, and storage medium. The robot's object localization can be applied to a control device, which can be installed on the robot or a computer device. The robot can be, for example, an industrial robot, a android robot, or a cleaning robot. The computer device can include a server or a terminal device. The server can be a single server or a server cluster consisting of multiple servers. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device.

[0016] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] Please see Figure 1 , Figure 1 This is a schematic flowchart of a robot's task object localization method provided in one embodiment of this application.

[0018] like Figure 1 As shown, the robot's object localization method includes steps S101 to S104.

[0019] Step S101: Continuously adjust the shooting position of the shooting device through the moving mechanism, and control the shooting device to shoot the object to be worked under different shooting positions.

[0020] The moving mechanism is used to change the relative position and / or relative attitude between the shooting device and the object to be worked on. The moving mechanism can be a robotic arm, a three-axis motion platform, a gimbal, or other equipment or structure capable of adjusting the spatial position of the shooting device. The shooting device can be used to photograph the object to be worked on or to photograph the robot's surrounding environment. The shooting device can be an integrated RGB-D camera; specifically, the integrated RGB-D camera can include an RGB color camera for acquiring RGB images, and a left infrared camera and a right infrared camera for acquiring infrared images. The shooting pose represents the position and attitude of the shooting device during shooting. It can be represented by position coordinates and attitude angles, or by a pose transformation matrix including rotation matrices and translation vectors. Different shooting poses can be preset or derived from the robot's initial shooting pose. The object to be worked on can be a button, knob, operating hole, metal structural component, or other objects requiring precise positioning by the robot.

[0021] For example, let the initial observation position of the control imaging device be P0. At the initial observation position P0, multiple compensation observation positions (Pi) are first generated near the initial observation position P0 according to a preset search strategy. The compensation observation positions can be represented as: Pi = P0 + ΔPi Where ΔPi represents the spatial offset of the imaging device.

[0022] For example, suppose the preset search strategy is to control the imaging device to move a preset spatial offset along the X and Y directions of the camera coordinate system, respectively, to generate multiple compensated observation positions located above, below, to the left, and to the right of the initial observation position. (Refer to...) Figure 2 , Figure 2 The diagram illustrates a scenario where a robot photographs a work object. As shown, when the robot is at point P0, P0 is used as the initial observation position. Four compensated observation positions are generated above, below, to the left, and to the right of the initial observation position, corresponding to points P1, P2, P3, and P4 in the diagram, respectively, according to preset spatial offsets. Specifically, ΔPi = 100mm. Correspondingly, the control device will move ±100mm along the X direction and ±100mm along the Y direction under the adjustment of the movable mechanism. After the photographing device is in position, it photographs the work object at each compensated observation position.

[0023] It is understandable that by adjusting the shooting posture of the shooting device, the observation angle and observation distance of the shooting device on the object to be worked can be changed. When the shooting device cannot obtain complete position information of the object to be worked in a certain shooting posture due to the specular reflection or structural occlusion of the object, the effective position information can be obtained in other shooting postures by adjusting the shooting posture, thereby obtaining relatively more complete position information of the object to be worked, which helps to improve the positioning accuracy of the robot on the object to be worked.

[0024] Step S102: Based on the image information captured by the shooting device under different shooting poses, obtain the first depth information of the object to be worked corresponding to each image information.

[0025] The image information may include various types of images acquired through shooting at the corresponding shooting pose, such as RGB images and infrared images. The first depth information is used to characterize the depth data of the object to be worked on at the corresponding shooting pose, and may include a depth map and / or its corresponding 3D point cloud data, etc. Specifically, the depth map can be determined based on the infrared image in the image information.

[0026] For example, the image information includes RGB images and infrared images. First, the object to be worked on is identified and segmented in the RGB image, and the infrared image is analyzed to obtain a depth map that represents the depth data of the object to be worked on. Then, based on the region of the object to be worked on in the RGB image, the depth region corresponding to the object to be worked on in the depth map is determined, and the depth region is spatially registered and coordinate transformed to obtain the first depth information.

[0027] It is understandable that by processing the image information collected under different shooting poses, the depth data of the object to be worked can be obtained for each shooting pose. This can provide the robot with more reference position information for locating the object to be worked, and help the robot to accurately locate the object to be worked.

[0028] In one embodiment, such as Figure 3 As shown, step S102 includes sub-steps S1021 to S1023.

[0029] Sub-step S1021: Identify the objects to be worked on in the image information obtained under different shooting poses.

[0030] For example, RGB images at various shooting poses can be processed using object detection algorithms, instance segmentation algorithms, semantic segmentation algorithms, or recognition algorithms based on preset color, shape, and texture features to determine the position and range of the object to be worked on within the corresponding RGB image. It is understood that by recognizing the object to be worked on, it is possible to distinguish it from the background area within the overall scene being captured, reducing the impact of irrelevant background data on the subsequent localization of the object.

[0031] Sub-step S1022: Based on the recognition results of each image information, perform segmentation processing on each image information to obtain multiple target image information containing the object to be worked.

[0032] For example, a target region mask M(x,y) is generated based on the recognition result of the object to be processed in the RGB image. Here, M(x,y)=1 indicates that it belongs to the target region (i.e., the region where the object to be processed is located), and M(x,y)=0 indicates the background region. Specifically, based on the target region mask, the region of the object to be processed in the RGB image is extracted, resulting in target image information containing the object to be processed corresponding to each image information.

[0033] Understandably, by segmenting the target region, only the image and depth information corresponding to the object to be worked on can be retained, avoiding the background, workbench or other irrelevant structures from being mistakenly taken as part of the object to be worked on. Removing the interference of irrelevant objects or content helps the robot to locate the object to be worked on more accurately.

[0034] Sub-step S1023: Based on multiple target image information containing objects to be worked, obtain the first depth information corresponding to each image information.

[0035] Understandably, by first determining the target image information containing the object to be worked on, the interference of background area and other irrelevant object image data on the first depth information acquisition process can be reduced, improving the accuracy and effectiveness of the first depth information, thereby providing more reliable depth data for subsequent positioning of the object to be worked on.

[0036] In one embodiment, the target image information includes RGB image information and depth information. Based on multiple target image information containing the object to be processed, first depth information corresponding to each image information is obtained, including: performing spatial registration processing on each RGB image information and its corresponding depth information to determine a depth value matching each pixel in the RGB image information; determining the three-dimensional spatial coordinates of each pixel in each RGB image information based on the calibration parameters of the shooting device, the pixel coordinates of each pixel, and the matching depth value; and determining the first depth information corresponding to each image information based on the shooting pose corresponding to each image information and the three-dimensional spatial coordinates of each pixel in each RGB image information.

[0037] In this system, RGB image information is used to represent the color, texture, and edge features of the object to be processed, while depth information is used to represent the distance between each spatial point of the object and the imaging device. The depth information can include a depth map obtained by binocular matching of the left and right infrared images, or 3D point cloud data obtained by further transformation of the depth map. The RGB image information and depth information can be acquired by the imaging device in the same shooting pose; for example, the RGB image information and depth information can be acquired by an integrated RGB-D camera in the same shooting pose.

[0038] It should be noted that because the installation position, imaging parameters, resolution, and field of view of RGB cameras and infrared cameras may differ, the pixel positions of spatial points on the same object may differ in the RGB image and the depth map. Therefore, spatial registration processing is required for the RGB image information and the corresponding depth information according to the calibration parameters of the shooting device. This spatial registration processing may include one or more of the following: distortion correction, coordinate transformation, resolution adjustment, and image planar projection processing. Specifically, spatial registration processing is used to determine the depth value corresponding to the target pixel in the RGB image, and coordinate transformation processing is used to convert 3D points obtained from different shooting poses to the same spatial reference frame.

[0039] For example, based on the pixel coordinates (u, v) in the depth map and the corresponding depth value Z, the 3D spatial coordinates of the pixel are calculated using camera intrinsics: , The system obtains the three-dimensional points of the area corresponding to the object to be worked on in the camera coordinate system: Pcam(X,Y,Z), and combines the camera pose information to transform the depth data obtained from different observation positions to a unified coordinate system: Pworld=Ti*Pcam, thereby achieving spatial unification of depth data from different shooting poses.

[0040] It is understandable that by performing spatial registration and coordinate transformation on RGB image information and depth information, the depth data obtained under different shooting poses can correspond spatially with the object to be processed, so as to ensure the reliability of subsequent data fusion.

[0041] In some embodiments, the shooting device includes a binocular infrared camera, and the image information captured by the shooting device in different shooting poses includes a first infrared image and a second infrared image. The first infrared image and the second infrared image are respectively acquired by different cameras in the binocular camera.

[0042] Accordingly, in some embodiments, the acquisition of depth information includes: determining matching pixels in the first infrared image and the second infrared image based on the binocular matching relationship between the first infrared image and the second infrared image; and determining depth information corresponding to the image information based on the parallax between the matching pixels and the calibration parameters of the binocular infrared camera.

[0043] The calibration parameters may include at least one of the following: focal length of the binocular infrared camera, principal point coordinates, baseline distance, and distortion parameters. Parallax is used to characterize the positional difference of matched pixels in the first and second infrared images, and depth information can be determined based on the relationship between parallax and the baseline distance of the binocular infrared camera.

[0044] It is understandable that by performing binocular matching on the first infrared image and the second infrared image, the accurate depth information of the object to be operated can be obtained by utilizing the imaging differences of different cameras in the shooting device at the same target location.

[0045] In other embodiments, the imaging device includes a monocular infrared camera, and the image information captured by the imaging device in different shooting poses includes infrared images. Accordingly, the acquisition of depth information includes: determining the depth information corresponding to the image information based on the infrared images and the depth measurement parameters of the imaging device.

[0046] Step S103: Perform fusion processing on multiple first depth information to obtain the fused second depth information of the object to be worked on.

[0047] The fusion process involves mapping the first depth information, transformed from each shooting pose to a unified coordinate system, according to their spatial location, and then weighting and fusing depth data located at the same position or within a preset spatial neighborhood based on the reliability of each depth point. The second depth information represents the depth information of the object to be processed after fusing effective depth data from multiple shooting poses. The second depth information may include a complete depth map, a 3D point cloud in a unified coordinate system, or 3D spatial data of the target surface.

[0048] Understandably, by fusing depth information from multiple shooting poses, the robot can use effective depth data obtained from other shooting poses to compensate for the missing depth areas in its current / initial shooting pose. This reduces the impact of specular reflection, structural occlusion, insufficient parallax, or binocular matching failure on the target's 3D localization results, thus enabling more accurate localization of the target object.

[0049] In one embodiment, such as Figure 4 As shown, step S103 includes sub-steps S1031 to S1032.

[0050] Sub-step S1031: Obtain the image reliability of each shooting pose.

[0051] Image reliability is used to characterize the reliability of image information obtained under a given shooting pose. Image reliability can be determined for the entire target image, or it can be determined for each pixel or each three-dimensional point within the target area.

[0052] In one embodiment, obtaining the image reliability of each shooting pose includes: calculating the matching degree of at least two image information captured in each shooting pose, and determining the first effective depth information of each image information based on the matching degree corresponding to each shooting pose; calculating the first depth variation of the object to be worked on in each shooting pose, and determining the second effective depth information of each image information based on the first depth variation corresponding to each shooting pose; calculating the second depth variation of the object to be worked on in each shooting pose, and determining the third effective depth information of each image information based on the second depth variation corresponding to each shooting pose; and determining the image reliability of each shooting pose based on the first effective depth information, the second effective depth information, and the third effective depth information corresponding to each shooting pose.

[0053] Among them, each first depth variation degree is used to characterize the depth variation degree of multiple points in any local area on the object to be operated on under each shooting pose, and each second depth variation degree is used to characterize the depth variation degree of any point on the object to be operated on under each shooting pose.

[0054] For example, the calculation process for image reliability at each shooting pose is as follows: Calculate the matching degree between the left and right infrared images captured at each shooting pose, and determine the first reliability component Rmatch (i.e. the first effective depth information) based on the matching degree.

[0055] The matching degree represents the binocular matching quality between the left and right infrared images within the corresponding region. If the left and right infrared images cannot establish a valid match, or if the matching error exceeds a preset error threshold, the first reliability component of the corresponding depth point is reduced.

[0056] Furthermore, the depth continuity within the target area of ​​the object to be operated on is calculated under each shooting pose, and the second reliability component Rdepth (i.e., the second effective depth information) is determined based on the depth continuity.

[0057] Depth continuity characterizes whether the depth of adjacent locations within the target region changes smoothly. For a current depth point (p) within the target region, multiple depth values ​​within its neighborhood can be obtained. The mean and variance of the neighborhood depths are calculated, and a second reliability component is determined based on the difference between the current depth value and the mean neighborhood depth. When the neighborhood depth changes relatively continuously, the neighborhood depth variance is small, and the second reliability component is high; when the current depth point shows a significant abrupt change relative to its neighborhood, the second reliability component is reduced.

[0058] Furthermore, based on multiple continuously acquired depth frames, the degree of depth variation at the same target location is calculated, and a third reliability component is determined according to the degree of depth variation. Specifically, when the depth variation at the same target location in consecutive depth frames is small, the depth stability at that location is considered high, and the third reliability component is larger; when the depth variation is large, the third reliability component Rstable (i.e., the third effective depth information) is reduced.

[0059] By combining the first reliability component, the second reliability component, and the third reliability component obtained from the above calculations, the reliability of the corresponding depth point in the target area under each shooting pose can be determined: Ri=f(Rmatch,Rdepth,Rstable), which means determining the image reliability corresponding to each shooting pose.

[0060] Understandably, by comprehensively evaluating the effectiveness of binocular matching, local depth continuity, and continuous frame depth stability, it is possible to identify abnormal situations such as binocular matching failure, local abnormal jumps, and unstable depth measurement, thereby more comprehensively assessing the image reliability of each shooting pose and helping to reduce the interference of abnormal data on the subsequent first depth information fusion results.

[0061] Sub-step S1032: Using the image reliability of each shooting pose, weight the multiple first depth information to obtain the fused second depth information of the object to be operated.

[0062] It should be noted that in some cases, when depth data for a certain shooting pose is missing or its reliability is lower than a preset reliability threshold, the depth data for that shooting pose may not be used for the fusion calculation of the target position; in other cases, when only one shooting pose can obtain effective depth data, the depth value corresponding to that shooting pose can be directly used; and in still other cases, when multiple shooting poses can obtain effective depth data, a weighted average can be performed based on the reliability of each depth data.

[0063] It is understandable that by using the image reliability of each shooting pose to weight multiple first depth information, it is possible to differentiate the weighting of first depth information with different image reliability, fully retain the information contribution of first depth information with higher image reliability, and suppress the interference of abnormal first depth information with lower image reliability on the fusion result, thereby effectively improving the overall accuracy and reliability of the final obtained second depth information.

[0064] In one embodiment, the image reliability of each shooting pose is used to weight multiple first depth information to obtain the fused second depth information of the object to be operated on. This includes: calculating the sum of the products of the image reliability of each shooting pose and the first depth information to obtain the weighted information sum value of the object to be operated on; calculating the sum of the image reliability of each shooting pose to obtain the reliability sum value of the object to be operated on; and calculating the ratio of the weighted information sum value to the reliability sum value to obtain the fused second depth information of the object to be operated on.

[0065] For example, the formula for calculating the second depth information by weighting multiple first depth information is as follows:

[0066] in, This is the second depth information. The first depth information obtained for the i-th shooting pose. for Corresponding image reliability.

[0067] Understandably, fusing effective first depth information obtained from different shooting poses can avoid depth loss caused by specular reflection, occlusion, and binocular matching failure under a single observation angle, and achieve accurate recovery of the true depth information of the object to be operated on.

[0068] Step S104: Locate the object to be worked based on the second depth information of the merged object to be worked.

[0069] The positioning of the object to be worked on may include determining one or more of the following: the three-dimensional position, orientation, surface normal direction, and spatial dimensions of the object to be worked on in a unified coordinate system.

[0070] For example, after obtaining the fused second depth information, the second depth information can be converted into a target point cloud corresponding to the object to be worked on, and the target center position of the object to be worked on can be calculated based on the 3D point cloud. Further, the surface normal direction of the object to be worked on can be calculated based on the spatial distribution of each 3D point in the target point cloud. For example, a plane can be fitted to any local neighborhood in the target point cloud, and the normal vector of the fitted plane can be used as the normal direction of the corresponding target surface. Additionally, the spatial dimensions of the object to be worked on can be calculated based on the projection range of the target point cloud along different coordinate axes, and so on.

[0071] Understandably, using the second depth information, which has higher accuracy and better integrity after fusion, to locate the object to be worked on can effectively improve the accuracy and stability of the positioning results and solve problems such as missing depth, excessive positioning error caused by abnormal depth data, and positioning failure under a single observation perspective.

[0072] In one embodiment, the robot's object localization method further includes: based on the localization result of the object to be worked on, outputting the target three-dimensional position and posture information of the object to be worked on, so that the robot can perform precise operations on the object to be worked on.

[0073] The target's three-dimensional position represents the spatial position of the object to be worked on within a predefined unified coordinate system, and may include at least one of the following: target center coordinates, target key point coordinates, and target spatial dimensions. Attitude information represents the direction and orientation of the object to be worked on relative to the unified coordinate system, and may include one or more of the following: target surface normal vector, target attitude angles (roll angle, pitch angle, or yaw angle), and target local coordinate axis directions.

[0074] For example, the target center coordinates can be calculated from the fused 3D point cloud of the object to be worked on, and the target spatial dimensions can be determined based on the spatial distribution of each point in the 3D point cloud. Alternatively, the target surface normal vector can be obtained by performing planar fitting on the target surface point cloud, and the target attitude angle can be determined based on the target surface normal vector and the principal direction of the target point cloud. Further, the target's 3D position and attitude information can be converted to the robot coordinate system or the robot end effector coordinate system and sent to the robot's controller to control the robot end effector to move to the position and attitude corresponding to the object to be worked on.

[0075] Understandably, by outputting positioning results including spatial position and orientation, the robot can not only determine the location of the object to be worked on, but also determine the direction required for the end effector to approach or operate the object, thereby improving the accuracy of the robot's operations such as grasping, pressing, plugging, assembling or inspecting.

[0076] In some embodiments, the robot's object localization method further includes: based on the localization result of the object to be operated, outputting the target three-dimensional position and posture information of the object to be operated, as well as the localization reliability, so that the robot can perform precise operation on the object to be operated.

[0077] Among them, the positioning reliability is used to characterize the accuracy and stability of the target's three-dimensional position and attitude information, and is used by the robot to determine whether to operate on the object to be worked on.

[0078] For example, the localization reliability of the target object can be determined based on one or more of the following: the number of point clouds with effective depth within the target area, the coverage ratio of effective depth points in the target area, the depth consistency among multiple shooting poses, the spatial continuity of the fused point cloud, the computational stability of the target center position, and the stability of the target pose estimation. For instance, a higher localization reliability can be determined when the number of point clouds with effective depth within the target area is large, the coverage ratio of effective depth points is high, the consistency among depth data acquired from multiple shooting poses is high, the spatial continuity of the fused point cloud is good, and the computational results of the target center position and target pose are relatively stable; conversely, a lower localization reliability can be determined.

[0079] Understandably, by outputting the target's 3D position and orientation information, along with the positioning reliability, the robot can determine the operating position and direction of the end effector based on the target's 3D position and orientation information, and judge whether the current positioning result meets the task requirements based on the positioning reliability. When the positioning reliability is higher than a preset reliability threshold, the robot can be controlled to perform the corresponding operation on the target object; when the positioning reliability is lower than the preset reliability threshold, the moving mechanism can be controlled to change the shooting posture of the imaging device and re-acquire image information, or the operation on the target object can be paused.

[0080] This application discloses a robot object localization method, device, robot, and storage medium. By continuously adjusting the shooting posture of the shooting device through a moving mechanism, the shooting device can acquire image information of the object to be worked and the corresponding first depth information from different perspectives. Then, by fusing the first depth information corresponding to each shooting posture representing different perspectives, a second depth information with higher accuracy and completeness corresponding to the object to be worked is obtained. This avoids the problem of missing depth information due to the material characteristics of the object itself or the influence of external environmental factors under a single perspective. As a result, the robot can accurately locate the object to be worked based on the second depth information, thereby greatly improving the positioning accuracy of the robot for the object.

[0081] This application also provides a control device. Please refer to... Figure 5 , Figure 5This is a schematic block diagram of a control device provided in an embodiment of this application.

[0082] like Figure 5 As shown, the control device 200 includes a processor 201, a memory 202 and a network interface connected via a system bus 203. The memory 202 may include a storage medium and internal memory 202. The storage medium may be non-volatile or volatile.

[0083] The storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor 201 to perform any type of robot control.

[0084] The processor 201 provides computing and control capabilities to support the operation of the entire control unit 200.

[0085] The internal memory 202 provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor 201, the processor 201 can perform any kind of robot control.

[0086] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the control device 200 to which the present application is applied. The specific control device 200 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0087] It should be understood that processor 201 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0088] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: The shooting position of the shooting device is continuously adjusted by the active mechanism, and the shooting device is controlled to shoot the object to be worked under different shooting positions. Based on the image information captured by the shooting device under different shooting poses, the first depth information of the object to be worked corresponding to each image information is obtained; Multiple first-depth information data are fused to obtain the fused second-depth information of the object to be worked on; Based on the second depth information of the merged object to be worked on, the object to be worked on is located.

[0089] In one embodiment, when the processor performs fusion processing on multiple first depth information to obtain fused second depth information of the object to be processed, it is used to implement: The image reliability of each shooting pose is obtained. The image reliability is used to characterize the reliability of the image information obtained under the shooting pose. By utilizing the image reliability of each shooting pose, multiple first depth information are weighted and processed to obtain the fused second depth information of the object to be processed.

[0090] In one embodiment, the processor, when implementing image reliability for each shooting pose, is used to: Calculate the matching degree of at least two image information obtained under each shooting pose, and determine the first effective depth information of each image information based on the matching degree corresponding to each shooting pose; Calculate the first depth variation of the object to be worked on under each shooting pose, and determine the second effective depth information of each image information based on the first depth variation corresponding to each shooting pose; wherein, each first depth variation is used to characterize the degree of depth variation of multiple points in any local area on the object to be worked on under each shooting pose. Calculate the second depth variation of the object to be worked on under each shooting pose, and determine the third effective depth information of each image information based on the second depth variation corresponding to each shooting pose; wherein, each second depth variation is used to characterize the degree of depth variation of any point on the object to be worked on under each shooting pose. Based on the first effective depth information, the second effective depth information, and the third effective depth information corresponding to each shooting pose, the image reliability of each shooting pose is determined.

[0091] In one embodiment, when the processor performs weighted processing on multiple first depth information values ​​using the image reliability of each shooting pose to obtain the fused second depth information of the object to be processed, it is used to implement: The sum of the products of the image reliability of each shooting pose and the first depth information is calculated to obtain the weighted information and value of the object to be processed; Calculate the sum of the image reliability for each shooting pose to obtain the reliability and value of the object to be operated on; The ratio of the weighted information sum to the reliability sum is calculated to obtain the second depth information of the object to be worked after fusion.

[0092] In one embodiment, when the processor acquires the first depth information of the object to be worked corresponding to each image information captured by the imaging device at different shooting poses, it is used to implement: Identify the objects to be processed in images captured from different shooting positions; Based on the recognition results of each image information, the image information is segmented to obtain multiple target image information containing the object to be worked on; Based on multiple target image information containing objects to be worked on, obtain the first depth information corresponding to each image information.

[0093] In one embodiment, the target image information includes RGB image information and depth information. When the processor obtains the first depth information corresponding to each of the multiple target image information containing the object to be processed, it is used to: Based on the calibration parameters of the shooting device, the pixel coordinates of each pixel, and the matching depth value, the three-dimensional spatial coordinates of each pixel in each RGB image information are determined. Based on the shooting pose corresponding to each image information and the three-dimensional spatial coordinates of each pixel in each RGB image information, the first depth information corresponding to each image information is determined.

[0094] In one embodiment, the processor is also used to implement: Based on the localization results of the object to be operated, the target three-dimensional position and posture information of the object to be operated are output so that the robot can perform precise operations on the object to be operated.

[0095] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the control device described above can be referred to the corresponding process in the aforementioned robot control embodiments, and will not be repeated here.

[0096] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0097] Please see Figure 6 , Figure 6 This is a schematic block diagram of a robot provided in an embodiment of this application.

[0098] like Figure 6 As shown, the robot 300 includes: Robot body 301; At least one movable mechanism 302 is provided on the robot body, and one of the movable mechanisms carries a camera device 303, which changes its pose as the movable mechanism is adjusted. An end effector 304 is mounted on the robot body and / or moving mechanism for performing target operations; The control device 305 is used to control the moving mechanism 302, the shooting device 303, and the end effector 303. The control device 305 may be the control device 200 described in the preceding embodiments.

[0099] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of each module of the robot described above can be referred to the corresponding processes in the aforementioned robot object positioning method embodiments, and will not be repeated here.

[0100] In some embodiments, the robot 300 may further include a movable chassis (not shown in the figure), which is mounted on the robot body 301, and the robot 300 changes position as the movable chassis moves. It is understood that the movable chassis enables the robot 300 to adapt to different spaces and work locations, effectively expanding the robot 300's work coverage. Simultaneously, the movable chassis can work with the movable mechanism 302 to adjust the shooting posture of the shooting device 303, further enriching the selectivity of the shooting posture, thereby helping the robot 300 obtain more complete and accurate depth information of the object to be worked on, and contributing to further improving the accuracy of the object positioning results.

[0101] It can adapt to different spatial sizes and working positions in different scenarios, expanding the robot's working coverage. At the same time, it can work with the moving mechanism to adjust the shooting position of the shooting device, further enriching the selectivity of shooting poses, providing a basis for obtaining more complete and accurate depth information of the object to be worked on, and helping to further improve the accuracy of the positioning results of the object to be worked on.

[0102] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can be referred to in various embodiments of the robot's object positioning method of this application.

[0103] The computer-readable storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as a hard disk or memory of the computer device. Alternatively, it can be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the computer device.

[0104] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0105] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for locating a robot's work object, characterized in that, The robot includes a camera and a movable mechanism that carries the camera, and the method includes: The shooting position of the shooting device is continuously adjusted by the active mechanism, and the shooting device is controlled to shoot the object to be worked under different shooting positions. Based on the image information captured by the shooting device under different shooting poses, the first depth information of the object to be worked corresponding to each of the image information is obtained; Multiple pieces of the first depth information are fused to obtain the fused second depth information of the object to be worked on. The object to be worked on is located based on the second depth information of the fused object.

2. The robot's object localization method as described in claim 1, characterized in that, The process of fusing multiple pieces of the first depth information to obtain the fused second depth information of the object to be worked on includes: The image reliability of each of the aforementioned shooting poses is obtained, and the image reliability is used to characterize the reliability of the image information obtained under the stated shooting pose. By utilizing the image reliability of each shooting pose, multiple first depth information are weighted and processed to obtain the fused second depth information of the object to be operated on.

3. The robot's object localization method as described in claim 2, characterized in that, The acquisition of image reliability for each of the aforementioned shooting poses includes: Calculate the matching degree of at least two image information obtained under each of the shooting poses, and determine the first effective depth information of each of the image information based on the matching degree corresponding to each of the shooting poses; Calculate the first depth variation of the object to be worked on under each of the shooting poses, and determine the second effective depth information of each of the image information based on the first depth variation corresponding to each of the shooting poses; wherein, each of the first depth variations is used to characterize the degree of depth variation of multiple position points in any local area on the object to be worked on under each of the shooting poses. Calculate the second depth variation of the object to be worked on under each of the shooting poses, and determine the third effective depth information of each of the image information based on the second depth variation corresponding to each of the shooting poses; wherein, each of the second depth variations is used to characterize the degree of depth variation of any position point on the object to be worked on under each of the shooting poses. Based on the first effective depth information, the second effective depth information, and the third effective depth information corresponding to each of the shooting poses, the image reliability of each shooting pose is determined.

4. The robot's object localization method as described in claim 2, characterized in that, The method of weighting multiple first depth information values ​​based on the image reliability of each shooting pose to obtain the fused second depth information of the object to be worked includes: The sum of the products of the image reliability of each shooting pose and the first depth information is calculated to obtain the weighted information and value of the object to be operated on; Calculate the sum of the image reliability of each of the aforementioned shooting poses to obtain the reliability and value of the object to be operated on; The ratio of the weighted information sum to the reliability sum is calculated to obtain the fused second depth information of the object to be worked on.

5. The robot's object localization method as described in claim 1, characterized in that, The step of obtaining the first depth information of the object to be worked corresponding to each image information captured by the shooting device under different shooting poses includes: Identify the objects to be processed in images captured from different shooting positions; Based on the recognition results of each image information, the image information is segmented to obtain multiple target image information containing the object to be worked on; Based on multiple target image information containing the object to be worked, obtain the first depth information corresponding to each of the image information.

6. The robot's object localization method as described in claim 5, characterized in that, The target image information includes RGB image information and depth information; The step of obtaining the first depth information corresponding to each of the multiple target image information containing the object to be worked on includes: Spatial registration processing is performed on each of the RGB image information and the corresponding depth information to determine the depth value that matches each pixel in the RGB image information from the depth information; Based on the calibration parameters of the shooting device, the pixel coordinates of each pixel, and the matching depth value, the three-dimensional spatial coordinates of each pixel in each RGB image information are determined; Based on the shooting pose corresponding to each of the image information and the three-dimensional spatial coordinates of each pixel in each of the RGB image information, the first depth information corresponding to each of the image information is determined.

7. The robot object localization method according to any one of claims 1-6, characterized in that, The method further includes: Based on the positioning results of the object to be worked on, the target three-dimensional position and posture information of the object to be worked on are output so that the robot can perform precise operation on the object to be worked on.

8. A control device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the robot's object localization method as described in any one of claims 1 to 7.

9. A robot, characterized in that, include: The robot itself; At least one movable mechanism is provided on the robot body, and one of the movable mechanisms carries a camera device, the camera device changing its pose as the movable mechanism is adjusted; An end effector, disposed on the robot body and / or the movable mechanism, is used to perform a target operation; The control device as claimed in claim 8 is used to control the active mechanism, the imaging device, and the end effector.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the robot's object localization method as described in any one of claims 1 to 8.

Images