Pose determination method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a pose determination method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-recognized image and a depth image corresponding to the to-be-recognized image, wherein the to-be-recognized image comprises at least one target object; determining a region image corresponding to the first surface of each target object from the to-be-recognized image; for each regional image, determining point cloud data of the regional image based on the regional image and the depth image; determining plane information and boundary information of the first surface based on the point cloud data; and determining pose information of the target object based on the plane information and the boundary information. According to the invention, the reliability and accuracy of pose determination can be improved.

Description

Technical Field

[0001] The present application relates to posture determination technology, and in particular to a posture determination method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] When a robot is carrying objects such as boxes, it needs to use the RGB images and depth images collected by the sensor to estimate the position and posture of the box, so as to help the robot successfully grasp the box and carry it. However, in the process of estimating the position and posture of the box, due to the limitations of the sensor itself, when the camera is too close to the box or the box is exposed to sunlight, there may be a large amount of missing depth information on the box, which will lead to inaccurate or even impossible position and posture estimation of the box. Summary of the invention

[0003] The embodiments of the present application provide a posture determination method, device, computer-readable storage medium, and computer program product, which can improve the reliability and accuracy of posture determination.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for determining a posture, the method comprising:

[0006] Acquire an image to be identified and a depth image corresponding to the image to be identified, wherein the image to be identified includes at least one target object;

[0007] Determining, from the image to be identified, a region image corresponding to the first surface of each of the target objects;

[0008] For each of the regional images, determining point cloud data of the regional image based on the regional image and the depth image;

[0009] Based on the point cloud data, determining plane information and boundary information of the first surface;

[0010] Based on the plane information and the boundary information, position and posture information of the target object is determined.

[0011] The present application provides a posture determination device, including:

[0012] An acquisition module, used to acquire an image to be recognized and a depth image corresponding to the image to be recognized, wherein the image to be recognized includes at least one target object;

[0013] a determination module, for determining, from the image to be identified, a region image corresponding to the first surface of each of the target objects; for each of the region images, determining point cloud data of the region image based on the region image and the depth image; determining plane information and boundary information of the first surface based on the point cloud data; determining the position information of the target object based on the plane information and the boundary information;

[0014] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0015] A memory for storing computer executable instructions or computer programs;

[0016] The processor is used to implement the posture determination method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the posture determination method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the posture determination method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] In an embodiment of the present application, an image to be identified and a depth image corresponding to the image to be identified are obtained, and the image to be identified includes at least one target object. Then, a regional image corresponding to the first surface of each target object is determined from the image to be identified. For each regional image, the point cloud data of the regional image is determined based on the regional image and the depth image, the plane information and boundary information of the first surface are determined based on the point cloud data, and the position and posture information of the target object is determined based on the plane information and the boundary information. Among them, by simultaneously determining the point cloud data of the regional image according to the regional image and the depth image, the obtained point cloud data can be enriched, and more point cloud data can improve the reliability and accuracy of determining the plane information and boundary information of the first surface, thereby improving the reliability and accuracy of determining the position and posture information. Moreover, by determining the plane information and boundary information of the first surface by plane fitting, the spatial information of the first surface can be accurately identified, thereby improving the accuracy of determining the position and posture information of the target object. In addition, since the overall position and posture information is determined only by the first surface of the target object in the present application, the integrity requirements for the image to be identified are low, and the scope of application of the position and posture determination method can be expanded. Therefore, the reliability and accuracy of the position and posture determination can be improved through the embodiment of the present application, and the scope of application of the position and posture determination method can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the architecture of a posture determination system 100 provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0023] Figure 3A It is a flowchart of a method for determining a posture provided in an embodiment of the present application;

[0024] Figure 3B is a schematic diagram of a process for determining point cloud data provided by an embodiment of the present application;

[0025] Figure 3C is a schematic diagram of a process for determining plane information and boundary information provided by an embodiment of the present application;

[0026] Figure 3D is a schematic diagram of a process for determining corner point coordinates provided in an embodiment of the present application;

[0027] Figure 3E is a schematic diagram of a process for determining a second supplementary point cloud provided in an embodiment of the present application;

[0028] Figure 3F It is a schematic diagram of the process of moving a target object provided by an embodiment of the present application;

[0029] Figure 4is a schematic diagram of a carton structure provided in an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of an image to be identified provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) Red, Green, Blue image (RGB image): a color image format consisting of three channels, in which each pixel is composed of three color components, corresponding to red, green and blue respectively. The value range of each component is usually 0 to 255, indicating the intensity of the color.

[0039] 2) Red, Green, Blue Depth Image (RGBD Image): A color image format containing four channels, of which the RGB channels represent the red, green, and blue information in the color image, and the Depth channel represents the depth information corresponding to each pixel, indicating the distance of each pixel from the camera.

[0040] 3) Depth image: An image format that records the distance information from each pixel in the scene to the camera or sensor, usually captured by a depth camera or sensor. The depth image does not provide color information, but instead provides a grayscale image in which the grayscale value of each pixel represents the distance from the pixel to the camera.

[0041] 4) 2D-3D Pose Estimation: The process of inferring the 3D spatial position and orientation of an object from a 2D image or image sequence using computer vision technology. It aims to determine the pose of an object in 3D space, including its position (usually expressed as x, y, z coordinates) and orientation (usually expressed as rotation matrix, Euler angles or quaternion).

[0042] 5) Posture: The position and orientation of an object in three-dimensional space. Position refers to the specific location of an object in three-dimensional space, usually represented by three-dimensional coordinates (x, y, z). Orientation refers to the orientation of the object, which can be described by different mathematical representations such as rotation matrices, Euler angles, and quaternions.

[0043] 6) Instance segmentation mask: A segmentation technique that can accurately segment the outline or region of each independent object in the image.

[0044] In related technologies, in order to enable the robot to move boxes, it is necessary to use a visual algorithm to find the specific pose of the box in the robot coordinate system. Commonly used pose estimation algorithms are divided into pose estimation based on RGB images and pose estimation based on RGBD images.

[0045] Among them, pose estimation based on RGB images relies on a pre-established 3D object model, which requires relatively rich object textures, and estimates the pose of the object by combining feature point matching and 2D-3DPnP methods. However, in the task of moving boxes, since the boxes themselves lack texture information, the solution based on RGB images is not suitable for pose estimation of boxes.

[0046] Pose estimation based on RGBD images relies on the participation of depth information, and pose estimation is performed based on the point cloud matching method. Since the pose estimation algorithm based on RGBD images is very dependent on the integrity of depth information, the point cloud matching method will fail when the depth information is seriously missing. In actual situations, when the robot moves a box, the camera may be close to the box (below the camera's range limit), the working environment may be outdoors (the camera's depth information is lost due to sunlight interference), and the box may be black (serious light absorption leads to loss of depth information). This can easily lead to loss of depth information, resulting in inaccurate pose estimation based on RGBD images or even inability to perform pose estimation.

[0047] Based on the above problems, the embodiments of the present application provide a method, device, equipment, computer-readable storage medium and computer program product for determining posture, which can improve the reliability and accuracy of posture determination. The following describes the exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminals such as intelligent robots (such as logistics handling robots, industrial robots), laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a terminal will be described.

[0048] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the posture determination system 100 provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0049] The terminal 400 may be a target device such as an unmanned vehicle, a drone, a robot, or an aircraft. When the target object 500 needs to be transported, the terminal 400 is used to obtain an image to be identified and a depth image corresponding to the image to be identified by using a camera or sensor set on the head, and then determine the area image corresponding to the first surface of each target object 500 from the image to be identified; for each area image, determine the point cloud data of the area image based on the area image and the depth image; determine the plane information and boundary information of the first surface based on the point cloud data; determine the posture information of the target object 500 based on the plane information and the boundary information, and then control the target device to grab the target object 500 and transport it based on the posture information of the target object 500.

[0050] Alternatively, the terminal 400 is used to obtain the image to be identified and the depth image corresponding to the image to be identified by the head camera or sensor when the target object 500 needs to be transported, and then send the image to be identified and the depth image to the server 200. The server 200 receives the image to be identified and the depth image, and determines the regional image corresponding to the first surface of each target object 500 from the image to be identified; for each regional image, based on the regional image and the depth image, determines the point cloud data of the regional image; based on the point cloud data, determines the plane information and boundary information of the first surface; based on the plane information and the boundary information, determines the posture information of the target object 500. Then, the server 200 controls the target device to grab the target object 500 and transport it based on the posture information of the target object 500, or the server 200 sends the posture information to the terminal 400, and the terminal 400 controls the target device to grab the target object 500 and transport it based on the posture information of the target object 500.

[0051] Take the above-mentioned terminal as an example, for example, the electronic device for determining the posture Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0052] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0053] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0054] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0055] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0056] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0057] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0058] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0059] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0060] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0061] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The position determination device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 4551 and a determination module 4552. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0062] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the posture determination method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0063] The posture determination method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0064] The following describes the posture determination method provided by the embodiment of the present application. As mentioned above, the electronic device implementing the posture determination method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0065] It should be noted that in the example of posture determination below, the target object is a rectangular object. Based on the understanding of the following text, those skilled in the art can apply the posture determination method provided in the embodiment of the present application to determine the posture information of other types of target objects.

[0066] See also Figure 3A , Figure 3A is a flow chart of the method for determining the posture provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0067] In step 101, an image to be identified and a depth image corresponding to the image to be identified are obtained.

[0068] Here, the image to be identified is collected by a camera or a camera, and the image to be identified is usually an RGB image, which contains color information. In addition, the image to be identified includes at least one target object, which is usually an object with a regular shape (such as a cuboid or a cube), such as a box, furniture, book, etc. The depth image corresponds to the image to be identified, and needs to be collected using a camera or sensor with depth perception capability. The depth image includes the depth information corresponding to each pixel in the image to be identified. Among them, in order to ensure that the image to be identified and the depth image can accurately correspond, it is usually necessary to synchronously collect the two images through a camera or sensor and ensure that they are aligned in time. In addition, some basic preprocessing can be performed on the image to be identified, such as adjusting the brightness and contrast, or filtering and denoising, to improve the quality of the image. The depth image may contain noise or inaccurate depth values, and some algorithms can be used to filter the depth image, such as bilateral filtering or median filtering, to remove noise and smooth the depth information.

[0069] In step 102, a region image corresponding to the first surface of each target object is determined from the image to be identified.

[0070] Here, the first surface is usually the upper surface of the target object. Taking the target object as a rectangular box as an example, the first surface is the top surface of the box. The regional image is the image area corresponding to the first surface, that is, the regional image only includes the first surface of one of the target objects. Since a to-be-recognized image may include multiple target objects, and each target object has a corresponding first surface, an image to be recognized may include multiple regional images. Among them, multiple sample images containing target objects can be obtained in advance, and the first surfaces of the target objects in the sample images can be annotated one by one in the form of instance masks, and a training set is constructed by using multiple batches of annotated sample images, and then the model is trained using the training set to obtain a segmentation model for identifying the first surface. In this way, the first surface of each target object in the image to be recognized can be segmented out separately by the segmentation model to obtain a regional image.

[0071] In step 103 , for each regional image, point cloud data of the regional image is determined based on the regional image and the depth image.

[0072] Here, the point cloud data includes the coordinates of the pixels in the area image in three-dimensional space (usually x, y, z), and may also include other information, such as color, intensity, or texture coordinates. By aligning the area image and the depth image to ensure that the two are aligned in space, the depth information can be matched with the pixels of the area image, and then the depth value corresponding to each pixel of the area image in the depth image is converted into point coordinates in three-dimensional space to obtain point cloud data. Specifically, the depth image usually provides a mapping relationship from pixels to three-dimensional space, and this mapping relationship can be directly applied to convert the depth value into point cloud data.

[0073] In some embodiments, see Figure 3B , step 103 can be implemented through steps 1031 to 1034, including:

[0074] In step 1031, the position information of the regional image in the image to be identified is obtained.

[0075] Here, since the regional image is segmented from the image to be recognized, the regional image has a corresponding position in the image to be recognized, and the position of the regional image can be segmented in the image to be recognized in the form of a bounding box as position information. Alternatively, the coordinates of each vertex of the regional image can be used as the position information of the regional image in the image to be recognized.

[0076] In step 1032, depth information of the regional image is acquired from the depth image based on the position information.

[0077] Here, since the image to be identified and the depth image can be aligned, the position of the regional image in the depth image can be determined accordingly through the position information of the regional image in the image to be identified. Then, the depth information corresponding to the regional image in the depth image is used as the depth information of the regional image. Therefore, the depth information includes the depth value of each pixel in the regional image.

[0078] In step 1033, based on each depth value and the camera internal parameters, the coordinate information of each pixel point in the camera coordinate system is determined.

[0079] Here, the camera internal parameters include focal length, principal point coordinates, etc. These parameters describe the geometric characteristics of camera imaging. In the camera coordinate system, the optical center (optical center) of the camera is taken as the origin, the x-axis and y-axis are along the horizontal and vertical directions of the camera sensor respectively, and the z-axis points from the optical center to the front of the camera, that is, the imaging direction. The pixel coordinates of each pixel are converted into physical coordinates in the image coordinate system. Through the mapping relationship between the depth value and the camera internal parameters, the position of the physical coordinates of each pixel in the camera coordinate system is calculated, thereby obtaining the coordinate information of the pixel in the camera coordinate system.

[0080] In step 1034 , point cloud data of the area image is determined based on each coordinate information.

[0081] Here, all the determined coordinate information is combined together as point cloud data of the regional image. The coordinate information can also be combined with other information such as color and texture as point cloud data of the regional image.

[0082] In an embodiment of the present application, the position information of the regional image in the image to be identified is obtained, and based on the position information, the depth information of the regional image is obtained from the depth image, and based on each depth value and the internal parameters of the camera, the coordinate information of each pixel in the camera coordinate system is determined, and based on each coordinate information, the point cloud data of the regional image is determined. In this way, by converting the depth information of the regional image into point cloud data, the position of the first surface of the target object relative to the camera can be intuitively represented, thereby facilitating the determination of the position and posture information of the target object based on the first surface, and improving the reliability and accuracy of the position and posture determination.

[0083] In step 104 , plane information and boundary information of the first surface are determined based on the point cloud data.

[0084] Here, the plane information is used to determine the plane where the first surface is located, and the boundary information is used to determine the edge of the first surface. The plane information can be a plane equation, that is, a mathematical expression of the plane where the first surface is located in the camera coordinate system. The boundary information can be a line segment equation, that is, a mathematical expression of the line segment corresponding to the edge of the first surface in the camera coordinate system.

[0085] In some embodiments, see Figure 3C , step 104 can be implemented through steps 1041 to 1043, including:

[0086] In step 1041, plane fitting is performed based on the point cloud data to obtain plane information for determining the target plane where the first surface is located.

[0087] Here, the target plane refers to the plane where the first surface is located. The point cloud data includes coordinate information of point clouds corresponding to multiple first surfaces, and the coordinate information of the point clouds can be plane-fitted by the least squares method to obtain the normal vector of the first surface and the plane equation of the target plane where the first surface is located.

[0088] In some possible implementations, the normal vector of the first surface can be determined by the coordinate information of the point cloud, and then the plane equation of the plane where the first surface is located can be determined based on the normal vector. Among them, since three non-collinear points can determine a unique plane, at least three non-collinear point clouds can be selected from the point cloud data, and two vectors can be constructed based on the coordinate information of these three point clouds. Then, the normal vector of the plane where all the point clouds are located is obtained by calculating the cross product of the two vectors. Then, any point cloud is selected, and based on the coordinate information and normal vector of the point cloud, the plane equation is constructed through the point normal equation, and the plane equation is used as the plane information of the target plane where the first surface is located.

[0089] In step 1042 , the corner point coordinates of each corner point are determined based on the point cloud data.

[0090] Here, since corner points are usually located at characteristic positions on a plane, such as edge intersections or points where curvature changes significantly, the corner points can be detected in the target plane where the first surface is located through curvature analysis or edge detection algorithms, thereby obtaining the corner point coordinates of the detected corner points.

[0091] In some embodiments, see Figure 3D , step 1042 can be implemented by steps 421 and 422, including:

[0092] In step 421 , a target point cloud is determined from the point cloud data.

[0093] Here, the target point cloud is the point cloud corresponding to the corner point of the first surface. Since the corner point is usually located at a characteristic position on the plane, such as an edge intersection or a point with a significant curvature change, the curvature of each point cloud in the neighborhood can be determined, and the point cloud with a curvature higher than the threshold can be determined as the point cloud corresponding to the corner point.

[0094] In step 422 , when the number of target point clouds is greater than a first preset value, the corner point coordinates of each corner point are determined based on the coordinates of each target point cloud.

[0095] Here, the first preset value is predetermined according to the shape of the first surface of the target object, and the minimum number of corner points required to determine the positions of all corner points of the first surface minus 1 is used as the first preset value. Taking the first surface as a rectangle as an example, the rectangle has four corner points, and the coordinates of all corner points can be determined when the coordinates of at least three corner points are known, so the minimum number of corner points required is 3, so the first preset value is 2.

[0096] Below, the first surface is a rectangle and the first preset value is 2 as an example for explanation: when the number of target point clouds is greater than 2, it means that the coordinates of 3 or 4 corner points are known. When the coordinates of 4 corner points are known, the corner point coordinates of all corner points can be directly obtained according to the coordinates of the target point cloud; when the coordinates of 3 corner points are known, mathematical deduction can be performed according to the coordinates of the 3 corner points to obtain the coordinates of the 4th corner point, thereby obtaining the corner point coordinates of all corner points. For example, three corner points A (1, 1, 0), B (3, 1, 0), and C (3, 3, 0) are known. Since the Z coordinates of A and B are the same, the X coordinates of B and C are the same, and the Y coordinates of C and A are the same, it can be determined that the rectangle is on the XY plane. Therefore, the X coordinate of point D can be the same as that of point A, and the Y coordinate can be the same as that of point C. Therefore, the coordinates of point D are (1, 3, 0).

[0097] In the embodiment of the present application, the target point cloud is determined from the point cloud data, and when the number of the target point cloud is greater than the first preset value, the corner point coordinates of each corner point are determined based on the coordinates of each target point cloud. In this way, the plane information is determined by plane fitting, and when the number of the target point cloud is greater than the first preset value, the corner point coordinates of each corner point are directly determined. The calculation method is simple, which improves the convenience and efficiency of determining the corner point coordinates.

[0098] In actual application scenarios, due to the influence of the camera's field of view, environmental factors, the color of the target object, etc., the point cloud data of the first surface may be incomplete, and the number of target point clouds may be equal to or even less than the first preset value.

[0099] In some embodiments, when the number of target point clouds is equal to a first preset value, the coordinates of the corner points can be determined by the following process: obtaining the size information of the target object; generating a first supplementary point cloud based on the target point cloud and the size information; and determining the coordinates of the corner points by combining the coordinates of the target point cloud and the coordinates of the first supplementary point cloud.

[0100] Here, size information refers to a series of measurement data used to describe the size of an object, usually including the length, width, height (for three-dimensional objects) or diameter, radius (for circular objects) of the object. Taking the target object as a rectangular parallelepiped as an example, the size information is usually expressed in length, width and height, and the units can be centimeters, inches, meters, etc. The first supplementary point cloud refers to the point cloud corresponding to the corner points that were not originally included in the point cloud data when the number of target point clouds is equal to the first preset value. Therefore, the target point cloud and the first supplementary point cloud are combined to include all the corner points of the first surface, so that the coordinates of all corner points can be determined based on the coordinates of the target point cloud and the coordinates of the first supplementary point cloud.

[0101] Continuing with the example that the first surface is a rectangle and the first preset value is 2, when the number of target point clouds is equal to 2, it means that the coordinates of the two corner points are known, and the two corner points must be two corner points located on one edge. When the size information of the target object is known, the length and width of the rectangle of the first surface can be determined according to the size information of the target object. In this way, according to the coordinates of the corner points of the two known rectangles, combined with the length and width of the rectangle, mathematical deduction can also be performed to obtain the corner point coordinates of all corner points.

[0102] In some embodiments, the size information includes the length and width of the first surface, and the first supplementary point cloud can be generated by the following process: determining the distance between the target point clouds; when the difference between the distance and the width is less than a second preset value, determining the coordinates of the first supplementary point cloud based on the coordinates and length of the target point cloud; when the difference between the distance and the length is less than a second preset value, determining the coordinates of the first supplementary point cloud based on the coordinates and width of the target point cloud.

[0103] Here, the distance between the target point clouds refers to the actual distance between the corner points corresponding to the target point clouds. The coordinate distance between the target point clouds can be determined first, and then the actual distance can be determined according to the corresponding relationship between the coordinate distance and the actual distance. As described above, the corner points corresponding to the two target point clouds must be two corner points located on the same edge, so the actual distance between the target point clouds will conform to the length or width of the first surface. Since the actual length and width of the first surface may have errors with the distance between the target point clouds, a second preset value can be set according to the error, such as 0.5 cm, 7 mm, etc. When the difference between the distance and the width is less than the second preset value, it means that the two target point clouds correspond to two corner points located on the wide side of the first surface. At this time, the coordinates of the corresponding first supplementary point cloud can be calculated according to the coordinates and length of the target point cloud. When the difference between the distance and the length is less than the second preset value, it means that the two target point clouds correspond to two corner points located on the long side of the first surface. At this time, the coordinates of the corresponding first supplementary point cloud can be calculated according to the coordinates and width of the target point cloud.

[0104] For example, the length of the first surface is 60 cm, the width is 40 cm, the coordinates of the target point cloud are A(10, 10, 0), B(51, 10, 0), and the second preset value is 2 cm. For the convenience of explanation, take the coordinate distance of the target point cloud and the actual distance as an example. At this time, the distance is 41 cm, and the difference between the distance and the width is less than 2 cm. The target point cloud corresponds to two corner points on the width of the rectangle. Through mathematical deduction, it can be determined that the coordinates of the first supplementary point cloud are C(10, 70, 0), D(51, 70, 0).

[0105] In an embodiment of the present application, when the number of target point clouds is equal to a first preset value, the size information of the target object is obtained, and a first supplementary point cloud is generated based on the target point cloud and the size information, thereby determining the coordinates of the corner points. Wherein, when the size information includes the length and width of the first surface, the distance between the target point clouds is determined; when the difference between the distance and the width is less than a second preset value, the coordinates of the first supplementary point cloud are determined based on the coordinates and length of the target point cloud; when the difference between the distance and the length is less than a second preset value, the coordinates of the first supplementary point cloud are determined based on the coordinates and width of the target point cloud. In this way, by determining the coordinates of the first supplementary point cloud, the missing corner point clouds in the point cloud data are supplemented, thereby enriching more point cloud data and improving the reliability and accuracy of determining the posture information.

[0106] In some embodiments, when the number of target point clouds is less than a first preset value, the corner point coordinates can be determined by the following process: based on the point cloud data, generating a second supplementary point cloud located in the target plane; adding the second supplementary point cloud to the point cloud data to obtain updated point cloud data; based on the target point cloud and the updated point cloud data, determining the corner point coordinates.

[0107] Here, we continue to use the first surface as a rectangle and the first preset value as 2 as an example: when the number of target point clouds is less than 2, it means that only one corner point of the rectangle is known. At this time, the point cloud data is seriously missing, so the point cloud data needs to be supplemented. Among them, the target plane refers to the plane where the first surface is located. The second supplementary point cloud refers to the point cloud corresponding to any position in the first surface that was not originally included in the point cloud data when the number of target point clouds is less than the first preset value. The updated point cloud data includes most of the point clouds of the first surface, as well as the point clouds corresponding to all corner points of the first surface. Therefore, based on the target point cloud and the updated point cloud data, the coordinates of the corner points can be determined.

[0108] In some embodiments, see Figure 3E , a second supplementary point cloud located in the target plane can be generated through steps 201 to 204, including:

[0109] In step 201 , a center point is determined based on the coordinates of each point cloud in the point cloud data.

[0110] Here, the center point refers to the center point of the coordinates of all point clouds. The most densely populated point cloud coordinates can be determined as the center point by a clustering algorithm, or the average value of the coordinates of all point clouds can be calculated as the center point.

[0111] In step 202, a plurality of candidate point clouds located in the target plane are generated based on the center point at preset intervals.

[0112] Here, taking the center point as the starting position, multiple candidate point clouds located in the target plane are uniformly generated in the target plane according to the preset interval along multiple different directions. Among them, the preset interval is used to determine the distance between the candidate point clouds, which can be personalized according to the needs. The smaller the preset interval, the more candidate point clouds are generated. For example, the center point is (1, 1, 0), the preset interval is 1 unit, and the target plane is the XY plane, that is, (z=0). It is necessary to generate candidate point clouds according to 5x5, that is, there are five candidate point clouds in each direction. At this time, based on the x coordinate of the center point, 2 points are generated to the left and right, and each point is separated by 1 unit; similarly, based on the y coordinate of the center point, 2 points are generated upward and downward, and each point is separated by 1 unit; then, the generated points on the x-axis and y-axis are combined to generate 5x5 grid points. In this way, there are 25 points on the grid, each point has a coordinate of 0 on the z-axis, and the distance between each point is 1 unit. These 25 points are candidate point clouds.

[0113] In step 203, for each candidate point cloud, the candidate point cloud is projected into the image to be identified to obtain a projected pixel point.

[0114] Here, the candidate point cloud in the three-dimensional space is projected into the two-dimensional image to be identified through the intrinsic and extrinsic parameters of the camera. Specifically, the three-dimensional coordinates of the candidate point cloud are combined with the intrinsic and extrinsic parameters of the camera, and the projection position of the pixel points corresponding to the candidate point cloud in the image to be identified is obtained through matrix operation to obtain the projected pixel points. Among them, the intrinsic parameters include focal length and principal point coordinates, which describe the properties of the camera itself, and the extrinsic parameters include the rotation and translation of the camera, which describe the position and direction of the camera relative to the scene.

[0115] In step 204, when the projected pixel point is located in the regional image, the candidate point cloud is determined as the second supplementary point cloud.

[0116] Here, if the pixel of the candidate point cloud after projection is within the range of the regional image in the image to be identified, the candidate point cloud is determined as the second supplementary point cloud; otherwise, the candidate point cloud is deleted. Here, the corresponding coordinate range within the range of the regional image can be determined first, and then the coordinate information of each pixel of the candidate point cloud after projection in the image to be identified is determined, and it is determined whether the coordinate information meets the coordinate range. If it meets the coordinate range, it means that the pixel of the candidate point cloud after projection is within the range of the regional image.

[0117] In an embodiment of the present application, when the number of target point clouds is less than a first preset value, a second supplementary point cloud located in the target plane is generated based on the point cloud data, and the second supplementary point cloud is added to the point cloud data to obtain updated point cloud data, thereby determining the coordinates of the corner points. When generating the second candidate point cloud, the center point of the point cloud is first determined, and multiple candidate point clouds are generated and projected at preset intervals based on the center point, and the candidate point cloud whose pixels are located in the regional image after projection is determined as the second supplementary point cloud. In this way, when the point cloud data is severely missing, the point cloud data can be supplemented by the second supplementary point cloud, and the coordinates of the corner points of the first surface can also be determined, thereby improving the reliability of the posture determination method.

[0118] In step 1043, boundary information is determined based on the corner point coordinates.

[0119] Here, the coordinates of the corner points may be directly used as the boundary information, or the line segment equation corresponding to each edge of the first surface may be determined by the coordinates of the corner points, and the line segment equation may be used as the boundary information.

[0120] In the embodiment of the present application, plane fitting is performed based on the point cloud data to obtain the plane information of the target plane where the first surface is located; the corner point coordinates of each corner point are determined based on the point cloud data; and the boundary information is determined based on the corner point coordinates. In this way, the spatial information of the first surface can be determined quickly and accurately, and the position and posture information of the target object can be determined based on the plane information and boundary information of the first surface, thereby improving the efficiency and accuracy of determining the position and posture of the target object.

[0121] In step 105, the position and posture information of the target object is determined based on the plane information and the boundary information.

[0122] Here, the pose information is used to describe the position and orientation of the target object in the environment. Through the plane information and boundary information, the position and orientation of the first surface of the target object in space can be determined. Since the first surface is the upper surface of the target object, the overall pose information of the target object can be determined based on the position and orientation of the first surface in space.

[0123] In some embodiments, the pose information of the target object can be determined by the following process: obtaining the size information of the target object; determining the position information of the second surface of the target object based on the plane information, boundary information and size information; and determining the plane information, boundary information and the position information of the second surface as the pose information.

[0124] Here, the second surface includes every surface of the target object except the first surface. Taking the target object as a cuboid as an example, the first surface is the upper surface of the cuboid, then the second surface includes the front surface, rear surface, bottom surface, left surface and right surface of the cuboid. When the plane information and boundary information of the first surface are obtained, the position information of any other surface can be derived according to the size information and shape characteristics of the target object. Then, the plane information, boundary information of the first surface and the position information of the second surface are determined together as the posture information of the target object. In this way, the information contained in the posture information can be enriched, the accuracy of the posture information can be improved, and other operations such as moving the target object according to the posture information can be facilitated.

[0125] In an embodiment of the present application, an image to be identified and a depth image corresponding to the image to be identified are obtained, and the image to be identified includes at least one target object. Then, a regional image corresponding to the first surface of each target object is determined from the image to be identified. For each regional image, the point cloud data of the regional image is determined based on the regional image and the depth image, the plane information and boundary information of the first surface are determined based on the point cloud data, and the position and posture information of the target object is determined based on the plane information and the boundary information. Among them, by simultaneously determining the point cloud data of the regional image according to the regional image and the depth image, the obtained point cloud data can be enriched, and more point cloud data can improve the reliability and accuracy of determining the plane information and boundary information of the first surface, thereby improving the reliability and accuracy of determining the position and posture information. Moreover, by determining the plane information and boundary information of the first surface by plane fitting, the spatial information of the first surface can be accurately identified, thereby improving the accuracy of determining the position and posture information of the target object. In addition, since the overall position and posture information is determined only by the first surface of the target object in the present application, the integrity requirements for the image to be identified are low, and the scope of application of the position and posture determination method can be expanded. Therefore, the reliability and accuracy of the position and posture determination can be improved through the embodiment of the present application, and the scope of application of the position and posture determination method can be improved.

[0126] In some embodiments, see Figure 3F After determining the position and posture information of the target object, the target object can also be moved through steps 301 to 305, including:

[0127] In step 301, a target grasping position of a target object is determined based on the position information, and joint movement information of a target device grasping the target object is determined based on the target grasping position.

[0128] Here, the target device is an intelligent handling robot, which is used to grasp the target object and move it. The target device includes a joint structure and a grasping structure, the joint structure is used to control the movement of the grasping structure, and the grasping structure is used to grasp the target object. The target grasping position refers to the optimal grasping position of the grasping structure when the target device grasps the target object. The joint movement information is used to control the joint structure of the target device so that the grasping structure of the target device can move to the target grasping position and grasp the target object.

[0129] In step 302, based on the joint movement information, the target device is controlled to grasp the target object.

[0130] Here, the joint structure of the target device is controlled by the joint movement information, so that the grasping structure of the target device can move to the target grasping position and grasp the target object.

[0131] In step 303, the target position of the target object and the current position of the target device are acquired.

[0132] Here, the target position of the target object refers to the target location to which the target object needs to move, and the current location of the target device refers to the location where the target device is currently located. The target position may be predetermined, and the current location may be acquired in real time by a location sensor.

[0133] In step 304, path planning is performed based on the target position and the current position to obtain a moving path of the target device.

[0134] Here, the current position is taken as the starting point and the target position is taken as the end point, and a path from the starting point to the end point is generated through path planning. In the planning, the moving distance should be minimized, unnecessary turns should be avoided, and potential obstacles should be avoided to obtain the moving path.

[0135] In step 305, the target device is controlled to move based on the moving path, and when the target position is reached, the target device is controlled to perform a release action to place the target object at the target position.

[0136] Here, the target device is controlled to move from the current position along the moving path until it reaches the target position. After reaching the target position, the target device performs a release action, and the gripping structure releases the force applied to the target object, so that the target object is naturally placed at the target position, thereby achieving the handling of the target object.

[0137] In an embodiment of the present application, the target grasping position of the target object is determined based on the posture information, and the joint movement information of the electronic device grasping the target object is determined based on the target grasping position; based on the joint movement information, the target device is controlled to grasp the target object; the target position of the target object and the current position of the target device are obtained; the path planning is performed based on the target position and the current position to obtain the moving path of the target device; the target device is controlled to move based on the moving path, and when the target position is reached, the target device is controlled to perform a release action to place the target object at the target position. In this way, by grasping the target object through the posture information, the success rate of the target device grasping the target object can be improved, and damage to the target object can be avoided, thereby improving the degree of automation and success rate of moving the target object.

[0138] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described. The embodiment of the present application can be applied to scenarios where robots carry boxes in indoor and outdoor scenarios.

[0139] When the robot is moving boxes, it needs to use the RGB images and depth images collected by the sensor to determine the position and posture of the box. However, due to the limitations of the sensor itself, when the camera is too close to the box or the box is exposed to sunlight, there may be a large amount of missing depth information on the box, which will lead to inaccurate or even impossible position determination. Based on this, the embodiment of the present application combines the prior knowledge that the robot can know the specific structure and size of the box in advance in the actual application scenario, and proposes a point cloud completion and posture determination algorithm based on the predicted box structure, which uses the only small amount of depth information on the box to complete other missing depth information on the box and determine the position and posture of the box.

[0140] Below, we take the target object as a carton as an example. The structure of the carton is as follows: Figure 4 In order to determine the position of the box, the robot usually installs a fixed camera on its head and lowers its head to a certain extent after approaching the box to ensure that the box is within the imaging range of the camera.

[0141] In actual scenarios, due to the limited field of view of the camera, the top surface (i.e., the first surface) of the box may not be complete within the camera's imaging (i.e., the image to be recognized). Figure 5 As shown, six boxes are included, namely box 51, box 52, box 53, box 54 and box 55, wherein the top surfaces of box 51 and box 55 are incomplete.

[0142] When estimating the position of the box relative to the robot, we first need to find the box in the image to be identified, and then determine the position of the box. Therefore, we need to train a box top segmentation network. In multiple sample images containing boxes, we mark the top surfaces of the boxes one by one in the form of instance masks. We build a training set by marking multiple batches of sample images, and then use this training set to train the model to obtain a box top segmentation model. The box top segmentation model can segment the top surface of each box in the image to be identified separately.

[0143] After obtaining the regional image of the top surface of each box in the image to be identified, the position and posture of each box in the image to be identified can be determined. The following is an example of determining the position and posture information of one box:

[0144] The depth information of the box in the depth image is obtained according to the corresponding regional image, and the point cloud data of the top surface of the box in the camera coordinate system is obtained in combination with the intrinsic parameters of the camera. When the point cloud data of the top surface of the box is relatively complete, the plane equation and boundary of the upper surface of the box can be fitted by rectangular plane fitting (as long as the box exposes at least two corner points, the normal vector and corner point coordinates of the plane in the camera coordinate system can be known). Among them, in the case where only two corner points of the box may be exposed, the coordinates of the remaining two corner points that are not exposed can be inferred by combining the known box size information.

[0145] In the actual process, when the robot moves the box, the camera may be close to the box (below the camera's range limit), the working environment may be outdoors (the camera's depth information is lost due to sunlight interference), the box may be black (severe light absorption leads to loss of depth information), etc., which will cause serious loss of point cloud data on the top surface of the box, and it is impossible to fit the plane equation and boundary of the upper surface of the box by rectangular plane fitting. This embodiment of the application proposes a point cloud data completion method that can effectively complete the missing point cloud data. The following are the specific steps of the point cloud data completion method:

[0146] 1. Based on the existing point cloud, use plane fitting to obtain the normal vector of the top surface of the box in the camera coordinate system and the center point of all point clouds (the difference from rectangular plane fitting is that the center point of all point clouds is not necessarily the center point of the top surface of the box);

[0147] 2. Based on any point cloud and normal vector, the plane where the top surface of the box is located, that is, the target plane, can be determined, and then multiple dense candidate point clouds are generated at a certain preset interval in the target plane;

[0148] 3. Back-project multiple candidate point clouds into the image to be identified using the camera's external and internal parameters. If a point cloud falls within the area of ​​the top surface of the box after back-projection, the candidate point cloud is considered to be one of the point clouds on the top surface of the box, that is, the second supplementary point cloud, and is retained in set A; otherwise, the candidate point cloud is discarded.

[0149] 4. Using the point cloud set A, the plane equation and boundary of the upper surface of the box are fitted by rectangular plane fitting, and then the 6D pose information of the box is obtained by combining the size information of the box.

[0150] In the embodiment of the present application, a method is proposed to obtain the plane equation and boundary of the upper surface of the box by fitting the rectangular plane, and then obtain the 6D pose of the photo paper by combining the known size of the box. By completing the point cloud, the problem that the plane equation and boundary of the upper surface of the box cannot be determined by the rectangular plane fitting method when the point cloud is seriously missing is solved. Therefore, the reliability and accuracy of pose determination can be improved by the embodiment of the present application.

[0151] The following is a description of an exemplary structure of the posture determination device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the posture determination device 455 of the memory 450 may include:

[0152] An acquisition module 4551 is used to acquire an image to be recognized and a depth image corresponding to the image to be recognized, wherein the image to be recognized includes at least one target object;

[0153] Determination module 4552 is used to determine the area image corresponding to the first surface of each target object from the image to be identified; for each area image, determine the point cloud data of the area image based on the area image and the depth image; based on the point cloud data, determine the plane information and boundary information of the first surface; based on the plane information and the boundary information, determine the posture information of the target object.

[0154] In some embodiments, the determination module 4552 is also used to obtain the position information of the area image in the image to be identified; based on the position information, obtain the depth information of the area image from the depth image, and the depth information includes the depth value of each pixel in the area image; based on each of the depth values ​​and the internal parameters of the camera, determine the coordinate information of each of the pixel points in the camera coordinate system; based on each of the coordinate information, determine the point cloud data of the area image.

[0155] In some embodiments, the determination module 4552 is also used to perform plane fitting based on the point cloud data to obtain plane information of the target plane where the first surface is located; based on the point cloud data, determine the corner point coordinates of each corner point; based on the corner point coordinates, determine the boundary information.

[0156] In some embodiments, the determination module 4552 is also used to determine the target point cloud from the point cloud data, and the target point cloud is the point cloud corresponding to the corner points of the first surface; when the number of the target point clouds is greater than a first preset value, the corner point coordinates of each corner point are determined based on the coordinates of each of the target point clouds.

[0157] In some embodiments, the determination module 4552 is also used to obtain the size information of the target object when the number of the target point cloud is equal to the first preset value; generate a first supplementary point cloud based on the target point cloud and the size information; and determine the corner point coordinates by using the coordinates of the target point cloud and the coordinates of the first supplementary point cloud.

[0158] In some embodiments, the determination module 4552 is also used to determine the distance between the target point clouds; when the difference between the distance and the width is less than a second preset value, the coordinates of the first supplementary point cloud are determined based on the coordinates of the target point cloud and the length; when the difference between the distance and the length is less than the second preset value, the coordinates of the first supplementary point cloud are determined based on the coordinates of the target point cloud and the width.

[0159] In some embodiments, the determination module 4552 is also used to generate a second supplementary point cloud located in the target plane based on the point cloud data when the number of the target point cloud is less than the first preset value; add the second supplementary point cloud to the point cloud data to obtain updated point cloud data; and determine the corner point coordinates based on the target point cloud and the updated point cloud data.

[0160] In some embodiments, the determination module 4552 is also used to determine the center point based on the coordinates of each point cloud in the point cloud data; based on the center point, generate multiple candidate point clouds located in the target plane at preset intervals; for each candidate point cloud, project the candidate point cloud into the image to be identified to obtain a projected pixel point; when the projected pixel point is located in the regional image, determine the candidate point cloud as the second supplementary point cloud.

[0161] In some embodiments, the determination module 4552 is also used to obtain size information of the target object; determine the position information of a second surface of the target object based on the plane information, the boundary information and the size information, the second surface including each surface of the target object except the first surface; and determine the plane information, the boundary information and the position information of the second surface as the posture information.

[0162] In some embodiments, the determination module 4552 is also used to determine the target grasping position of the target object based on the posture information, and determine the joint movement information of the electronic device grasping the target object based on the target grasping position; based on the joint movement information, control the target device to grasp the target object; obtain the target position of the target object and the current position of the target device; perform path planning based on the target position and the current position to obtain the moving path of the target device; control the target device to move based on the moving path, and when reaching the target position, control the target device to perform a release action to place the target object at the target position.

[0163] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the above-mentioned posture determination method of the embodiment of the present application.

[0164] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the posture determination method provided in the present application embodiment, for example, Figure 3A The pose determination method shown.

[0165] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0166] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0167] As an example, computer-executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored in a portion of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0168] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0169] In summary, an image to be identified and a depth image corresponding to the image to be identified are obtained through the embodiment of the present application, and the image to be identified includes at least one target object. Then, the regional image corresponding to the first surface of each target object is determined from the image to be identified. For each regional image, the point cloud data of the regional image is determined based on the regional image and the depth image, the plane information and boundary information of the first surface are determined based on the point cloud data, and the pose information of the target object is determined based on the plane information and the boundary information. Among them, by simultaneously determining the point cloud data of the regional image based on the regional image and the depth image, the obtained point cloud data can be enriched, and more point cloud data can improve the reliability and accuracy of determining the plane information and boundary information of the first surface, thereby improving the reliability and accuracy of determining the pose information. Moreover, by determining the plane information and boundary information of the first surface by plane fitting, the spatial information of the first surface can be accurately identified, thereby improving the accuracy of determining the pose information of the target object. In addition, since the overall pose information is determined only by the first surface of the target object in the present application, the integrity requirements for the image to be identified are low, and the scope of application of the pose determination method can be expanded. Therefore, the embodiments of the present application can improve the reliability and accuracy of posture determination, and increase the scope of application of the posture determination method.

[0170] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Acquire an image to be identified and a depth image corresponding to the image to be identified, wherein the image to be identified includes at least one target object; Determining, from the image to be identified, a region image corresponding to the first surface of each of the target objects; For each of the regional images, determining point cloud data of the regional image based on the regional image and the depth image; Based on the point cloud data, determining plane information and boundary information of the first surface; Based on the plane information and the boundary information, position and posture information of the target object is determined.

2. The method according to claim 1, characterized in that The step of determining point cloud data of the regional image based on the regional image and the depth image includes: Obtaining position information of the regional image in the image to be identified; Based on the position information, acquiring depth information of the regional image from the depth image, the depth information including a depth value of each pixel in the regional image; Based on each of the depth values ​​and the camera internal parameters, determining the coordinate information of each of the pixel points in the camera coordinate system; Based on each of the coordinate information, point cloud data of the area image is determined.

3. The method according to claim 1, characterized in that The determining, based on the point cloud data, plane information and boundary information of the first surface includes: Performing plane fitting based on the point cloud data to obtain plane information of a target plane where the first surface is located; Based on the point cloud data, determining the corner point coordinates of each corner point; Based on the corner point coordinates, the boundary information is determined.

4. The method according to claim 3, characterized in that The step of determining the corner point coordinates of each corner point based on the point cloud data includes: Determine a target point cloud from the point cloud data, the target point cloud being a point cloud corresponding to a corner point of the first surface; When the number of the target point clouds is greater than a first preset value, the corner point coordinates of each corner point are determined based on the coordinates of each of the target point clouds.

5. The method according to claim 4, characterized in that The method further comprises: When the number of the target point clouds is equal to the first preset value, acquiring size information of the target object; Based on the target point cloud and the size information, generating a first supplementary point cloud; The coordinates of the target point cloud and the coordinates of the first supplementary point cloud are used to determine the coordinates of the corner point.

6. The method according to claim 5, characterized in that The size information includes the length and width of the first surface, and generating a first supplementary point cloud based on the target point cloud and the size information includes: Determining the distance between the target point clouds; When the difference between the distance and the width is less than a second preset value, determining the coordinates of the first supplementary point cloud based on the coordinates of the target point cloud and the length; When the difference between the distance and the length is less than the second preset value, the coordinates of the first supplementary point cloud are determined based on the coordinates of the target point cloud and the width.

7. The method according to claim 4, characterized in that The method further comprises: When the number of the target point cloud is less than the first preset value, generating a second supplementary point cloud located in the target plane based on the point cloud data; adding the second supplementary point cloud to the point cloud data to obtain updated point cloud data; The corner point coordinates are determined based on the target point cloud and the updated point cloud data.

8. The method according to claim 7, characterized in that The step of generating a second supplementary point cloud located in the target plane based on the point cloud data comprises: Determine a center point based on the coordinates of each point cloud in the point cloud data; Based on the center point, generating a plurality of candidate point clouds located in the target plane at preset intervals; For each candidate point cloud, project the candidate point cloud onto the image to be identified to obtain a projected pixel point; When the projected pixel point is located in the regional image, the candidate point cloud is determined as a second supplementary point cloud.

9. The method according to any one of claims 1 to 8, characterized in that: The determining the position and posture information of the target object based on the plane information and the boundary information includes: Obtaining size information of the target object; Determine, based on the plane information, the boundary information, and the size information, position information of a second surface of the target object, the second surface including each surface of the target object except the first surface; The plane information, the boundary information and the position information of the second surface are determined as the posture information.

10. The method according to any one of claims 1 to 8, characterized in that: After determining the position and posture information of the target object, the method further includes: Determine a target grasping position of the target object based on the position information, and determine joint movement information of a target device grasping the target object based on the target grasping position; Based on the joint movement information, controlling the target device to grasp the target object; Acquire a target position of the target object and a current position of the target device; Perform path planning based on the target position and the current position to obtain a moving path of the target device; The target device is controlled to move based on the moving path, and when the target position is reached, the target device is controlled to perform a release action to place the target object at the target position.

11. A posture determination device, characterized in that: The device comprises: An acquisition module, used to acquire an image to be recognized and a depth image corresponding to the image to be recognized, wherein the image to be recognized includes at least one target object; A determination module is used to determine, from the image to be identified, the area image corresponding to the first surface of each of the target objects; for each of the area images, based on the area image and the depth image, determine the point cloud data of the area image; based on the point cloud data, determine the plane information and boundary information of the first surface; based on the plane information and the boundary information, determine the posture information of the target object.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 10 when executing computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images