Target extraction method, system, robot and storage medium

Abstract

The present application discloses a target extraction method, system, robot and storage medium, the method comprising: obtaining image data of a target to be extracted in a target scene; the image data comprises a depth image; performing feature extraction on the image data to obtain image feature information, the image feature information comprises a target mask; according to the target mask, converting the depth image into a point cloud space; linearizing the pixel edge of the target mask to obtain a linearized contour; obtaining reference posture parameters according to the linearized contour; and extracting the target to be extracted in the point cloud space according to the reference posture parameters. The technical solution provided by the present application not only improves the accuracy of target extraction, but also simplifies the process and improves the extraction efficiency.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a target extraction method, system, robot and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robotic grasping technology has been widely used. In the industrial field, it can help workers complete heavy picking and placing tasks, such as material sorting, loading and unloading, and smart retail; in the service industry, it can help the disabled or the elderly complete daily grasping tasks.

[0003] At present, robots mainly use simplified manipulators and simplified grasping models to perform grasping actions. The existing simplified manipulators are generally two-finger hands, and the two-finger hand grasping posture acquisition includes two technical solutions: indirect grasping posture acquisition and direct grasping posture acquisition.

[0004] Indirect acquisition of grasping posture: rigid transformation from the object coordinate system to the camera coordinate system, including 3D rotation matrix and 3D translation matrix. After determining the pose of the object coordinate system in the camera coordinate system, the distribution of the object in the camera space is calculated, the graspable position is calculated, and finally the 6D pose of the two-finger hand is set for grasping. This scheme can only obtain accurate 6D grasping posture when the object has a 3D model and the object is rigid, but it cannot accurately obtain the grasping posture for some non-rigid objects.

[0005] Directly obtain grasping posture: manually observe two-dimensional images, annotate data in the two-dimensional images, and then use these data to train the robot for deep learning, and directly obtain the 6D grasping posture of the two-fingered hand from RGBD or RGB. Since the robot grasping behavior faces a complex environment with multiple chaotically stacked objects, annotating data for such images is an extremely difficult and complex task, and the work efficiency is low. Summary of the invention

[0006] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a target extraction method, system, robot and storage medium, which not only do not require complex target posture annotation and simplify the process of obtaining target posture, but also can effectively improve the accuracy of extracting the target to be extracted and improve the efficiency of extracting the target to be extracted.

[0007] In a first aspect of the present application, a target extraction method is provided, comprising: acquiring image data of a target to be extracted in a target scene; the image data comprising a depth image; performing feature extraction on the image data to obtain image feature information, the image feature information comprising a target mask; converting the depth image into a point cloud space according to the target mask; linearizing the pixel edges of the target mask to obtain a linearized contour; acquiring reference posture parameters according to the linearized contour, the reference posture parameters being partial parameters for a robot to extract the target to be extracted; and extracting the target to be extracted in the point cloud space according to the reference posture parameters.

[0008] According to the target extraction method of the first aspect of the present application, there are at least the following beneficial effects: by linearizing the pixel edges of the target mask, a linearized contour can be obtained, and the linearized contour can simplify the target to be extracted, and accurate target posture parameters can be obtained from the linearized contour, thereby improving the accuracy of extracting the target to be extracted. In addition, the entire process of obtaining the target posture does not require complex posture annotation, which simplifies the process of obtaining the target posture and improves the extraction efficiency.

[0009] According to some embodiments of the present application, the reference posture parameters include the grasping center point, the grasping direction and the initial grasping width. According to the reference posture parameters, the target to be extracted is extracted in the point cloud space, including: according to the reference posture parameters, the mapping relationship between the reference posture parameters and the point cloud space is obtained; according to the mapping relationship, the target posture parameters are obtained, the target posture parameters include the reference posture parameters, the grasping width and the grasping depth, and the target posture parameters are all parameters for the robot to extract the target to be extracted; the target posture parameters are transmitted to the execution end of the robot, and the execution end is controlled to grasp the target to be extracted. By converting the depth image into the point cloud space, the three-dimensional space position of the target to be extracted can be obtained. By forming a mapping relationship between the reference posture parameters and the point cloud space, the target posture parameters can be obtained, and then the target posture parameter information is transmitted to the execution end of the robot, so that the execution end can accurately grasp the target to be extracted.

[0010] According to some embodiments of the present application, the image data also includes an RGB picture or a grayscale picture, and feature extraction is performed on the image data to obtain image feature information, and the image feature information includes a target mask, including: extracting features from the image data to obtain a first feature map; wherein the first feature map includes a first feature and a second feature, the first feature is a feature of the RGB picture or the grayscale picture, and the second feature is a feature of the depth picture; the first feature of the first feature map is processed using a local attention mechanism to obtain a second feature map; the second feature of the first feature map is processed using a local attention mechanism to obtain a third feature map; the second feature map and the third feature map are fused to obtain a fourth feature map; according to the fourth feature map, the object category and the grasping confidence in the target scene are obtained; according to the object category, the target mask is obtained; according to the grasping confidence and the target mask, a graspable mask of the target to be extracted is obtained. The first feature map is obtained by extracting features from the image data, and the image data includes an RGB picture or a grayscale picture and a depth picture, so the first feature map includes the first feature corresponding to the RGB picture or the grayscale picture, and the second feature corresponding to the depth picture. By using the local attention mechanism to process the first feature and the second feature of the first feature map, the second feature map and the third feature map can be obtained respectively, and then the second feature map and the third feature map are fused to obtain the fourth feature map. The fourth feature map contains the object categories in the target scene, the confidence of each category of objects, the target mask and the grasping confidence of the target, and then the graspable mask can be screened out according to the grasping confidence. The local attention mechanism can better perceive the environment around the grasping target, provide predictions for the grasping of the two-finger hand, set the prediction of the grasping confidence, and finally effectively improve the accuracy of grasping the target.

[0011] According to some embodiments of the present application, a first feature of a first feature map is processed using a local attention mechanism to obtain a second feature map, including: obtaining a first overall feature map of a target scene, and obtaining a first sub-feature map set on the first overall feature map with a first step length in the height direction and a second step length in the width direction; obtaining a one-dimensional local channel attention feature map and a two-dimensional local space attention feature map of each first sub-feature map in the first sub-feature map set; converting each first sub-feature map according to a preset first coefficient and a second coefficient to obtain a plurality of second sub-feature maps, wherein the first coefficient is a coefficient of a one-dimensional local channel attention feature map of the first sub-feature map, and the second coefficient is a coefficient of a two-dimensional local space attention feature map of the first sub-feature map; and splicing the plurality of second sub-feature maps using a splicing function to obtain a second feature map. By dividing the first overall feature map along height and width, a plurality of sub-feature maps can be obtained, and then the plurality of sub-feature maps can be spliced ​​by a splicing function to obtain a second feature map. The local attention mechanism can filter the features of the first overall feature map, remove unnecessary features, and thereby obtain new features. This makes it possible to adapt to complex stacking scenarios and improve the accuracy of grabbing targets.

[0012] According to some embodiments of the present application, the pixel edge of the target mask is linearized to obtain a linearized contour, including: obtaining a first matrix of a grabbable mask according to the target mask; obtaining a second matrix composed of edge pixel coordinates of the grabbable mask according to the gradient change of the first matrix; sorting the coordinates of the second matrix clockwise or counterclockwise to obtain a third matrix; taking the first coordinate in the third matrix as the starting point, obtaining a first coordinate set in the third matrix whose distance from the first coordinate is greater than or equal to a preset width; taking the second coordinate in the first coordinate set as the starting point, obtaining a second coordinate set in the first coordinate set whose distance from the second coordinate is greater than or equal to a preset width; if the distance between any coordinates in the second coordinate set is greater than or equal to the preset width, the second coordinate set is the coordinate of the linearized contour of the target to be extracted. The pixel edge coordinates of the target mask are sorted to obtain the third matrix. The linearized contours on both sides of the first coordinate can be obtained by retaining the other edge pixel coordinates in the third matrix whose distance from the first coordinate is greater than or equal to the preset width, and removing the pixel coordinates whose distance from the first coordinate is less than the preset width. Similarly, the linearized contours of other edge pixel coordinates can be obtained. This simplifies the model of the target to be extracted and improves the capture efficiency.

[0013] According to some embodiments of the present application, the method further includes: if there are at least two coordinates in the second coordinate set whose distance is less than the preset width, then one of the two coordinates whose distance is less than the preset width is removed, and the coordinate of the other point is used as the second coordinate, and the step of executing the second coordinate in the coordinate set as the starting point is used to obtain the second coordinate set in the first coordinate set whose distance from the second coordinate is greater than or equal to the preset width. Removing the pixel coordinates whose distance from the first coordinate is less than the preset width can obtain the linearized contours on both sides of the first coordinate. Similarly, the linearized contours of other edge pixel coordinates can be obtained. This simplifies the model of the target to be extracted and improves the capture efficiency.

[0014] In a second aspect of the present application, a target extraction system is provided, comprising: at least one memory, at least one processor and at least one program instruction, wherein the program instruction is stored in the memory and can be executed on the processor, and the processor is used for the target extraction method of the first aspect of the present application.

[0015] A third aspect of the present application provides a robot comprising the target extraction system of the second aspect.

[0016] According to some embodiments of the present application, it also includes: an image acquisition module, the image acquisition module is used to acquire an RGB image or a grayscale image and a depth image of a target scene; a mask segmentation module, the mask segmentation module is used to segment the mask of the target scene into a target mask; a grasping module, including a two-fingered hand, the two-fingered hand is used to grasp the target to be extracted; an analysis control module, connected to the image acquisition module, the mask segmentation module and the grasping module, the analysis control module is used to: analyze the RGB image or the grayscale image and the depth image of the target scene acquired by the image acquisition module to obtain mask information of the target scene; pass the mask information of the target scene to the mask segmentation module, and receive the target mask information transmitted by the mask segmentation module; analyze the target mask information to obtain target posture parameter information; and control the two-fingered hand to extract the target to be extracted according to the target posture parameter information.

[0017] According to the robot of the third aspect embodiment of the present application, at least the following beneficial effects are achieved: through the image acquisition module, an RGB image or grayscale image and a depth image of the target scene can be obtained, the mask segmentation module can segment the mask of the target scene into a target mask, and then the target to be extracted is extracted using a two-fingered hand, the analysis and control module can analyze the image data acquired by the image acquisition module, and obtain mask information based on the image data, by passing the mask information of the target scene to the mask segmentation module, the mask segmentation module segments the target mask from the mask of the target scene, the mask segmentation module transmits the target mask information to the analysis and control module, the analysis and control module analyzes the target mask information to obtain target posture parameter information, and controls the two-fingered hand to extract the target to be extracted based on the target posture parameter information. It can not only adapt to scenes with simple environments, but also to complex stacking scenes, and in the process of model training, there is no need to perform complex target posture annotation, which simplifies the process and effectively improves the efficiency of extracting the target to be extracted.

[0018] The fourth aspect of the present application provides a storage medium having program instructions stored thereon, the program instructions being used to execute the target extraction method of the first aspect of the present application.

[0019] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0021] Figure 1 A flow chart of a target extraction method according to an embodiment of the present application;

[0022] Figure 2 A logic diagram of the mask segmentation part of the target extraction method of the embodiment of the present application;

[0023] Figure 3 A logical diagram of target extraction of a target extraction method according to an embodiment of the present application;

[0024] Figure 4 Schematic diagram of the first feature graph of the target extraction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0026] In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0027] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0028] In the description of this application, unless otherwise clearly defined, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above words in this application in combination with the specific content of the technical solution. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of this application.

[0029] Reference Figure 1 In a first aspect of the present application, a target extraction method is provided, comprising:

[0030] S100, acquiring image data of a target to be extracted in a target scene; the image data includes a depth picture;

[0031] S200, extracting features from the image data to obtain image feature information, where the image feature information includes a target mask;

[0032] S300, converts the depth image into point cloud space according to the target mask;

[0033] S400, linearizing the pixel edge of the target mask to obtain a linearized contour;

[0034] S500, obtaining reference posture parameters according to the linearized contour, where the reference posture parameters are partial parameters for the robot to extract the target to be extracted;

[0035] S600: extracting a target to be extracted in the point cloud space according to the reference posture parameters.

[0036] Reference Figure 2Specifically, the target scene can be photographed by a depth camera to capture an RGB image or a grayscale image of the target scene, as well as a depth image corresponding to the RGB image or the grayscale image; the mask segmentation module can segment the target mask in the target scene from the RGB image or the grayscale image and the corresponding depth image, and can give the category of the object corresponding to each mask in the target scene, the confidence of each object and the grasping confidence, and then filter out the graspable masks according to the grasping confidence. The grasping module converts the corresponding depth image into point cloud space according to the mask of the target to be extracted, and linearizes the pixel edge coordinates of the target mask to obtain a linearized contour. According to the linearized contour, reference posture parameters such as the grasping center point, grasping direction and initial opening distance can be obtained. These grasping posture parameters are mapped to the corresponding point cloud space, and the grasping width and grasping depth of the robot execution end, i.e. the two-finger hand, are set to generate an initial grasping box. Then, the grasping box is rotated according to the local point cloud normal vector of the grasping center point, and it steps along the axis direction. Collision detection is performed with the corresponding point cloud space, and the appropriate opening distance is calculated to obtain all the grasping posture parameters.

[0037] By linearizing the pixel edges of the target mask, a linearized contour can be obtained, and the linearized contour can simplify the target to be extracted. The linearized contour can obtain accurate target posture parameters, thereby improving the accuracy of extracting the target to be extracted. In addition, the entire process of obtaining the target posture does not require complex posture annotation, which simplifies the process of obtaining the target posture and improves the extraction efficiency.

[0038] Reference Figure 3 In some embodiments of the present application, the reference posture parameters include the grasping center point, the grasping direction and the initial grasping width. According to the reference posture parameters, the target to be extracted is extracted in the point cloud space, including: according to the reference posture parameters, obtaining the mapping relationship between the reference posture parameters and the point cloud space; according to the mapping relationship, obtaining the target posture parameters of the two-fingered hand, the target posture parameters include the reference posture parameters, the grasping width and the grasping depth, and the target posture parameters are all the parameters for the robot to extract the target to be extracted; transmitting the target posture parameters to the two-fingered hand, and controlling the two-fingered hand to grasp the target to be extracted. By converting the depth image into the point cloud space, the three-dimensional spatial position of the target to be extracted can be obtained. By forming a mapping relationship between the reference posture parameters and the point cloud space, the target posture parameters can be obtained, and then the target posture parameter information is transmitted to the two-fingered hand, so that the two-fingered hand can accurately grasp the target to be extracted.

[0039] Specifically, the grasping center point, grasping direction and initial opening distance can be formed into a correspondence with the point cloud space. According to the correspondence, the grasping width and grasping depth of the two-fingered hand are set to generate an initial grasping box. Then, the grasping box is rotated according to the local point cloud normal vector of the grasping center point, so that the grasping box steps closer to the grasping center point along the normal vector direction. The grasping box and the point cloud space are subjected to collision detection to obtain a suitable grasping width and grasping depth, thereby obtaining the target posture parameters. The analysis and control module controls the two-fingered hand to grasp the target to be extracted according to the target posture parameters.

[0040] Reference Figure 3 In some embodiments of the present application, the image data also includes an RGB picture or a grayscale picture, and feature extraction is performed on the image data to obtain image feature information, and the image feature information includes a target mask, including: extracting features from the image data to obtain a first feature map; wherein the first feature map includes a first feature and a second feature, the first feature is a feature of the RGB picture or the grayscale picture, and the second feature is a feature of the depth picture; the first feature of the first feature map is processed using a local attention mechanism to obtain a second feature map; the second feature of the first feature map is processed using a local attention mechanism to obtain a third feature map; the second feature map and the third feature map are fused to obtain a fourth feature map; according to the fourth feature map, the object category and the grasping confidence in the target scene are obtained; according to the object category, the target mask is obtained; according to the grasping confidence and the target mask, the graspable mask of the target to be extracted is obtained. By extracting features from the image data, the first feature map is obtained, and the image data includes an RGB picture or a grayscale picture and a depth picture, so the first feature map includes the first feature corresponding to the RGB picture or the grayscale picture, and the second feature corresponding to the depth picture. By using the local attention mechanism to process the first feature and the second feature of the first feature map, the second feature map and the third feature map can be obtained respectively, and then the second feature map and the third feature map are fused to obtain the fourth feature map. The fourth feature map contains the object categories in the target scene, the confidence of each category of objects, the target mask and the grasping confidence of the target, and then the graspable mask can be screened out according to the grasping confidence. The local attention mechanism can better perceive the environment around the grasping target, provide predictions for the grasping of the two-finger hand, set the prediction of the grasping confidence, and finally effectively improve the accuracy of grasping the target.

[0041] Specifically, the features of the RGB image or grayscale image and the features of the depth image contained in the first feature map are processed using the local attention mechanism to obtain a second feature map including the features of the RGB image or grayscale image, and a third feature map including the features of the depth image.

[0042] In some embodiments of the present application, a first feature of a first feature map is processed using a local attention mechanism to obtain a second feature map, including: obtaining a first overall feature map of the target scene, and obtaining a first sub-feature map set on the first overall feature map with a first step length in the height direction and a second step length in the width direction; obtaining a one-dimensional local channel attention feature map and a two-dimensional local space attention feature map of each first sub-feature map in the first sub-feature map set; converting each first sub-feature map according to a preset first coefficient and a second coefficient to obtain a plurality of second sub-feature maps, wherein the first coefficient is a coefficient of a one-dimensional local channel attention feature map of the first sub-feature map, and the second coefficient is a coefficient of a two-dimensional local space attention feature map of the first sub-feature map; and splicing the plurality of second sub-feature maps using a splicing function to obtain a second feature map. By dividing the first overall feature map along height and width, a plurality of sub-feature maps can be obtained, and then the plurality of sub-feature maps can be spliced ​​by a splicing function to obtain a second feature map. The local attention mechanism can filter the features of the first overall feature map, remove unnecessary features, and thus obtain new features. This makes it possible to adapt to complex stacking scenarios and improve the accuracy of grabbing targets.

[0043] Reference Figure 4 , the processing of the first feature of the first feature map is: the first feature map is The one-dimensional local channel attention feature is And the two-dimensional local spatial attention feature On the feature map F, the first feature map is divided into different sub-feature maps along the height and width directions with m and n as the step length. The local channel attention feature map on the sub-feature map is Local spatial attention feature map are the results of max and pooling operations on the channels of the sub-feature map, respectively. Then a 3×3 convolution operation is performed to restore the number of channels. σ is the sigmoid function.

[0044] M' C =σ(Conv(AvgPool(F');MaxPool(F')))

[0045] =σ(Conv(P avg ;P max ))

[0046] In addition, in each feature map F of each sub-feature map F' s 'Perform max and pooling until Then an MLP convolution operation is performed on it.

[0047] M'S =σ(MLP(AvgPool(F' S ))+MLP(MaxPool(F' S )))

[0048] =σ(MLP(AvgPool(P'avg)+MaxPool(P'max)))

[0049] Set the first coefficient γ for the local channel attention feature map and the spatial attention feature map respectively S ,γ C is the second coefficient, where the first coefficient and the second coefficient are obtained by experiment,

[0050]

[0051] where F' n is the new sub-feature map, Represents the multiplication between feature maps.

[0052] The most useful method is to concatenate all the sub-feature graphs F' n The second feature map F after the local attention mechanism is obtained by splicing n ,

[0053] F n =concatnate(F' n ),

[0054] Similarly, the second feature of the first feature map is processed using the above-mentioned local attention mechanism to obtain the third feature map.

[0055] In some embodiments of the present application, straightening the pixel edges of the target mask to obtain a straightened contour includes: obtaining a first matrix of the graspable mask according to the target mask; obtaining a second matrix composed of the edge pixel coordinates of the graspable mask according to the gradient change of the first matrix; sorting the coordinates of the second matrix clockwise or counterclockwise to obtain a third matrix; taking the first coordinate in the third matrix as the starting point, and obtaining a first coordinate set in the third matrix where the distance from the first coordinate is greater than or equal to a preset width; taking the second coordinate in the first coordinate set as the starting point, and obtaining a second coordinate set in the first coordinate set where the distance from the second coordinate is greater than or equal to a preset width; if the distance between any coordinates in the second coordinate set is greater than or equal to the preset width, then the second coordinate set is the coordinates of the straightened contour of the target to be extracted. By sorting the pixel edge coordinates of the target mask, a third matrix is obtained. By retaining the other edge pixel coordinates in the third matrix whose distance from the first coordinate is greater than or equal to the preset width and removing the pixel coordinates whose distance from the first coordinate is less than the preset width, the straightened contours on both sides of the first coordinate can be obtained. Similarly, the straightened contours of other edge pixel coordinates can be obtained. Thereby simplifying the model of the target to be extracted and improving the grasping efficiency.

[0056] In some embodiments of the present application, it further includes: if there are at least two coordinates in the second coordinate set whose distance is less than the preset width, then remove one of the two coordinates with a spacing less than the preset width, and use the coordinate of the other point as the second coordinate, and execute the step of taking the second coordinate in the coordinate set as the starting point to obtain a second coordinate set in the first coordinate set where the distance from the second coordinate is greater than or equal to the preset width. By removing the pixel coordinates whose distance from the first coordinate is less than the preset width, the straightened contours on both sides of the first coordinate can be obtained. Similarly, the straightened contours of other edge pixel coordinates can be obtained. Thereby simplifying the model of the target to be extracted and improving the grasping efficiency.

[0057] Specifically, assume that the known graspable mask matrix is M; according to the gradient change of the grasped mask matrix M, a matrix N composed of the mask edge pixel coordinates can be obtained. Sort the coordinates in the matrix N clockwise to obtain N'. Suppose the width of the fingers of the two-finger hand is about d pixels. Starting from any point in N', calculate the distance d' point by point:

[0058] If d'<d, then discard this point;

[0059] If d'≥d, then retain this point. Finally, the edge matrix N” after the straightened contour is obtained, and the straightened contour is completed至此完成直线化轮廓。

[0060] Take out any two adjacent points in the matrix N" to form a line segment AB, and get the line CD along the direction of the common perpendicular line of the straight line AB and according to the distribution of the object. So far, the four points ABCD are obtained, and their coordinates are (a0, a1), (b0, b1), (c0, c1), (d0, d1), so as to obtain the coordinates of the grasping point O, which is the center point of the four points ABCD, and the corresponding grasping width AD or BC, and the grasping horizontal angle θ, which is the slope of AD or BC; the original depth map matrix is ​​D; the internal parameter matrix of the depth camera is I, and the depth map matrix D of the graspable target can be obtained. o =M×D; the point cloud matrix of the graspable target P=D o ×I.

[0061] Map the obtained grasping point coordinates O and the grasping horizontal angle θ to the point cloud space of the grasped object, and calculate the normal vector n corresponding to the grasping point O in the point cloud space, and set the real two-finger hand grasping width D and grasping depth H, and set a smaller grasping width D', so as to form the initial grasping rectangular bounding box in the point cloud space, gradually increase the grasping width L', and obtain the appropriate grasping width L according to the collision detection. So far, all the grasping parameters are obtained, grasping point O, grasping direction theta, grasping depth H, grasping width L, and finger width D.

[0062] In a second aspect of the present application, a target extraction system is provided, comprising: at least one memory, at least one processor and at least one program instruction, wherein the program instruction is stored in the memory and can be executed on the processor, and the processor is used for the target extraction method of the first aspect of the present application.

[0063] A third aspect of the present application provides a robot comprising the target extraction system of the second aspect.

[0064] In some embodiments of the present application, it also includes: an image acquisition module, the image acquisition module is used to acquire RGB images or grayscale images and depth images of the target scene; a mask segmentation module, the mask segmentation module is used to segment the mask of the target scene into a target mask; a grasping module, including a two-fingered hand, the two-fingered hand is used to grasp the target to be extracted; an analysis control module, connected to the image acquisition module, the mask segmentation module and the grasping module, the analysis control module is used to: analyze the RGB images or grayscale images and depth images of the target scene acquired by the image acquisition module to obtain mask information of the target scene; pass the mask information of the target scene to the mask segmentation module, and receive the target mask information transmitted by the mask segmentation module; analyze the target mask information to obtain target posture parameter information; and control the two-fingered hand to extract the target to be extracted according to the target posture parameter information.

[0065] Through the image acquisition module, the RGB image or grayscale image and depth image of the target scene can be obtained, the mask segmentation module can segment the mask of the target scene into the target mask, and then use the two-fingered hand to extract the target to be extracted. The analysis and control module can analyze the image data collected by the image acquisition module, and obtain mask information based on the image data. By passing the mask information of the target scene to the mask segmentation module, the mask segmentation module segments the target mask from the mask of the target scene, and the mask segmentation module transmits the target mask information to the analysis and control module, which analyzes the target mask information to obtain the target posture parameter information, and controls the two-fingered hand to extract the target to be extracted according to the target posture parameter information. It can not only adapt to scenes with simple environments, but also to complex stacking scenes. In addition, in the process of model training, there is no need to perform complex target posture annotation, which simplifies the process and effectively improves the efficiency of extracting the target to be extracted.

[0066] Specifically, the image acquisition module may be a depth camera. The grasping module may be set as a two-finger hand.

[0067] The fourth aspect of the present application provides a storage medium having program instructions stored thereon, the program instructions being used to execute the target extraction method of the first aspect of the present application.

[0068] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present application.

[0069] Reference below Figure 1 and Figure 4 , a target extraction method according to an embodiment of the present application is described in detail with a specific embodiment. It is worth noting that the following description is only an exemplary description and is not a specific limitation of the present application.

[0070] Reference Figures 1 to 4 , the present application embodiment provides a target extraction method, comprising:

[0071] Obtain an RGB image or grayscale image of the target scene and the corresponding depth image through a depth camera;

[0072] Perform feature extraction on the RGB image or the grayscale image and the corresponding depth image to obtain a first feature map, where the first feature map includes a first feature and a second feature, where the first feature corresponds to a feature of the RGB image or the grayscale image, and the second feature corresponds to a feature of the corresponding depth image;

[0073] The first feature and the second feature in the first feature map are processed respectively by using the local attention mechanism to obtain the second feature map and the third feature map;

[0074] The second feature map and the third feature map are subjected to feature fusion to obtain a fourth feature map;

[0075] According to the fourth feature map obtained after fusion, the object category and grasping confidence in the target scene can be obtained;

[0076] According to the object category and grasping confidence in the target scene, the target mask and the graspable mask of the target to be extracted can be obtained;

[0077] According to the target mask, the depth image can be converted into point cloud space;

[0078] The pixel edges of the target mask and the grabbable mask are linearized to obtain a linearized contour;

[0079] Reference posture parameters such as grasping center point, grasping direction and initial grasping width can be obtained from the linearized contour;

[0080] By mapping the reference posture parameters to the point cloud space, we can obtain the reference posture parameters and target posture parameters such as grasping width and grasping depth;

[0081] According to the target posture parameters, the two-fingered hand can be precisely controlled to extract the target to be extracted.

[0082] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] Program instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate form. Storage media include: any entity or device capable of carrying computer program codes, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electric carrier signals and telecommunication signals.

[0084] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0085] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0086] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present application.

Claims

1. A target extraction method, characterized in that: The following steps are involved: Acquire image data of a target to be extracted in a target scene; the image data includes a depth picture; Performing feature extraction on the image data to obtain image feature information, the image feature information includes a target mask, and the image data also includes an RGB image or a grayscale image; wherein the performing feature extraction on the image data to obtain image feature information includes: performing feature extraction on the image data to obtain a first feature map; wherein the first feature map includes a first feature and a second feature, the first feature is a feature of the RGB image or the grayscale image, and the second feature is a feature of the depth image; using a local attention mechanism to process the first feature of the first feature map to obtain a second feature map; using a local attention mechanism to process the second feature of the first feature map to obtain a third feature map; fusing the second feature map and the third feature map to obtain a fourth feature map; according to the fourth feature map, obtaining an object category and a grasping confidence in the target scene; according to the object category, obtaining a target mask; according to the grasping confidence and the target mask, obtaining a graspable mask of the target to be extracted; According to the target mask, converting the depth image into a point cloud space; Linearizing the pixel edges of the target mask to obtain a linearized contour; Acquire reference posture parameters according to the linearized contour, wherein the reference posture parameters are partial parameters for the robot to extract the target to be extracted; The target to be extracted is extracted in the point cloud space according to the reference posture parameters.

2. The target extraction method according to claim 1, characterized in that: The reference posture parameters include a grasping center point, a grasping direction, and an initial grasping width. The point cloud space extracts the target to be extracted according to the reference posture parameters, including: According to the reference posture parameters, obtaining a mapping relationship between the reference posture parameters and the point cloud space; According to the mapping relationship, the target posture parameters are acquired, wherein the target posture parameters include the reference posture parameters, the grasping width and the grasping depth, and the target posture parameters are all parameters for the robot to extract the target to be extracted; The target posture parameters are transmitted to the execution end of the robot, and the execution end is controlled to extract the target to be extracted.

3. The target extraction method according to claim 1, characterized in that: The using a local attention mechanism to process the first feature of the first feature map to obtain a second feature map includes: Acquire a first overall feature map of the target scene, and obtain a first sub-feature map set on the first overall feature map by a first step length in a height direction and a second step length in a width direction; Respectively obtain a one-dimensional local channel attention feature map and a two-dimensional local spatial attention feature map of each first sub-feature map in the first sub-feature map set; According to a preset first coefficient and a second coefficient, each of the first sub-feature maps is converted to obtain a plurality of second sub-feature maps, wherein the first coefficient is a coefficient of a one-dimensional local channel attention feature map of the first sub-feature map, and the second coefficient is a coefficient of a two-dimensional local spatial attention feature map of the first sub-feature map; The plurality of the second sub-feature maps are concatenated using a concatenation function to obtain a second feature map.

4. The target extraction method according to claim 1, characterized in that: The linearizing the pixel edge of the target mask to obtain a linearized contour includes: According to the target mask, a first matrix of a graspable mask is obtained; According to the gradient change of the first matrix, a second matrix consisting of edge pixel coordinates of the grabbable mask is obtained; Sort the coordinates of the second matrix clockwise or counterclockwise to obtain a third matrix; Taking the first coordinate in the third matrix as a starting point, obtaining a first coordinate set in the third matrix whose distance from the first coordinate is greater than or equal to a preset width; Taking the second coordinate in the first coordinate set as a starting point, obtaining a second coordinate set in the first coordinate set whose distance from the second coordinate is greater than or equal to the preset width; If the distances between any coordinates in the second coordinate set are greater than or equal to the preset width, the second coordinate set is the coordinates of the linearized contour of the object to be extracted.

5. The target extraction method according to claim 4, characterized in that: Also includes: If there are at least two coordinates in the second coordinate set whose distance is less than the preset width, remove one of the two coordinates whose distance is less than the preset width, and take the coordinate of the other point as the second coordinate, and execute the step of taking the second coordinate in the coordinate set as the starting point to obtain a second coordinate set in the first coordinate set whose distance from the second coordinate is greater than or equal to the preset width.

6. A target extraction system, characterized in that: include: At least one memory, at least one processor and at least one program instruction, wherein the program instruction is stored in the memory and can be run on the processor, and the processor is used to execute the target extraction method according to any one of claims 1 to 5.

7. A robot, characterized in that: Includes the target extraction system as described in claim 6.

8. The robot according to claim 7, characterized in that: Also includes: An image acquisition module, which is used to acquire RGB images or grayscale images and depth images of the target scene; A mask segmentation module, the mask segmentation module is used to segment the mask of the target scene into the target mask; A grasping module, comprising a two-finger hand, wherein the two-finger hand is used to grasp the object to be extracted; An analysis control module is connected to the image acquisition module, the mask segmentation module and the capture module, and is used for: Analyze the RGB picture or grayscale picture and depth picture of the target scene collected by the image acquisition module to obtain mask information of the target scene; Passing the mask information of the target scene to the mask segmentation module, and receiving the target mask information transmitted by the mask segmentation module; Analyzing the target mask information to obtain target posture parameter information; According to the target posture parameter information, the two-fingered hand is controlled to extract the target to be extracted.

9. A storage medium, characterized in that: The storage medium stores program instructions, and the program instructions are used to execute the target extraction method according to any one of claims 1 to 5.

Images