Semantic segmentation method, robot grabbing pose estimation method based on semantic segmentation, computer program product and terminal

Abstract

The invention discloses a semantic segmentation method, a robot grabbing pose estimation method based on semantic segmentation, a computer program product and a terminal, and belongs to the field of semantic segmentation, and the method comprises the steps: constructing a semantic segmentation network, and enabling a decoder of the semantic segmentation network to comprise a lightweight feature fusion module which is used for carrying out multi-scale conversion processing on low-level features and high-level features, performing cross aggregation processing on different scale features, performing heterogeneous convolution processing, and finally performing feature fusion processing; training the semantic segmentation network; and inputting the to-be-segmented image into the trained semantic segmentation network to obtain a semantic segmentation result. Through the lightweight feature fusion module, the problem of feature fusion with inconsistent semantics and scales can be effectively solved, feature fusion and information interaction are better realized, pixel loss is reduced, and the quality of fused features is remarkably improved, so that the precision of semantic segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation, and in particular to a semantic segmentation method, a robot grasping posture estimation method based on semantic segmentation, a computer program product and a terminal. Background Art

[0002] DeepLabv3+ is an advanced semantic segmentation model that introduces a decoder module based on DeepLabv3 to improve the segmentation accuracy of object boundaries. The model adopts an encoder-decoder architecture, which mainly consists of two parts: encoder and decoder. The encoder part is responsible for extracting high-level semantic information of the image, while the decoder part is responsible for restoring the spatial information of the image to obtain accurate segmentation results. In the encoder part, the input passes through the backbone network to obtain two outputs: one is a low-level feature and the other is a high-level feature. Although the features output by the Deeplabv3+ network on the semantic path contain rich global contextual semantic information, due to the 1 / 4 ratio difference in the size of low-level features and high-level features, these features cannot be directly combined with the Attentional Feature Fusion (AFF) network applicable to the same size and number of channels. If AFF is directly applied to semantic features with a large multiple, pixel loss may occur, which in turn affects the quality of the fused features and reduces the accuracy of semantic segmentation. Summary of the invention

[0003] The purpose of the present invention is to overcome the problems of the prior art and to provide a semantic segmentation method, a robot grasping pose estimation method based on semantic segmentation, a computer program product and a terminal.

[0004] The object of the present invention is to achieve the following technical solution: a semantic segmentation method, the method comprising the following steps:

[0005] Construct a semantic segmentation network, including an encoder and a decoder. The encoder includes a backbone network and a feature pyramid pooling module connected in sequence. The backbone network is used to extract low-level features of the input image. The feature pyramid pooling module extracts high-level features of the input image based on the feature map output by the backbone network.

[0006] The decoder includes a lightweight feature fusion module and an image spatial information recovery module connected in sequence; the lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule; the multi-scale feature aggregation submodule includes a multi-scale conversion unit, a cross-scale information interaction unit and a same-size feature generation unit; the multi-scale conversion unit is used to perform multi-scale conversion processing on low-level features and high-level features; the cross-scale information interaction unit is used to perform cross-aggregation processing on features of different scales to achieve cross-scale information interaction; the same-size feature generation unit is used to perform heterogeneous convolution processing to generate features of the same size; the attention feature fusion submodule is used to perform feature fusion processing on features of the same size; the image spatial information recovery module is used to perform convolution processing and up-sampling processing on the fused features obtained by the fusion processing, and output a semantic segmentation map of the input image;

[0007] Train the semantic segmentation network;

[0008] The image to be segmented is input into the trained semantic segmentation network to obtain the semantic segmentation result.

[0009] In one example, performing multi-scale conversion processing on low-level features and high-level features includes:

[0010] The low-level features are sequentially upsampled and convolved, and the high-level features are convolved twice to complete the multi-scale conversion process.

[0011] In one example, cross-aggregation processing is performed on features of different scales, including:

[0012] The low-level features, the low-level features that have completed the multi-scale conversion process, and the high-level features are subjected to dot multiplication processing to obtain the first cross-aggregation feature; and the high-level features, the low-level features that have completed the multi-scale conversion process, and the high-level features are subjected to dot multiplication processing to obtain the second cross-aggregation feature.

[0013] In one example, performing heterogeneous convolution processing includes:

[0014] A heterogeneous convolution is performed on the first cross-aggregation feature, a heterogeneous convolution is performed on the second cross-aggregation feature, and elements are added to the features of the same size obtained by the two heterogeneous convolutions.

[0015] In one example, the backbone network outputs first low-level features and second low-level features of different sizes, and the semantic segmentation network compresses the first low-level features and then inputs them into the feature pyramid pooling module;

[0016] The first low-level feature and the first high-level feature output by the feature pyramid pooling module are subjected to multi-scale feature fusion processing by a lightweight feature fusion module to obtain a second high-level feature;

[0017] The second high-level features and the second low-level features are sequentially subjected to multi-scale feature fusion processing, convolution processing, and upsampling processing by a lightweight feature fusion module, and a semantic segmentation map of the input image is output.

[0018] It should be further explained that the technical features corresponding to the above-mentioned examples of semantic segmentation methods can be combined or replaced with each other to form a new technical solution.

[0019] The present invention also includes a robot grasping posture estimation method based on semantic segmentation, comprising the following steps:

[0020] Using any one of the above examples or a combination of multiple examples to form a semantic segmentation method to perform semantic segmentation processing on the grasping operation image, extract the contour of the target to be grasped, and then calculate a grasping box located on the contour of the target to be grasped;

[0021] The pixel coordinates of the center point of the grab box are converted into coordinates in three-dimensional space using camera calibration;

[0022] Calculate the coordinate value of the center point of the grab frame in the robot base coordinate system based on hand-eye calibration;

[0023] Based on the inverse kinematics algorithm, the angle values ​​of each servo of the robot's manipulator arm when grasping the target to be grasped are solved, thereby realizing the grasping posture control of the manipulator arm.

[0024] In one example, the calculation expression of the coordinate value of the center point of the grabbing frame in the robot base coordinate system is:

[0025]

[0026] in, represents the transformation matrix from the camera coordinate system to the robot base coordinate system; R, T are the rotation matrix and translation vector from the camera coordinate system to the robot base coordinate system calculated by hand-eye calibration; X arm ,Y arm ,Z arm They respectively represent the horizontal coordinate, vertical coordinate and depth coordinate of the center of the grab frame in the robot base coordinate system.

[0027] In one example, the robotic arm is a three-section robotic arm, and the angle calculation expression of the three servos when grasping the target to be grasped is:

[0028] x=l1cosθ1+l2cos(θ1+θ2)+l3cos(θ1+θ2+θ3)

[0029] y=l1sinθ1+l2sin(θ1+θ2)+l3sin(θ1+θ2+θ3)

[0030] α=θ1+θ2+θ3

[0031] Among them, x, y are the horizontal and vertical coordinates of the end points of the robotic arm respectively; θ1, θ2, θ3 are the angle values ​​of the three servos when grasping the target to be grasped; l1, l2, l3 are the lengths of the three sections of the robotic arm respectively; α represents the angle between the robotic arm and the horizontal plane.

[0032] It should be further explained that the technical features corresponding to the above examples can be combined or replaced with each other to form a new technical solution.

[0033] The present invention also includes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the semantic segmentation method formed by any one of the above examples or a combination of multiple examples.

[0034] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor runs the computer instructions, the steps of the robot grasping posture estimation method formed by any one or more of the above examples are executed.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. In one example, a lightweight feature fusion module is used to perform multi-scale conversion, cross-aggregation, and heterogeneous convolution on low-level features and high-level features to generate features of the same size. Feature fusion processing is performed on this basis, which can effectively solve the problem of feature fusion with inconsistent semantics and scales, better realize feature fusion and information interaction, reduce pixel loss, and significantly improve the quality of fused features, thereby improving the accuracy of semantic segmentation.

[0037] 2. In one example, the language segmentation network is applied to the robot grasping posture estimation scenario, which can ensure the segmentation accuracy of the grasping operation image, accurately identify the target contour from a complex background, and then accurately extract the contour of the target to be grasped in different postures and angles, determine the center point of the grasping frame, and then based on coordinate transformation and robotic arm grasping posture control, achieve efficient and stable grasping of targets in different postures and angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0039] Figure 1 A method flow chart provided for an example of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a lightweight feature fusion module provided for an example of the present invention;

[0041] Figure 3 A schematic diagram of the DeepLabv3+ network structure provided for an example of the present invention;

[0042] Figure 4 A simplified model diagram of a robotic arm provided as an example of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In one example, if Figure 1 As shown, a semantic segmentation method comprises the following steps:

[0048] S10: Build a semantic segmentation network.

[0049] The semantic segmentation network in this example is the DeepLabv3+ network. Based on the existing DeepLabv3+ network integrated with the attention feature fusion module (Attentional Feature Fusion, AFF), the AFF structure is further optimized, and a lightweight feature fusion module (LightAttentional Feature Fusion, LAFF) is proposed, including a multi-scale feature aggregation submodule and an attention feature fusion submodule (Attention feature fusion module). That is, by introducing a lightweight feature fusion module, the challenges brought by scale changes and small targets can be effectively addressed, thereby improving the accuracy of semantic segmentation.

[0050] Specifically, the semantic segmentation network includes an encoder and a decoder. The encoder includes a backbone network and a feature pyramid pooling module (Atrous Spatial Pyramid Pooling, ASPP) connected in sequence. The backbone network is used to extract low-level features of the input image. The backbone network can be a deep convolutional neural network (such as ResNet). In this example, MobileNetV2 is preferably used to meet the requirements of lightweight data processing. The feature pyramid pooling module extracts high-level features of the input image based on the feature map (low-level features) output by the backbone network.

[0051] The decoder includes a lightweight feature fusion module and an image spatial information recovery module connected in sequence; the lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule; the multi-scale feature aggregation submodule includes a multi-scale conversion unit, a cross-scale information interaction unit and a same-size feature generation unit; the multi-scale conversion unit is used to perform multi-scale conversion processing on low-level features and high-level features; the cross-scale information interaction unit is used to perform cross-aggregation processing on features of different scales to achieve cross-scale information interaction; the same-size feature generation unit is used to perform heterogeneous convolution processing to generate features of the same size; the attention feature fusion submodule is used to perform feature fusion processing on features of the same size; the image spatial information recovery module is used to perform convolution processing and up-sampling processing on the fused features obtained by the fusion processing, and output a semantic segmentation map of the input image;

[0052] S20: Train the semantic segmentation network.

[0053] First, prepare and preprocess the dataset for training, including images and their corresponding annotations; then, based on the constructed DeepLabv3+ network, select a suitable loss function such as cross entropy loss, and optimizer such as SGD or Adam, and set the learning rate and training cycle; then, iteratively train the model on the training set, evaluate the performance on the validation set, and adjust the hyperparameters to optimize the model; finally, use the test set to evaluate the trained model to determine its generalization ability. During the entire training process, data enhancement, learning rate adjustment, and other operations may also be required to further improve the performance and robustness of the model.

[0054] S30: Input the image to be segmented into the trained semantic segmentation network to obtain a semantic segmentation result, that is, the contour of the foreground object can be obtained.

[0055] In one example, if Figure 2 As shown, the multi-scale conversion unit includes a low-level feature conversion subunit and a high-level feature conversion subunit. The low-level feature conversion subunit includes an upsampling layer (Upsample) and a first convolutional layer (Conv1) connected in sequence; the high-level feature conversion subunit includes a second convolutional layer (Conv2) and a third convolutional layer (Conv3) connected in sequence.

[0056] In the LAFF structure, suppose there are two input features X∈R of different sizes W×H×C and Y∈R W / 2×H / 2×C / 2 , where Y is the high-level semantic feature generated by the feature pyramid pooling module, and X is the low-level feature obtained by the backbone network through convolution. The multi-scale conversion processing of low-level features and high-level features includes:

[0057] The low-level feature X is upsampled by a factor of 2 using bilinear interpolation, and then a 3×3 convolution with a step size of 2 is performed; the high-level feature Y is subjected to a 3×3 convolution with a step size of 2 and a 5×5 convolution with a step size of 2, thereby completing multi-scale conversion in order to perform cross aggregation.

[0058] In one example, if Figure 2 As shown, the cross-scale information interaction unit includes a first dot product layer and a second dot product layer. The first dot product layer is connected to the first convolution layer, the third convolution layer, and the backbone network. The second dot product layer is connected to the first convolution layer, the third convolution layer, and the feature pyramid pooling module.

[0059] In the LAFF structure, suppose there are two input features X∈R of different sizes W×H×C and Y∈R W / 2×H / 2×C / 2 , where Y is the high-level semantic feature generated by the feature pyramid pooling module, and X is the low-level feature obtained by the backbone network through convolution. The cross-aggregation processing of features of different scales includes:

[0060] The low-level feature X, the low-level feature that has completed the multi-scale conversion process, and the high-level feature are subjected to dot multiplication processing to obtain a first cross-aggregation feature; and the high-level feature Y, the low-level feature that has completed the multi-scale conversion process, and the high-level feature are subjected to dot multiplication processing to obtain a second cross-aggregation feature.

[0061] This example implements cross-aggregation of features at different scales through pixel-by-pixel multiplication operations, so that the features of X and Y can be effectively fused at different resolutions and cross-scale information interaction can be achieved.

[0062] In one example, if Figure 2 As shown in the figure, the feature generation unit of the same size uses heterogeneous convolution with smaller parameters to further strengthen the feature learning of the two branches, including the first heterogeneous convolution layer (HetConv1), the second heterogeneous convolution layer (HetConv2) and the first element addition layer. The first heterogeneous convolution layer is connected to the first point product layer, the second heterogeneous convolution layer is connected to the second point product layer, and the output ends of the first heterogeneous convolution layer and the second heterogeneous convolution layer are both connected to the first element addition layer.

[0063] Perform heterogeneous convolution processing, including:

[0064] Perform heterogeneous convolution on the first cross-aggregation feature and heterogeneous convolution on the second cross-aggregation feature to generate features u∈R of the same size W×H×C and v∈R W×H×C , and then u∈R W×H×C and v∈R W×H×C Perform element addition processing to obtain preliminary fusion features.

[0065] The corresponding examples of the above multi-scale conversion unit, cross-scale information interaction unit, and same-size feature generation unit are combined. The calculation expression of the entire data processing process is as follows:

[0066]

[0067] in, Indicates dot product processing; Dw 3×3 represents 3×3 convolution; Dw 5×5 Indicates 5×5 convolution; Up indicates upsampling processing; Represents element addition processing; M represents mask function; Z represents preliminary fusion features.

[0068] In this example, LAFF, as a lightweight feature fusion module, can effectively solve the problem of feature fusion with inconsistent semantics and scales. Based on the original structure, LAFF can realize the exchange and integration of multi-scale features of different sizes, retaining more detailed information in low-level features while obtaining rich contextual high-level semantic information. This design can significantly improve the quality of fused features in a variety of network scenarios, effectively cope with the challenges brought by scale changes and small objects, and thus improve the accuracy of semantic segmentation.

[0069] In one example, the attention feature fusion submodule includes a local attention branch and a global attention branch, both of which are connected to the first element adding layer; the local attention branch includes a fourth convolution layer (Point-wiseConv4), a first activation function layer (ReLU1) and a fifth convolution layer (Point-wise Conv5) connected in sequence; the global attention branch includes a global average pooling layer (GAP), a sixth convolution layer (Point-wise Conv6), a second activation function layer (ReLU2) and a seventh convolution layer (Point-wise Conv7) connected in sequence, the output ends of the fifth convolution layer and the seventh convolution layer are connected to the second element adding layer, the second element adding layer is connected to the third activation function layer (Sigmoid), the third activation function layer is connected to the third point multiplication layer and the fourth point multiplication layer respectively, and the first heterogeneous convolution layer is connected to the third point multiplication layer, the second heterogeneous convolution layer is connected to the fourth point multiplication layer, and the output ends of the third point multiplication layer and the fourth point multiplication layer are both connected to the third element adding layer.

[0070] The local attention branch uses 1x1 convolution and ReLU activation function to calculate local attention and capture local features and details. The global attention branch calculates global attention through global average pooling, 1x1 convolution, ReLU activation function and other operations to capture the global context information of the image, add the local and global attention, and then generate weights through the sigmoid function. The input features are multiplied by the weights obtained by the sigmoid function to complete the weighted fusion processing of the features, thereby paying more attention to the key parts of the image.

[0071] Preferably, if Figure 3 As shown, this example decoder includes two lightweight feature fusion modules, namely a first lightweight feature fusion module (LAFF1) and a second lightweight feature fusion module (LAFF2), where the output end of the backbone network and the output end of the feature pyramid pooling module are both connected to the first lightweight feature fusion module, the output end of the backbone network and the output end of the first lightweight feature fusion module are both connected to the second lightweight feature fusion module, the second lightweight feature fusion module is connected to the image spatial information recovery module, and the image spatial information recovery module includes a convolution layer and an upsampling layer.

[0072] Furthermore, the backbone network is MobileNetV2, which is used to output feature maps (low-level features) of different sizes, including first low-level features L1 (shallow features of 1 / 8 the size of the original input image) and second low-level features L2 (shallow features of 1 / 2 the size of the original input image) of different sizes. The semantic segmentation network compresses the first low-level feature L1 and inputs it into the feature pyramid pooling module. The feature pyramid pooling module processes the feature map in parallel by using dilated convolutions and global average pooling with different expansion rates, thereby extracting features at different scales, and generating the first high-level feature after the features of different scales are processed by 3×3 convolution. The first low-level feature L1 and the first high-level feature L2 are subjected to multi-scale feature fusion processing by the first lightweight feature fusion module to obtain the second high-level feature H2; the second high-level feature H2 and the second low-level feature L2 are subjected to multi-scale feature fusion processing by the second lightweight feature fusion module in turn, and then convolution processing and up-sampling processing are performed by the image spatial information recovery module, thereby obtaining a prediction map (Prediction) with the same resolution as the input image, that is, the semantic segmentation map of the input image.

[0073] The present invention also includes a robot grasping posture estimation method based on semantic segmentation, comprising the following steps:

[0074] S100: Using any one of the above examples or a combination of multiple examples to form a semantic segmentation method to perform semantic segmentation processing on the input grasping operation image, extract the contour of the target to be grasped, and then calculate a grasping box positioned on the contour of the target to be grasped.

[0075] The grasping operation image is a two-dimensional image including the target to be grasped and the complex background. The grasping target can be any object, such as a pen, an earphone box, etc. This application uses Deeplabv3 + The semantic segmentation network extracts the precise contours of objects. DeepLabv3 + By introducing the dilated convolution and feature pyramid pooling modules, it is possible to capture multi-scale features and accurately segment the boundaries of objects, especially to separate the contours of objects to be grasped in complex backgrounds. Using the semantic segmentation results and the geometric information of the set grasping box, the optimal grasping box is screened out, and the center coordinates and the rotation angle of the grasping box are calculated. This process is designed to ensure that the grasping point corresponding to the selected grasping box is located on the object, not on the background or occlusion.

[0076] Furthermore, in order to improve the adaptability to objects of different sizes, the present invention designs a lightweight feature fusion module, through which feature maps at different levels can share similar semantic information, ensuring that the capture box can be accurately positioned on the object regardless of the size of the object.

[0077] S200: Convert the pixel coordinates of the center point of the capture frame into coordinates in three-dimensional space by using camera calibration.

[0078] The robot needs to know the exact position of the target to be grasped in three-dimensional space in order to navigate and grasp it. Through coordinate transformation, the robot can accurately locate the object and perform the task.

[0079] S300: Calculate the coordinate value of the center point of the grasping frame in the robot base coordinate system based on the hand-eye calibration.

[0080] Through hand-eye calibration, the pixel coordinates in the two-dimensional image can be converted into three-dimensional space coordinates in the robot base coordinate system, thereby achieving accurate positioning of the target to be grasped.

[0081] S400: Based on the inverse kinematics algorithm, the angle values ​​of the servos of the robot's manipulator arm when grasping the target to be grasped are solved, thereby realizing the grasping posture control of the manipulator arm.

[0082] The inverse kinematics algorithm can calculate the precise angles that each joint of the robot needs to reach based on the position of the target to be grasped, so that the robotic arm can move accurately to the predetermined position for grasping; at the same time, the inverse kinematics algorithm can improve the operating accuracy of the robot, and the control allows the robot to grasp or operate the target to be grasped more accurately.

[0083] In one example, the three-dimensional coordinate value of the pixel coordinate of the center point of the grabbing frame in the camera coordinate system is calculated according to the camera calibration principle. Assume that the pixel coordinate of the center point of the grabbing frame is (u, v), and the coordinate of the center point of the grabbing frame in the camera coordinate system is (X cam ,Y cam ,Z cam ), the camera calibration is used to convert the pixel coordinates of the center point of the grabbing frame into the coordinates in the three-dimensional space:

[0084]

[0085]

[0086] Among them, M intri is the intrinsic parameter matrix of the camera, which is obtained by camera calibration; a, b are the scaling factors of the camera in the horizontal and vertical directions; u0, v0 represent the horizontal and vertical coordinates of the midpoint of the captured image in the pixel coordinate system, respectively.

[0087] In one example, the coordinate value (X arm ,Y arm ,Z arm ), that is, the calculation expression of the position of the robot end effector when grasping is:

[0088]

[0089] in, Represents the transformation matrix from the camera coordinate system to the robot base coordinate system; R, T are the rotation matrix and translation vector from the camera coordinate system to the base coordinate system of the robot arm calculated by hand-eye calibration.

[0090] In one example, the robot arm is a three-stage robot arm, and the inverse kinematics solution is used to obtain the solution of the three steering gears of the robot arm. However, inverse kinematics analysis generally requires a large number of matrix operations, and the process is complex and the amount of calculation is large, so it is difficult to implement. In order to better meet our needs, the present invention uses a geometric method to analyze the robot arm.

[0091] Simplify the model of the robot arm, remove the base pan / tilt and actuator parts to get the main body of the robot arm as shown below: Figure 4 As shown. Figure 4 It can be seen that the coordinates (x, y) of the end point P of the robot arm are finally composed of three parts (x1+x2+x3, y1+y2+y3). Figure 4 θ1, θ2, θ3 are the angles of the servos to be solved, and α is the angle between the claw and the horizontal plane, which is a known quantity. Figure 4 It can be seen that the claw's top-view angle α = θ1 + θ2 + θ3, from which we can conclude:

[0092] x=l1 cosθ1+l2 cos(θ1+θ2)+l3 cos(θ1+θ2+θ3) (1)

[0093] y=l1 sinθ1+l2 sin(θ1+θ2)+l3 sin(θ1+θ2+θ3) (2)

[0094] Among them, x and y are given by the user, and l1, l2, and l3 are the inherent properties of the mechanical structure of the robot arm, which are known quantities. In order to facilitate calculation, we will process the known parts and consider them as a whole:

[0095] m=l3 cos(α)-x (3)

[0096] n=l3 sin(α)-y (4)

[0097] Substituting equations (3) and (4) into equations (1) and (2), we can obtain the following simplified formula:

[0098] l2=(l1cosθ1+m) 2 +(l1sinθ1+n) 2 (5)

[0099] According to formula (5), sinθ1 is calculated as:

[0100]

[0101] The solution can be obtained according to the root-finding formula of the quadratic equation in equation (6), where:

[0102] a=m 2 +n 2 (7)

[0103]

[0104] Based on this, the angle of θ1 can be calculated. Similarly, θ2 and θ3 can be calculated, and then the angles of the three servos can be obtained. Then, the servos can be controlled according to the angles to achieve coordinate position control.

[0105] The present invention discloses a robot grasping posture estimation method based on semantic segmentation, which better realizes feature fusion and information interaction based on a lightweight feature fusion module, reduces pixel loss and improves segmentation effect, and then accurately identifies the target contour from a complex background, and then accurately extracts the contour of the target to be grasped in different postures and angles, determines the center point of the grasping frame, and calculates the precise grasping angle and position according to the target posture, finally realizing high-precision robot grasping operation.

[0106] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the semantic segmentation method formed by any one of the above examples or a combination of multiple examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0107] The present invention also provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned semantic segmentation method, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned semantic segmentation method when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0108] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0109] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned semantic segmentation method.

[0110] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0111] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0112] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0113] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0114] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.

[0115] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A semantic segmentation method, characterized in that: The following steps are involved: Construct a semantic segmentation network, including an encoder and a decoder. The encoder includes a backbone network and a feature pyramid pooling module connected in sequence. The backbone network is used to extract low-level features of the input image. The feature pyramid pooling module extracts high-level features of the input image based on the feature map output by the backbone network. The decoder includes a lightweight feature fusion module and an image spatial information recovery module connected in sequence; the lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule; The multi-scale feature aggregation submodule includes a multi-scale conversion unit, a cross-scale information interaction unit and a same-size feature generation unit; the multi-scale conversion unit is used to perform multi-scale conversion processing on low-level features and high-level features; the cross-scale information interaction unit is used to perform cross-aggregation processing on features of different scales to achieve cross-scale information interaction; the same-size feature generation unit is used to perform heterogeneous convolution processing to generate features of the same size; The attention feature fusion submodule is used to perform feature fusion processing on features of the same size; The image spatial information recovery module is used to perform convolution processing and up-sampling processing on the fusion features obtained by the fusion processing, and output the semantic segmentation map of the input image; Train the semantic segmentation network; The image to be segmented is input into the trained semantic segmentation network to obtain the semantic segmentation result.

2. The semantic segmentation method according to claim 1, characterized in that: The multi-scale conversion processing of low-level features and high-level features includes: The low-level features are sequentially upsampled and convolved, and the high-level features are convolved twice to complete the multi-scale conversion process.

3. The semantic segmentation method according to claim 2, characterized in that: Cross-aggregation of features at different scales, including: The low-level features, the low-level features that have completed the multi-scale conversion process, and the high-level features are subjected to dot multiplication processing to obtain the first cross-aggregation feature; and the high-level features, the low-level features that have completed the multi-scale conversion process, and the high-level features are subjected to dot multiplication processing to obtain the second cross-aggregation feature.

4. The semantic segmentation method according to claim 3, characterized in that: Perform heterogeneous convolution processing, including: A heterogeneous convolution is performed on the first cross-aggregation feature, a heterogeneous convolution is performed on the second cross-aggregation feature, and elements are added to the features of the same size obtained by the two heterogeneous convolutions.

5. The semantic segmentation method according to claim 1, characterized in that: The backbone network outputs the first low-level features and the second low-level features of different sizes. The semantic segmentation network compresses the first low-level features and then inputs them into the feature pyramid pooling module. The first low-level feature and the first high-level feature output by the feature pyramid pooling module are subjected to multi-scale feature fusion processing by a lightweight feature fusion module to obtain a second high-level feature; The second high-level features and the second low-level features are sequentially subjected to multi-scale feature fusion processing, convolution processing, and upsampling processing by a lightweight feature fusion module, and a semantic segmentation map of the input image is output.

6. A robot grasping pose estimation method based on semantic segmentation, characterized in that: The following steps are involved: Using the semantic segmentation method described in any one of claims 1 to 5 to perform semantic segmentation processing on the grasping operation image, extract the outline of the target to be grasped, and then calculate and obtain a grasping frame positioned on the outline of the target to be grasped; The pixel coordinates of the center point of the grab box are converted into coordinates in three-dimensional space using camera calibration; Calculate the coordinate value of the center point of the grab frame in the robot base coordinate system based on hand-eye calibration; Based on the inverse kinematics algorithm, the angle values ​​of each servo of the robot's manipulator arm when grasping the target to be grasped are solved, thereby realizing the grasping posture control of the manipulator arm.

7. The semantic segmentation method according to claim 6, characterized in that: The calculation expression of the coordinate value of the center point of the grab frame in the robot base coordinate system is: in, represents the transformation matrix from the camera coordinate system to the robot base coordinate system; R, T are the rotation matrix and translation vector from the camera coordinate system to the robot base coordinate system calculated by hand-eye calibration; X arm ,Y arm ,Z arm They respectively represent the horizontal coordinate, vertical coordinate and depth coordinate of the center of the grab frame in the robot base coordinate system.

8. The semantic segmentation method according to claim 6, characterized in that: The robotic arm is a three-stage robotic arm. The angle calculation expression of the three servos when grasping the target to be grasped is: x=l1cosθ1+l2cos(θ1+θ2)+l3cos(θ1+θ2+θ3) y=l1sinθ1+l2sin(θ1+θ2)+l3sin(θ1+θ2+θ3) α=θ1+θ2+θ3 Among them, x, y are the horizontal and vertical coordinates of the end points of the robotic arm respectively; θ1, θ2, θ3 are the angle values ​​of the three servos when grasping the target to be grasped; l1, l2, l3 are the lengths of the three sections of the robotic arm respectively; α represents the angle between the robotic arm and the horizontal plane.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot grasping pose estimation method described in any one of claims 1 to 5 or claims 6 to 8 are implemented.

10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, the steps of the robot grasping pose estimation method described in any one of claims 1-5 or claims 6-8 are performed.

Images