Semantic segmentation method, robot grasping pose estimation method based on semantic segmentation, computer program product and terminal

Abstract

This invention discloses a semantic segmentation method, a robot grasping pose estimation method based on semantic segmentation, a computer program product, and a terminal, belonging to the field of semantic segmentation. The method includes: constructing a semantic segmentation network, whose decoder includes a lightweight feature fusion module for multi-scale transformation of low-level and high-level features, followed by cross-aggregation of features at different scales, heterogeneous convolution, and finally feature fusion; training the semantic segmentation network; and inputting the image to be segmented into the trained semantic segmentation network to obtain the semantic segmentation result. The lightweight feature fusion module effectively solves the problem of feature fusion with inconsistent semantics and scales, better achieving feature fusion and information interaction, reducing pixel loss, and significantly improving the quality of fused features, thereby improving the accuracy of semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation, and more particularly to a semantic segmentation method, a robot grasping pose estimation method based on semantic segmentation, a computer program product, and a terminal. Background Technology

[0002] DeepLabv3+ is an advanced semantic segmentation model that introduces a decoder module on top of DeepLabv3 to improve the segmentation accuracy of object boundaries. This model employs an encoder-decoder architecture, consisting of two main parts: an encoder and a decoder. The encoder extracts high-level semantic information from the image, while the decoder recovers the spatial information to obtain accurate segmentation results. In the encoder, the input passes through a backbone network to produce two outputs: low-level features and high-level features. Although the features output by the DeepLabv3+ network along the semantic path contain rich global contextual semantic information, the 1 / 4 size difference between low-level and high-level features prevents them from being directly combined with Attentional Feature Fusion (AFF) networks, which are suitable for the same size and number of channels. Directly applying AFF to semantic features with a large size difference may result in pixel loss, affecting the quality of the fused features and reducing the accuracy of semantic segmentation. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems of the prior art and provide a semantic segmentation method, a robot grasping pose estimation method based on semantic segmentation, a computer program product and terminal.

[0004] The objective of this invention is achieved through the following technical solution: a semantic segmentation method, which includes the following steps:

[0005] A semantic segmentation network is constructed, including an encoder and a decoder. The encoder includes a backbone network and a feature pyramid pooling module connected in sequence. The backbone network is used to extract low-level features of the input image, and the feature pyramid pooling module extracts high-level features of the input image based on the feature map output by the backbone network.

[0006] The decoder comprises a lightweight feature fusion module and an image spatial information restoration module connected sequentially. The lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule. The multi-scale feature aggregation submodule includes a multi-scale transformation unit, a cross-scale information interaction unit, and a same-size feature generation unit. The multi-scale transformation unit performs multi-scale transformation processing on low-level and high-level features. The cross-scale information interaction unit performs cross-aggregation processing on features of different scales to achieve cross-scale information interaction. The same-size feature generation unit performs heterogeneous convolution processing to generate features of the same size. The attention feature fusion submodule performs feature fusion processing on features of the same size. The image spatial information restoration module performs convolution processing and upsampling processing on the fused features obtained from the fusion processing to output a semantic segmentation map of the input image.

[0007] Train the semantic segmentation network;

[0008] The image to be segmented is input into the trained semantic segmentation network to obtain the semantic segmentation result.

[0009] In one example, multi-scale transformation processing of low-level and high-level features includes:

[0010] Low-level features are sequentially upsampled and convolved, while high-level features are convolved twice to complete the multi-scale transformation.

[0011] In one example, cross-aggregation of features at different scales is performed, including:

[0012] The first cross-aggregated feature is obtained by multiplying the low-level feature, the low-level feature that has undergone multi-scale transformation, and the high-level feature; the second cross-aggregated feature is obtained by multiplying the high-level feature, the low-level feature that has undergone multi-scale transformation, and the high-level feature.

[0013] In one example, heterogeneous convolution processing is performed, including:

[0014] Heterogeneous convolution is performed on the first cross-aggregation feature and the second cross-aggregation feature. The features with the same size obtained from the two heterogeneous convolution processes are then added element by element.

[0015] In one example, the backbone network outputs first and second low-level features of different sizes. The semantic segmentation network compresses the first low-level features and then inputs them into the feature pyramid pooling module.

[0016] The first low-level feature and the first high-level feature output by the feature pyramid pooling module are processed by the lightweight feature fusion module to perform multi-scale feature fusion processing to obtain the second high-level feature.

[0017] The second high-level feature and the second low-level feature are sequentially processed by a lightweight feature fusion module through multi-scale feature fusion, convolution, and upsampling to output a semantic segmentation map of the input image.

[0018] It should be further noted that the technical features corresponding to the above examples of semantic segmentation methods can be combined or replaced to form new technical solutions.

[0019] This invention also includes a robot grasping pose estimation method based on semantic segmentation, comprising the following steps:

[0020] The semantic segmentation method formed by any one or more of the above examples is used to perform semantic segmentation processing on the image of the grabbing task, extract the contour of the target to be grabbed, and then calculate the grabbing box located on the contour of the target to be grabbed.

[0021] Camera calibration is used to convert the pixel coordinates of the center point of the capture frame into coordinates in three-dimensional space.

[0022] The coordinates of the center point of the grasping frame in the robot base coordinate system are calculated based on hand-eye calibration.

[0023] The inverse kinematics algorithm is used to solve the angle values ​​of each servo motor of the robot arm when grasping the target, thereby realizing the grasping posture control of the robot arm.

[0024] In one example, the expression for calculating the coordinates of the center point of the grab box in the robot base coordinate system is:

[0025]

[0026] in, R represents the transformation matrix from the camera coordinate system to the robot's base coordinate system; R and T are the rotation matrix and translation vector from the camera coordinate system to the robot's base coordinate system, calculated from the hand-eye calibration; X arm ,Y arm Z arm These represent the x-coordinate, y-coordinate, and depth coordinates of the center of the grasping frame in the robot's base coordinate system, respectively.

[0027] In one example, the robotic arm is a three-segment robotic arm, and the expression for calculating the angle values ​​of the three servo motors when grasping the target is as follows:

[0028] x=l1cosθ1+l2cos(θ1+θ2)+l3cos(θ1+θ2+θ3)

[0029] y=l1sinθ1+l2sin(θ1+θ2)+l3sin(θ1+θ2+θ3)

[0030] α=θ1+θ2+θ3

[0031] Where x and y are the horizontal and vertical coordinates of the end point of the robotic arm, respectively; θ1, θ2, and θ3 are the angle values ​​of the three servo motors when grasping the target; l1, l2, and l3 are the lengths of the three segments of the robotic arm; and α represents the angle between the robotic arm and the horizontal plane.

[0032] It should be further noted that the technical features corresponding to the above examples can be combined or substituted to form new technical solutions.

[0033] The present invention also includes a computer program product comprising a computer program that, when executed by a processor, implements the steps of the semantic segmentation method formed by any or a combination of the above examples.

[0034] The present invention also includes a terminal comprising a memory and a processor, the memory storing computer instructions executable on the processor, wherein the processor, when executing the computer instructions, performs the steps of the robot grasping pose estimation method formed by any or more of the above examples.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. In one example, a lightweight feature fusion module performs multi-scale transformation, cross-aggregation, and heterogeneous convolution on low-level and high-level features to generate features of the same size. Feature fusion is then performed on this basis, which can effectively solve the problem of feature fusion with inconsistent semantics and scale, better realize feature fusion and information interaction, reduce pixel loss, significantly improve the quality of fused features, and thus improve the accuracy of semantic segmentation.

[0037] 2. In one example, the language segmentation network is applied to the robot grasping pose estimation scenario, which can ensure the segmentation accuracy of the grasping operation image, so as to accurately identify the target contour from the complex background, and then accurately extract the contour of the target to be grasped at different poses and angles, determine the center point of the grasping box, and then achieve efficient and stable grasping of targets at different poses and angles based on coordinate transformation and robotic arm grasping posture control. Attached Figure Description

[0038] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0039] Figure 1 A flowchart illustrating a method provided as an example of the present invention;

[0040] Figure 2 This is a schematic diagram of the lightweight feature fusion module structure provided as an example of the present invention;

[0041] Figure 3 This is a schematic diagram of the DeepLabv3+ network structure provided as an example of the present invention;

[0042] Figure 4 A simplified model diagram of a robotic arm provided as an example of the present invention. Detailed Implementation

[0043] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0045] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In one example, such as Figure 1 As shown, a semantic segmentation method includes the following steps:

[0048] S10: Construct a semantic segmentation network.

[0049] This example uses the DeepLabv3+ network for semantic segmentation. Building upon the existing DeepLabv3+ network with integrated Attentional Feature Fusion (AFF) module, the AFF structure is further optimized by proposing a Light Attentional Feature Fusion (LAFF) module. This LAFF module includes a multi-scale feature aggregation submodule and an attentional feature fusion submodule (attentional feature fusion module). By introducing the lightweight feature fusion module, the challenges posed by scale variations and small targets are effectively addressed, thereby improving the accuracy of semantic segmentation.

[0050] Specifically, the semantic segmentation network includes an encoder and a decoder. The encoder consists of a backbone network and an Atrous Spatial Pyramid Pooling (ASPP) module connected in sequence. The backbone network is used to extract low-level features from the input image. The backbone network can be a deep convolutional neural network (such as ResNet), but in this example, MobileNetV2 is preferred to meet the requirements of lightweight data processing. The Atrous Spatial Pyramid Pooling (ASPP) module extracts high-level features from the input image based on the feature map (low-level features) output by the backbone network.

[0051] The decoder comprises a lightweight feature fusion module and an image spatial information restoration module connected sequentially. The lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule. The multi-scale feature aggregation submodule includes a multi-scale transformation unit, a cross-scale information interaction unit, and a same-size feature generation unit. The multi-scale transformation unit performs multi-scale transformation processing on low-level and high-level features. The cross-scale information interaction unit performs cross-aggregation processing on features of different scales to achieve cross-scale information interaction. The same-size feature generation unit performs heterogeneous convolution processing to generate features of the same size. The attention feature fusion submodule performs feature fusion processing on features of the same size. The image spatial information restoration module performs convolution processing and upsampling processing on the fused features obtained from the fusion processing to output a semantic segmentation map of the input image.

[0052] S20: Train the semantic segmentation network.

[0053] First, prepare and preprocess the training dataset, including images and their corresponding labeled maps. Then, based on the constructed DeepLabv3+ network, select an appropriate loss function such as cross-entropy loss, and an optimizer such as SGD or Adam, and set the learning rate and training epochs. Next, iteratively train the model on the training set while evaluating its performance on the validation set, adjusting hyperparameters to optimize the model. Finally, evaluate the trained model using the test set to determine its generalization ability. Throughout the training process, data augmentation and learning rate adjustments may be necessary to further improve the model's performance and robustness.

[0054] S30: Input the image to be segmented into the trained semantic segmentation network to obtain the semantic segmentation result, that is, to obtain the foreground target contour.

[0055] In one example, such as Figure 2 As shown, the multi-scale transformation unit includes a low-level feature transformation subunit and a high-level feature transformation subunit. The low-level feature transformation subunit includes an upsampling layer and a first convolutional layer (Conv1) connected in sequence; the high-level feature transformation subunit includes a second convolutional layer (Conv2) and a third convolutional layer (Conv3) connected in sequence.

[0056] In the LAFF structure, assume there are two input features X∈R of different sizes. W×H×C and Y∈R W / 2×H / 2×C / 2 Where Y represents the high-level semantic features generated by the feature pyramid pooling module, while X represents the low-level features obtained by the backbone network through convolution. Multi-scale transformation processing of the low-level and high-level features includes:

[0057] The low-level feature X is subjected to bilinear interpolation with a 2x upsampling process, followed by a 3×3 convolution with a stride of 2; the high-level feature Y is subjected to a 3×3 convolution with a stride of 2 and a 5×5 convolution with a stride of 2, thereby completing the multi-scale transformation in order to perform cross-aggregation.

[0058] In one example, such as Figure 2 As shown, the cross-scale information interaction unit includes a first dot product layer and a second dot product layer. The first dot product layer is connected to the first convolutional layer, the third convolutional layer, and the backbone network. The second dot product layer is connected to the first convolutional layer, the third convolutional layer, and the feature pyramid pooling module.

[0059] In the LAFF structure, assume there are two input features X∈R of different sizes. W×H×C and Y∈R W / 2×H / 2×C / 2 Where Y represents the high-level semantic features generated by the feature pyramid pooling module, and X represents the low-level features obtained by the backbone network through convolution. Cross-aggregation processing is performed on features at different scales, including:

[0060] The first cross-aggregated feature is obtained by multiplying the low-level feature X, the low-level feature that has undergone multi-scale transformation, and the high-level feature Y, and the low-level feature that has undergone multi-scale transformation, and the high-level feature Y.

[0061] This example achieves cross-aggregation of features at different scales through pixel-by-pixel multiplication, enabling the effective fusion of features of X and Y at different resolutions and realizing cross-scale information interaction.

[0062] In one example, such as Figure 2 As shown, the same-size feature generation unit further enhances the feature learning of the two branches by using heterogeneous convolution with a smaller number of parameters. It includes a first heterogeneous convolutional layer (HetConv1), a second heterogeneous convolutional layer (HetConv2), and a first element-adding layer. The first heterogeneous convolutional layer is connected to the first dot product layer, and the second heterogeneous convolutional layer is connected to the second dot product layer. The outputs of the first heterogeneous convolutional layer and the second heterogeneous convolutional layer are both connected to the first element-adding layer.

[0063] Perform heterogeneous convolution processing, including:

[0064] Heterogeneous convolution is performed on the first cross-aggregation feature and the second cross-aggregation feature to generate features u∈R of the same size. W×H×C and v∈R W×H×C Then u∈R W×H×C and v∈R W×H×C Element addition processing is performed to obtain preliminary fusion features.

[0065] Combining the examples of the multi-scale transformation unit, cross-scale information interaction unit, and same-size feature generation unit, the calculation expression for the entire data processing process is as follows:

[0066]

[0067] in, Indicates dot product processing; Dw 3×3 Dw represents a 3×3 convolution; 5×5 This represents a 5×5 convolution; Up indicates upsampling. M represents the element addition process; Z represents the mask function; and Z represents the initial fusion feature.

[0068] In this example, LAFF, as a lightweight feature fusion module, effectively addresses the problem of feature fusion with inconsistent semantics and scale. Building upon the original structure, LAFF enables multi-scale feature exchange and integration at different sizes, preserving more detailed information from low-level features while acquiring rich contextual high-level semantic information. This design significantly improves the quality of fused features across various network scenarios, effectively addressing the challenges posed by scale variations and small targets, thereby enhancing the accuracy of semantic segmentation.

[0069] In one example, the attention feature fusion submodule includes a local attention branch and a global attention branch, both connected to the first element-adding layer. The local attention branch includes a fourth convolutional layer (Point-wiseConv4), a first activation function layer (ReLU1), and a fifth convolutional layer (Point-wise Conv5) connected in sequence. The global attention branch includes a global average pooling layer (GAP), a sixth convolutional layer (Point-wise Conv6), a second activation function layer (ReLU2), and a seventh convolutional layer (Point-wise Conv7) connected in sequence. The outputs of the fifth and seventh convolutional layers are connected to the second element-adding layer. The second element-adding layer is connected to the third activation function layer (Sigmoid). The third activation function layer is connected to the third and fourth dot-multiplication layers, respectively. The first heterogeneous convolutional layer is connected to the third dot-multiplication layer, and the second heterogeneous convolutional layer is connected to the fourth dot-multiplication layer. The outputs of the third and fourth dot-multiplication layers are both connected to the third element-adding layer.

[0070] The local attention branch uses 1x1 convolution and ReLU activation function to calculate local attention and capture local features and details; the global attention branch calculates global attention through operations such as global average pooling, 1x1 convolution, and ReLU activation function to capture global contextual information of the image. The local and global attention are added together, and then weights are generated by the sigmoid function. The input features are multiplied by the weights obtained by the sigmoid function to complete the weighted fusion of features, thereby focusing more on the key parts of the image.

[0071] Preferably, such as Figure 3 As shown, this example decoder includes two lightweight feature fusion modules: a first lightweight feature fusion module (LAFF1) and a second lightweight feature fusion module (LAFF2). The output of the backbone network and the output of the feature pyramid pooling module are both connected to the first lightweight feature fusion module, and the output of the backbone network and the output of the first lightweight feature fusion module are both connected to the second lightweight feature fusion module. The second lightweight feature fusion module is connected to the image spatial information restoration module, which includes a convolutional layer and an upsampling layer.

[0072] Furthermore, the backbone network is MobileNetV2, used to output feature maps (low-level features) of different sizes, including a first low-level feature L1 (shallow features 1 / 8 the size of the original input image) and a second low-level feature L2 (shallow features 1 / 2 the size of the original input image). The semantic segmentation network compresses the first low-level feature L1 and inputs it into the feature pyramid pooling module. The feature pyramid pooling module processes the feature map in parallel using dilated convolutions with different dilation rates and global average pooling, thereby extracting features at different scales. The features at different scales are then processed by 3×3 convolutions to generate the first high-level feature. The first low-level feature L1 and the first high-level feature L2 are then processed by the first lightweight feature fusion module for multi-scale feature fusion to obtain the second high-level feature H2. The second high-level feature H2 and the second low-level feature L2 are then processed by the second lightweight feature fusion module for multi-scale feature fusion, and then processed by the image spatial information recovery module for convolution and upsampling to obtain a prediction map with the same resolution as the input image, i.e., the semantic segmentation map of the input image.

[0073] This invention also includes a robot grasping pose estimation method based on semantic segmentation, comprising the following steps:

[0074] S100: Using any one or more of the above examples to form a semantic segmentation method, perform semantic segmentation processing on the input grabbing image, extract the contour of the target to be grabbed, and then calculate the grabbing box located on the contour of the target to be grabbed.

[0075] The image to be captured is a two-dimensional image including the target to be captured and a complex background. The target can be any object, such as a pen or an earphone case. This application uses Deeplabv3. + Semantic segmentation networks perform accurate contour extraction of objects. DeepLabv3 + By introducing dilated convolution and feature pyramid pooling modules, multi-scale features can be captured, and object boundaries can be accurately segmented, especially separating the contours of objects to be grasped in complex backgrounds. Using the semantic segmentation results and the defined grasping box geometry, the optimal grasping box is selected, and its center coordinates and rotation angle are calculated. This process aims to ensure that the grasping point corresponding to the selected grasping box is located on the object, not on the background or occluders.

[0076] Furthermore, in order to improve the adaptability to objects of different sizes, this invention designs a lightweight feature fusion module. Through this module, feature maps at different levels can share similar semantic information, ensuring that the grab box can be accurately positioned on the object regardless of its size.

[0077] S200: Uses camera calibration to convert the pixel coordinates of the center point of the capture box into coordinates in three-dimensional space.

[0078] Robots need to know the exact location of the target in three-dimensional space in order to navigate and grasp it. Through coordinate transformation, robots can accurately locate objects and perform tasks.

[0079] S300: Calculates the coordinates of the center point of the grasping frame in the robot base coordinate system based on hand-eye calibration.

[0080] By using hand-eye calibration, pixel coordinates in a two-dimensional image can be converted into three-dimensional spatial coordinates in the robot's base coordinate system, thereby achieving precise positioning of the target to be grasped.

[0081] S400: Based on the inverse kinematics algorithm, the angle values ​​of each servo motor of the robot arm when grasping the target are solved, thereby realizing the grasping posture control of the robot arm.

[0082] Inverse kinematics algorithms can calculate the precise angles that each joint of the robot needs to reach based on the position of the target to be grasped, thereby enabling the robotic arm to move accurately to the predetermined position for grasping. At the same time, inverse kinematics algorithms can improve the robot's operational accuracy, and the control enables the robot to grasp or manipulate the target to be grasped more accurately.

[0083] In one example, the 3D coordinates of the center point of the capture frame in camera coordinates are calculated based on camera calibration principles. Let the pixel coordinates of the center point of the capture frame be (u, v), and the coordinates of the center point of the capture frame in the camera coordinate system be (X, v). cam ,Y cam Z cam At this point, the calculation expression for converting the pixel coordinates of the center point of the capture frame into coordinates in three-dimensional space using camera calibration is as follows:

[0084]

[0085]

[0086] Among them, M intri is the camera's intrinsic parameter matrix, obtained from camera calibration; a and b are the camera's scaling factors in the horizontal and vertical directions, respectively; u0 and v0 represent the horizontal and vertical coordinates of the midpoint of the captured image in the pixel coordinate system, respectively.

[0087] In one example, the coordinates (X, X) of the center point of the grab box in the robot base coordinate system. arm ,Y arm Z arm The expression for calculating the position of the robot's end effector during grasping is:

[0088]

[0089] in, R represents the transformation matrix from the camera coordinate system to the robot base coordinate system; R and T are the rotation matrix and translation vector from the camera coordinate system to the robot's base coordinate system, calculated by hand-eye calibration.

[0090] In one example, the robotic arm is a three-segment robotic arm, and the solutions for the three servo motors of the robotic arm are obtained using inverse kinematics. However, inverse kinematics analysis generally involves a large number of matrix operations, which is complex and computationally intensive, making it difficult to implement. To better meet our needs, this invention uses a geometric method to analyze the robotic arm.

[0091] The robotic arm model is simplified by removing the base, gimbal, and actuator parts, resulting in the main body of the robotic arm as shown below. Figure 4 As shown. From Figure 4 We can see that the coordinates (x, y) of the robotic arm's endpoint P are ultimately composed of three parts (x1 + x2 + x3, y1 + y2 + y3). Figure 4 In this equation, θ1, θ2, and θ3 represent the angles of the servo motor that we are trying to solve for, and α is the angle between the claw and the horizontal plane, which is a known quantity. From... Figure 4 It can be seen that the downward angle of the claw is α = θ1 + θ2 + θ3, from which we can conclude:

[0092] x=l1 cosθ1+l2 cos(θ1+θ2)+l3 cos(θ1+θ2+θ3) (1)

[0093] y=l1 sinθ1+l2 sin(θ1+θ2)+l3 sin(θ1+θ2+θ3) (2)

[0094] Where x and y are provided by the user, and l1, l2, and l3 are inherent mechanical structural properties of the robotic arm, which are known quantities. For ease of calculation, we will process the known parts and consider them as a whole:

[0095] m=l3 cos(α)-x (3)

[0096] n=l3 sin(α)-y (4)

[0097] Substituting equations (3) and (4) into equations (1) and (2) and simplifying, we get:

[0098] l2=(l1cosθ1+m) 2 +(l1sinθ1+n) 2 (5)

[0099] Calculate sinθ1 according to equation (5):

[0100]

[0101] The quadratic equation can be solved using the quadratic formula in equation (6), where:

[0102] a = m 2 +n 2 (7)

[0103]

[0104] Based on this, the angle θ1 can be calculated, and similarly, θ2 and θ3 can be calculated, thus obtaining the angles of the three servo motors. Then, by controlling the servo motors according to the angles, the coordinate position can be controlled.

[0105] This invention provides a robot grasping pose estimation method based on semantic segmentation. It utilizes a lightweight feature fusion module to better achieve feature fusion and information interaction, reduce pixel loss and improve segmentation results. This allows for accurate identification of target contours in complex backgrounds, accurate extraction of the contours of the target to be grasped at different poses and angles, determination of the center point of the grasping box, and calculation of the precise grasping angle and position based on the target posture, ultimately achieving high-precision robot grasping operation.

[0106] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the semantic segmentation method formed by any or a combination of the above examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0107] The present invention also provides a terminal having the same inventive concept as any example or combination of examples corresponding to the semantic segmentation method described above, including a memory and a processor. The memory stores computer instructions executable on the processor, and the processor executes the steps of the semantic segmentation method described above when executing the computer instructions. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0108] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0109] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section above, based on various exemplary embodiments of the present invention. For example, the processing unit can execute one of the semantic segmentation methods described above.

[0110] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0111] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0112] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0113] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.

[0115] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method of semantic segmentation, characterized in that, Includes the following steps: A semantic segmentation network is constructed, including an encoder and a decoder. The encoder includes a backbone network and a feature pyramid pooling module connected in sequence. The backbone network is used to extract low-level features of the input image, and the feature pyramid pooling module extracts high-level features of the input image based on the feature map output by the backbone network. The decoder includes a lightweight feature fusion module and an image spatial information recovery module connected sequentially; the lightweight feature fusion module includes a multi-scale feature aggregation submodule and an attention feature fusion submodule; The multi-scale feature aggregation submodule includes a multi-scale transformation unit, a cross-scale information interaction unit, and a same-size feature generation unit. The multi-scale transformation unit is used to perform multi-scale transformation processing on low-level and high-level features. The cross-scale information interaction unit is used to perform cross-aggregation processing on features of different scales to achieve cross-scale information interaction. The same-size feature generation unit is used to perform heterogeneous convolution processing to generate features of the same size. The attention feature fusion submodule is used to perform feature fusion processing on features of the same size; The image spatial information restoration module is used to perform convolution and upsampling processing on the fused features obtained from the fusion process, and output a semantic segmentation map of the input image. Train the semantic segmentation network; The image to be segmented is input into the trained semantic segmentation network to obtain the semantic segmentation result; The multi-scale transformation unit includes a low-level feature transformation subunit and a high-level feature transformation subunit. The low-level feature transformation subunit includes a sequentially connected upsampling layer and a first convolutional layer. The advanced feature transformation subunit consists of a second and a third convolutional layer connected in sequence; for Multi-scale transformation of low-level and high-level features includes: Low-level features are subjected to bilinear interpolation with a 2x upsampling process, followed by a 3×3 convolution with a stride of 2; high-level features are subjected to a 3×3 convolution with a stride of 2 and a 5×5 convolution with a stride of 2, thus completing the multi-scale transformation. The cross-scale information interaction unit includes a first dot-multiplication layer and a second dot-multiplication layer. The first dot-multiplication layer is connected to the first convolutional layer, the third convolutional layer, and the backbone network. The second dot-multiplication layer is connected to the first convolutional layer, the third convolutional layer, and the feature pyramid pooling module. Cross-aggregation processing is performed on features at different scales, including: The first cross-aggregated feature is obtained by multiplying the low-level feature, the low-level feature that has undergone multi-scale transformation, and the high-level feature; the second cross-aggregated feature is obtained by multiplying the high-level feature, the low-level feature that has undergone multi-scale transformation, and the high-level feature.

2. The semantic segmentation method of claim 1, wherein, Perform heterogeneous convolution processing, including: Heterogeneous convolution is performed on the first cross-aggregation feature and the second cross-aggregation feature. The features with the same size obtained from the two heterogeneous convolution processes are then added element by element.

3. The semantic segmentation method according to claim 1, characterized in that, The backbone network outputs first and second low-level features of different sizes. The semantic segmentation network compresses the first low-level features and then inputs them into the feature pyramid pooling module. The first low-level feature and the first high-level feature output by the feature pyramid pooling module are processed by the lightweight feature fusion module to perform multi-scale feature fusion processing to obtain the second high-level feature. The second high-level feature and the second low-level feature are sequentially processed by a lightweight feature fusion module through multi-scale feature fusion, convolution, and upsampling to output a semantic segmentation map of the input image.

4. A method for robot grasping pose estimation based on semantic segmentation, characterized in that, Includes the following steps: The semantic segmentation method described in any one of claims 1-3 is used to perform semantic segmentation processing on the image of the grabbing operation, extract the contour of the target to be grabbed, and then calculate the grabbing box located on the contour of the target to be grabbed. Camera calibration is used to convert the pixel coordinates of the center point of the capture frame into coordinates in three-dimensional space. The coordinates of the center point of the grasping frame in the robot base coordinate system are calculated based on hand-eye calibration. The inverse kinematics algorithm is used to solve the angle values ​​of each servo motor of the robot arm when grasping the target, thereby realizing the grasping posture control of the robot arm.

5. The method of claim 4, wherein, The expression for calculating the coordinates of the center point of the grab box in the robot base coordinate system is as follows: ； ； wherein, represents the transformation matrix from the camera coordinate system to the robot base coordinate system; R, T are the rotation matrix and the translation vector from the camera coordinate system to the base coordinate system of the robot arm calculated by the hand-eye calibration; respectively represent the horizontal coordinate, the vertical coordinate and the depth coordinate of the center of the grasping frame in the robot base coordinate system.

6. The method of claim 5, wherein, The robotic arm is a three-segment robotic arm. The angle values ​​of the three servo motors when grasping the target are calculated as follows: ； ； ； wherein, are respectively the horizontal and vertical coordinates of the end point of the mechanical arm; are respectively the angle values of the three steering wheels when grabbing the target to be grabbed; are respectively the lengths of the three segments of the mechanical arm; denotes the angle between the mechanical arm and the horizontal plane.

7. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the semantic segmentation method according to any one of claims 1-3, or when the computer program is executed by a processor, it implements the steps of the robot grasping pose estimation method based on semantic segmentation according to claim 4 or 5.

8. A terminal comprising a memory and a processor, said memory having stored thereon computer instructions executable on said processor, characterized in that, When the processor executes the computer instructions, it performs the semantic segmentation method according to any one of claims 1-3; or when the computer program is executed by the processor, it implements the steps of the robot grasping pose estimation method based on semantic segmentation according to claim 4 or 5.

Images