Mechanical arm control method and system based on improved Swinin-Unet

Abstract

The invention relates to the technical field of computer vision and mechanical arm automation, and discloses a mechanical arm control method and system based on improved Swinin-Unet. Comprising the following steps: S0, acquiring an RGB color image and a depth image of a target working scene; s1, converting the depth image into a normal vector angle image, and splicing the depth image and the normal vector angle image to obtain a space image; s2, analyzing the RGB color image and the space image by using a grabbing model to obtain grabbing parameters; the capturing model is constructed based on improved Swinin-Unet, and comprises a coding part which is used for respectively extracting features of a plurality of set scales of the RGB color image and the space image and carrying out feature fusion; in the decoding part, the fusion features of the minimum scale are sampled upwards step by step to the input resolution, the fusion features of the corresponding scales are fused through jump connection, and then capture parameters are obtained through prediction; and S3, the mechanical arm is controlled to grab the target object based on the grabbing parameters. According to the method, the position information of the target object can be accurately judged in a multi-object disordered scene, and grabbing is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and robot arm automation, and in particular to a robot arm control method and system based on improved Swin-Unet. Background Art

[0002] In recent years, with the continuous development of automation technology, intelligent robots are gradually expanding from the industrial field to all walks of life. Among them, the automated application of civil engineering robots has become an emerging topic in recent years. By extracting and segmenting the collected visual image features, the automated operation of the robotic arm is realized, such as rock drilling robots, hoisting robotic arms, building decoration robots, etc. However, most traditional robots complete fixed assembly line tasks through pre-programming or manual teaching, and it is difficult to adapt to more complex construction site environments. Grasping objects is an important function of robots, and its accuracy and practicality reflect the intelligence level of robots to a certain extent. Researching reliable and efficient grasping algorithms will be the key to improving the automation level of construction robots.

[0003] According to the input data type, existing end-to-end crawl prediction algorithms can be divided into the following three types:

[0004] (1) Grasping posture detection method based on two-dimensional images: This method requires the geometric properties of the object to be acquired in advance, and is heavily dependent on the simplification of the physical model and the assumption of a fully observable environment, resulting in limited application scenarios, a single grasping object, and poor generalization. Compared with two-dimensional color images, three-dimensional vision can additionally reflect the spatial depth information of the target object and can also identify objects without texture and with occlusion. For disordered grasping scenarios of scattered objects, where the heights and volumes of each target are different, a three-dimensional vision grasping system must be used.

[0005] (2) Grasping posture detection method based on point cloud segmentation: This method can help the robotic arm identify specific objects more accurately, but it requires obtaining high-precision point cloud data first, which has high requirements for hardware equipment and is currently not easy to popularize in daily robotic arm grasping tasks.

[0006] (3) Grasping posture distribution map prediction method based on RGBD three-dimensional vision: For an input RGBD image, a model is built to output a set of prediction values ​​for each pixel in it, including grasping confidence and grasping parameters (such as grasping depth, width and rotation angle). This method can achieve good results, but it still has the following shortcomings:

[0007] (a) To find the graspable area, the robot must not only pay attention to the local geometric information of the object, but also consider its entire visual appearance. Especially in cluttered unstructured environments, handling different shapes, positions, and spatial relationships between objects is a difficult part of grasp detection.

[0008] (b) Existing 3D vision algorithms often treat the depth map as a grayscale image, concatenate it with the three color channels to form an RGBD four-channel input vector, and apply various 2D vision algorithms on this basis, such as Unet, ViT, Swin Transformer, etc., to use depth information to assist entity segmentation. However, in fact, traditional 2D vision algorithms may not be suitable for direct application in the processing of depth images. There are certain differences in the representation of semantic information between RGB color images and depth images. Depth images capture depth information and two-dimensional grid structures, and contain implicit local collection features and directional information, which are more like complementary to RGB color images. In addition, depth images are generally obtained by emitting infrared lasers through cameras and receiving corresponding reflected signals. They are easily disturbed by ambient light, smooth surfaces of objects, etc., and therefore have more noise and missing values. At this time, directly concatenating them with RGB may be counterproductive.

[0009] Therefore, how to fully utilize depth information to improve the performance of pose estimation remains a difficult and under-researched problem. Summary of the invention

[0010] Based on the current problems, the present invention provides a robotic arm control method and system based on improved Swin-Unet, which can solve the problems of insufficient utilization of depth information and weak local and global perception capabilities during three-dimensional visual grasping of the robotic arm.

[0011] The technical solution adopted by the present invention is:

[0012] The robot control method based on the improved Swin-Unet adopts the architecture of the improved Swin-Unet model, and constructs the grasping posture prediction model framework SU-Grasp based on the three-dimensional visual dual-stream encoding strategy based on the four-layer Swin-Unet network; the entire network architecture contains 5 modules, namely, point cloud preprocessing module, dual-stream encoder, bottleneck layer, decoder and skip connection layer;

[0013] The robot arm control method based on the improved Swin-Unet includes the following steps:

[0014] Step S0: Obtain an RGBD image of the target working scene, that is, data containing an RGB color image and a depth image, and use it as a model input;

[0015] Step S1: converting the depth image into a normal vector angle image, and performing a splicing operation on the depth image and the normal vector angle image to obtain a spatial image;

[0016] Step S2: Analyze the RGB color image and the spatial image using the grasping model to obtain grasping parameters; regard the RGB two-dimensional image as a color semantic modality, and regard the depth map D and the normal angle image I as a spatial semantic modality as inputs to the SU-Grasp network;

[0017] Step S3: Controlling the robotic arm to grasp the target object based on the grasping parameters.

[0018] Furthermore, in the architecture of the Swin-Unet model, the encoder, bottleneck layer and decoder are built based on the SwinTransformer Block model; the Swin-Unet network structure divides the input image into non-overlapping image blocks, which are input to the encoder to learn deep feature representation; after the context features are extracted, they are upsampled by the decoder with the image block expansion layer and connected to the encoder at the same level through skip connections to gradually restore the spatial resolution of the feature map and perform pixel-level segmentation prediction; the Swin-Unet model is used as the basic network framework of the SU-Grasp model to obtain local information and long-range connections at the same time;

[0019] In the implementation of the SU-Grasp model, three grasping outputs are connected in parallel to the top of the decoder, representing the grasping confidence Q, grasping width W, and grasping angle Θ, respectively. Each output is a heat map of the same size as the input RGBD image.

[0020] Further, in step S0, the data includes a Cornell dataset and a Jacquard dataset; step S0 also includes preprocessing the depth image, and the preprocessing includes a plane segmentation operation based on a random sampling consensus algorithm RANSAC, and an outlier removal operation based on a nearest neighbor search within a radius.

[0021] Furthermore, in step S1, the normal vector angle image is introduced to divide the three-dimensional vision into two semantic modes, color mode and spatial mode, and the information interaction between the two is enhanced by the multimodal fusion method; in the spatial mode, during the grasping process: first, the overall position and posture of the object to be grasped is determined; secondly, the local geometric information of the object surface is provided to assist in deciding the most stable grasping point.

[0022] Further, in step S1, the Open3D library is called to convert the preprocessed depth map into a normal vector angle image; the calculation process of the normal vector angle image is as follows:

[0023] For each pixel (u, v) in the depth image D, calculate the unit normal vector N(n) of the object surface corresponding to the pixel x ,n y ,n z );

[0024] Determine the angle between N and the three axes XYZ of the camera coordinate system;

[0025] Assume that the three axes XYZ are represented by vectors x(1,0,0), y(0,1,0), z(0,0,1), Ν(n x ,n y ,n z ) and the angles between these three axis vectors can be expressed as:

[0026] a x =arccos(N·x)

[0027] a y =arccos(N·y)

[0028] a z =arccos(N·z)

[0029] Where N is the unit normal vector of the object surface corresponding to the current pixel; x, y, z represent the unit normal vectors in the camera coordinate system that are in the same direction as the positive direction of the three coordinate axes; a x 、a y 、a z Corresponding to the angle between N and the x, y, and z axes; the arccos function is used to calculate the cosine of the angle between two vectors.

[0030] The angle (a x ,a y ,a z ) is normalized to the range of 0 to 255 and used to create a three-channel normal vector angle image I nrm , then each pixel has a normalized (a x ,a y ,a z ) as its value.

[0031] Furthermore, in step S1, the normal vector angle image is introduced into the robot grasping task to explicitly express the hidden semantics of the depth information on the surface direction of the object; the depth image and the RGB image are regarded as two modalities, the former describes the depth difference and structural information between different objects, and the latter reflects the appearance information including the color, shape and boundary of a scene; a dual-stream structure is used to explore the respective advantages of different modalities and their complementary properties;

[0032] The dual-stream structure uses two encoders to process inputs of different modalities respectively, and then uses a network architecture to perform cross-modal information fusion; the depth image D and the normal vector angular image are regarded as the same modality and spliced ​​into a four-channel spatial image vector, recorded as a DI image; the color information and spatial information are first encoded separately, and then the extracted information is integrated through cross-modal fusion to enhance the semantic representation of the object position and shape of the two, and finally complete the subsequent decoding operation.

[0033] Furthermore, the grasping model in step S2 is constructed based on the improved Swin-Unet, including: the encoding part extracts multiple set scale features of the RGB color image and the spatial image respectively to obtain color features and spatial features; then, the color features and spatial features of the same scale are feature-fused to obtain a plurality of first fused features; then, the first fused features of the minimum set scale are extracted to obtain second fused features; the decoding part predicts and obtains grasping parameters according to the first fused features and the second fused features;

[0034] In step S2, the multi-level structure of the Swin Transformer decoder is used to transform the encoder. The specific logic of the algorithm is as follows:

[0035] Set the hyperparameter C of the input→stage2 part of the original Swin-Unet encoder to 1 / 2 of the original one, and copy the structure to get two sub-encoders;

[0036] The RGB color image and the spatial image obtained by concatenating the depth image and the normal vector angle image are input into the two sub-encoders respectively; after one convolution operation, the data size in the two sub-encoders becomes

[0037] The two sub-encoders complete stage 1 and 2 respectively, and up-sample through two Swin Transformer Blocks. At this time, the feature dimension of the two outputs is 2C. Before performing stage 3 calculation, the data in the two sub-encoders are concatenated to restore the embedding dimension of the current feature map to 4C.

[0038] Continue to complete the calculation of stage3 and bottleneck layer in the encoder;

[0039] In the decoder, the jump connection operation maintains the first layer consistent with the original Swin-Unet structure, and concatenates the outputs of the two same-level sub-encoders and the decoder feature maps of the previous level in the second and third layers, as shown in the following formula:

[0040]

[0041] In the formula, They represent the input of the n-th level decoder and the output of the n-1-th level decoder respectively; and They represent the two-stream sub-encoder outputs corresponding to the n-th level decoder respectively;

[0042] Stage3 is selected as the information fusion position of the dual-stream structure; two sub-encoders based on the Swin-Unet structure are constructed to extract features of multiple set scales of RGB color images and spatial images respectively to obtain color features and spatial features; the image block embedding vector lengths of the two sub-encoders remain the same.

[0043] Furthermore, the process of grabbing parameters in step S2 includes: upsampling the second fused features of the smallest scale to the input resolution step by step; in each level of upsampling operation, fusing the first fused features or the second fused features of the corresponding scale through a jump connection, and then predicting and obtaining the grabbing parameters;

[0044] The grasping parameters include the grasping position, the grasping confidence, the grasping width and the grasping angle of the grasping position; the SU-Grasp network output is a heat map (Q, W, Θ) of the grasping center confidence, the grasping width and the grasping angle; when determining the actual grasping position, the position with the highest grasping confidence is obtained by retrieving the grasping quality heat map Q, which is defined as:

[0045]

[0046] In the formula, represents the grasping position with the highest grasping confidence, that is, the best grasping position recommended by the model; argmax pos It means to find the independent variable that makes the following expression take the maximum value; Q represents the capture confidence heat map;

[0047] Then, the predicted width w and angle θ of the corresponding position are extracted from the predicted heat map of the grasping width W and the grasping angle θ;

[0048] The grasping posture estimation is regarded as a regression problem, that is, a mapping relationship F is established through the SU-Grasp model: Where I represents input information, specifically RGBDI nrm Seven-channel vector, and Prediction heatmap representing the model output (Q, W, Θ);

[0049] Based on minimizing the prediction distance, the loss function is defined as:

[0050]

[0051] In the formula, represents the total loss value; N represents the number of samples; m represents different output indicators; w m Indicates the weight of each predicted heat map loss value, the default value is 1; is the predicted value of the model, which represents the prediction result of the i-th sample under index m; is the true label data, which represents the actual result of the i-th sample under index m; ||····|| represents the norm of the vector, usually the L2 norm, which indicates the gap between the predicted value and the true value.

[0052] Further, step S3 includes: retrieving the grasping position with the highest grasping confidence to obtain the target grasping position; extracting the grasping width and grasping angle of the target grasping position to obtain the target grasping width and target grasping angle; controlling the robotic arm to grasp the target object according to the target grasping position, target grasping width and target grasping angle.

[0053] A robotic arm control system based on improved Swin-Unet, based on the above-mentioned robotic arm control method, includes:

[0054] Input module, obtains RGB image and depth image of the target working scene;

[0055] A processing module converts the depth image into a normal vector angle image, and performs a splicing operation on the depth image and the normal vector angle image to obtain a spatial image;

[0056] Grasping model, using the grasping model to analyze the RGB image and the spatial image to obtain the grasping parameters;

[0057] A control module controls the robot arm to grasp the target object based on the grasping parameters;

[0058] Among them, the grasping model is built based on the improved Swin-Unet, including:

[0059] The encoding part extracts features of multiple set scales of the RGB color image and the spatial image respectively to obtain image color features and spatial features, and fuses the image color features and spatial features of the same scale to obtain a plurality of first fused features, and then extracts features of at least one scale of the first fused features of the minimum set scale to obtain a second fused feature; the decoding part predicts and obtains the capture parameters according to the first fused features and the second fused features.

[0060] The beneficial effects of the present invention are:

[0061] (1) To address the problem of insufficient spatial understanding in existing robotic arm visual grasping algorithms, the present invention introduces a dual-stream encoding mechanism based on sliding window attention and spatial semantic enhancement into a U-shaped network model, and uses two independent sub-encoders to extract multi-scale features of RGB color images and DI spatial images respectively, and realizes cross-modal information fusion through serial splicing; compared with directly inputting seven-channel data, the dual-stream encoder structure can first extract the shallow semantic features of each modality separately in a manner similar to "hierarchical management", and then more efficiently fuse the summarized information. At the same parameter level, it can also reduce the algorithm complexity, realize parallel computing and accelerate training, etc.;

[0062] (2) The present invention divides three-dimensional vision into two semantic modes, namely, color mode and spatial mode. By introducing the normal vector angle image as the prior information of the spatial semantic mode and adding it to the network input, the hidden semantics of the directionality of the object surface is explicitly expressed, so that the network with poor interpretability can directly obtain the depth information, thereby simplifying the network learning task; in addition, the normal vector angle image can clearly reflect the direction change of the edge of the object on the corresponding coordinate axis, and can also quickly point out the spatial position of different objects in conjunction with the depth map, which can solve the occlusion problem in the scene to a certain extent and enhance the accuracy of model prediction;

[0063] (3) The present invention changes the encoder in the Swin-Unet backbone network to a dual-stream structure, solving the problem of insufficient fusion of RGB color and spatial information caused by the introduction of the normal vector angle image. If the RGB color image and the DI spatial image are directly spliced ​​into a seven-channel initial vector and input into the Swin-Unet original network, the network will not be able to distinguish the information of each channel well due to the large differences in the distribution and semantic information of different modal data, resulting in regression of the model effect or even failure to converge. By introducing a dual-stream structure, the two sub-encoders are allowed to perform semantic understanding of the RGB color information and the DI spatial information separately within the modality, which can avoid interference between the two due to premature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the Swin-Unet network architecture diagram of the present invention;

[0065] Figure 2 is a grayscale image of a depth map and a normal vector angle of a certain example of the present invention;

[0066] Figure 3 It is a schematic diagram of the SU-Grasp dual-stream multi-modal fusion strategy of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings of the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Example 1

[0069] This embodiment provides a robot arm control method and system based on improved Swin-Unet, aiming to solve the problems of insufficient use of depth information and weak local and global perception capabilities during three-dimensional visual grasping of the robot arm.

[0070] The architecture of the improved Swin-Unet model used in this method is as follows:

[0071] like Figure 1 As shown in the figure, a four-layer Swin-Unet network is used to construct a grasping posture prediction model framework SU-Grasp based on a three-dimensional visual dual-stream encoding strategy. Depending on the specific situation, Swin-Unet networks with other layers can also be used. The entire network architecture consists of five modules, namely the point cloud preprocessing module, dual-stream encoder, bottleneck layer, decoder, and skip connection layer.

[0072] Swin-Unet network structure: The Swin-Unet model is a U-shaped architecture based on the Swin Transformer model. It consists of an encoder, a bottleneck layer, a decoder, and a jump connection. Among them, the encoder, the bottleneck layer, and the decoder are all built based on the Swin Transformer Block model. This network structure divides the input image into non-overlapping image blocks, which are input into the encoder to learn deep feature representation. After the context features are extracted, they are upsampled by a decoder with an image block expansion layer and connected to the encoder at the same level through a jump connection to gradually restore the spatial resolution of the feature map, which is convenient for pixel-level segmentation prediction. Using the Swin-Unet model as the basic network framework of the SU-Grasp model, local information (such as object contours and long-range connections) and long-range connections (such as the relationship between different visual entities in a cluttered environment) can be obtained simultaneously.

[0073] In the implementation of the SU-Grasp model, in order to achieve end-to-end parallel two-finger gripper grasp prediction, three grasp outputs are connected in parallel to the top of the decoder, representing the grasp confidence Q, grasp width W, and grasp angle Θ, respectively. Each output is a heat map of the same size as the input RGBD image. Using this pixel-level grasp representation, the best grasp posture in the global visual scene can be obtained in one forward propagation, thereby avoiding the need to generate multiple grasp candidate objects and saving computational costs.

[0074] This robot control method based on improved Swin-Unet focuses on two issues: multi-scale feature extraction and efficient fusion of spatial information and color information. For the former, the object detection algorithm Swin-Unet model is adapted for robot grasping tasks, and multi-scale is improved through a multi-level Swin Transformer Block model and a U-shaped structure with jump connections; for the latter, on the one hand, a normal vector angle image is constructed to provide prior information in the scene, and on the other hand, a dual-stream structure is used to realize the extraction and cross-fusion of color information and spatial information.

[0075] The robot arm control method based on improved Swin-Unet includes the following steps:

[0076] Step S0: Obtain the RGBD image of the target working scene, that is, the data containing the RGB color image and the depth image, and use it as the model input. Specifically, the data set is prepared as follows:

[0077] In the robot arm planar grasping task, commonly used datasets include Cornell and Jacquard. The Cornell dataset contains 885 pictures of 244 common objects in life. Each sample shows different postures of different objects and is equipped with grasping box annotations. There are 8019 grasping boxes marked in this dataset, including 5110 feasible grasping boxes and 2909 infeasible grasping boxes. The Jacquard dataset includes: RGB scene images rendered using Blender, depth images obtained through existing stereo vision algorithms, text files with grasping annotations, and binary background mask images that can be used for data enhancement.

[0078] Furthermore, step S0 also includes preprocessing the depth image, which includes a plane segmentation operation based on a random sampling consensus algorithm (RANSAC) and an outlier removal operation based on a radius neighbor search.

[0079] Step S1: converting the depth image into a normal vector angle image, and performing a splicing operation on the depth image and the normal vector angle image to obtain a spatial image.

[0080] During step S1, a normal vector angle image is introduced. The three-dimensional vision is divided into two semantic modes, color mode and spatial mode, and an attempt is made to enhance the information interaction between the two through multimodal fusion methods. Combined with the process of human grasping objects, the spatial mode plays two key roles in the grasping process: first, it helps to determine the overall position and posture of the object to be grasped; second, it provides local geometric information on the surface of the object to assist in deciding the most stable grasping point. In order to avoid letting the network learn the hidden semantics in the depth information by itself, this embodiment directly expresses it explicitly through the normal vector angle image, thereby simplifying the learning task. Therefore, the normal vector angle image is added to the network input as prior information of the spatial semantic mode.

[0081] Further, the Open3D library is called to convert the preprocessed depth map into a normal angle image. To calculate the normal angle image, for each pixel (u, v) in the depth image D, the unit normal vector N (n x ,n y ,n z ). Then, determine the angle between N and the three axes of the camera coordinate system XYZ. Let the three axes XYZ be represented by vectors x(1,0,0), y(0,1,0), and z(0,0,1), then Ν(n x ,n y ,n z ) and the angles between these three axis vectors can be expressed as:

[0082] a x =arccos(N·x)

[0083] a y =arccos(N·y)

[0084] a z =arccos(N·z)

[0085] Where N is the unit normal vector of the object surface corresponding to the current pixel; x, y, z represent the unit normal vectors in the camera coordinate system that are in the same direction as the positive direction of the three coordinate axes; a x 、a y 、a z Corresponding to the angle between N and the x, y, and z axes; the arccos function is used to calculate the cosine of the angle between two vectors.

[0086] Finally, the angle (a x ,a y ,a z ) is normalized to the range of 0 to 255 and used to create a three-channel normal vector angle image I nrm , then each pixel has a normalized (a x ,a y ,az ) as its value.

[0087] Figure 2 This is the grayscale visualization result of the depth map and normal vector angle of a certain example, where Figure a is the original scene image and Figure b is a x Single channel grayscale image, Figure C is a z Single channel grayscale image, Figure d is a y Single channel grayscale image. It can be found that the normal vector angle image can clearly reflect the direction change of the object edge on the corresponding coordinate axis. In combination with the depth map, it can also quickly point out the spatial position of different objects, which can solve the occlusion problem in the scene to a certain extent. The ablation experiment proves that the normal vector angle image can effectively accelerate the model convergence process and help the model better understand the spatial features.

[0088] In order to more efficiently fuse RGB color information and depth information, the encoder part of the Swin-Unet model in this embodiment is innovatively adjusted.

[0089] First, the normal vector angle image is introduced into the robot arm grasping task to explicitly express the hidden semantics of the depth information on the surface direction of the object. Secondly, the depth image and RGB image are regarded as two modalities. The former describes the depth difference and structural information between different objects, while the latter reflects the appearance information such as color, shape and boundary of a scene. Therefore, this embodiment adopts the dual-stream structure which is widely used in the field of salient object detection to explore the advantages of different modalities and their complementary properties.

[0090] The dual-stream structure refers to a network architecture that uses two encoders to process inputs of different modalities respectively, and then fuses information through a cross-modal method. Considering the depth image D and the normal vector angle image I nrm Because of the semantic similarity in representing spatial information, the two are regarded as the same modality and spliced ​​into a four-channel spatial image vector (denoted as DI image). The color information and spatial information are first encoded separately, and then the extracted information is integrated through the cross-modal fusion method to enhance the semantic representation of the object position and shape of the two, and finally complete the subsequent decoding operation.

[0091] Step S2: Analyze the RGB color image and the spatial image using the grasping model to obtain grasping parameters. The RGB two-dimensional image is regarded as a color semantic modality, and the depth map D and the normal angle image I are regarded as spatial semantic modalities as inputs to the SU-Grasp network.

[0092] The grasping model is based on the improved Swin-Unet, including:

[0093] The encoding part extracts multiple set scale features of the RGB color image and the spatial image respectively to obtain color features and spatial features. Then, the color features and spatial features of the same scale are fused to obtain a number of first fused features. Then, the first fused features of the minimum set scale are extracted to obtain the second fused features. The decoding part predicts the capture parameters based on the first fused features and the second fused features.

[0094] Furthermore, the process of predicting the grasping parameters includes: upsampling the second fused features of the smallest scale to the input resolution step by step. In each level of upsampling operation, the first fused features or the second fused features of the corresponding scale are fused through a jump connection to predict the grasping parameters.

[0095] Considering that SU-Grasp is a hierarchical U-shaped network, and after comprehensive consideration of the algorithm complexity and code implementation difficulty, the multi-level structure of the Swin Transformer decoder is used to innovatively transform the current encoder, such as Figure 3 As shown, the specific logic of the algorithm is as follows:

[0096] (1) Set the hyperparameter C (i.e., the length of each image block embedding vector) in the “input→stage2” part of the original Swin-Unet encoder to 1 / 2 of the original one, and copy the structure to obtain two sub-encoders, as shown in Figure 3 Shown in the red box.

[0097] (2) Input the RGB color image and the DI spatial image into the two sub-encoders respectively. After one convolution operation, the data size in the two sub-encoders becomes

[0098] (3) The two sub-encoders complete stage 1 and 2 respectively, and up-sample through two Swin Transformer Blocks. At this time, the feature dimension of the two outputs is 2C. Before performing stage 3 calculation, the data in the two sub-encoders are concatenated to restore the embedding dimension of the current feature map to 4C.

[0099] (4) Continue to complete the calculation of stage3 and bottleneck layer in the encoder.

[0100] (5) In the decoder, the skip connection operation maintains the first layer consistent with the original Swin-Unet structure. At the same time, the outputs of the two sub-encoders of the same level and the feature maps of the decoder of the previous level need to be concatenated in the second and third layers, as shown in the following formula:

[0101]

[0102] In the formula, They represent the input of the n-th level decoder and the output of the n-1-th level decoder respectively; and They respectively represent the two-stream sub-encoder outputs corresponding to the n-th level decoder.

[0103] Figure 3 The scheme shown is to select stage3 as the information fusion position of the dual-stream structure. Of course, the cross-modal timing can also be advanced or delayed as needed, thanks to the hierarchical structure of the Swin-Unet model. Furthermore, the color features and spatial features are obtained by extracting features of multiple set scales of RGB color images and spatial images respectively, which is achieved by constructing two sub-encoders based on the Swin-Unet structure. In addition, the image block embedding vector lengths of the two sub-encoders remain the same.

[0104] Furthermore, the grasping parameters include the grasping position, the grasping confidence, the grasping width and the grasping angle of the grasping position. The SU-Grasp network output is a heat map (Q, W, Θ) of the grasping center confidence, the grasping width and the grasping angle. When determining the actual grasping position, the position with the highest grasping confidence can be obtained by retrieving the grasping quality heat map Q, which is defined as:

[0105]

[0106] In the formula, represents the grasping position with the highest grasping confidence, that is, the best grasping position recommended by the model; argmax pos It means to find the independent variable that makes the following expression take the maximum value; Q represents the capture confidence heat map.

[0107] Then, the predicted width w and angle θ of the corresponding position are extracted from the predicted heat map of grasping width W and grasping angle θ.

[0108] In this embodiment, grasping posture estimation is regarded as a regression problem, that is, a mapping relationship F is established through the SU-Grasp model: Where I represents input information, specifically RGBDI nrm Seven-channel vector, and A prediction heatmap representing the output of the model (Q, W, Θ).

[0109] Based on the idea of ​​minimizing the prediction distance, the loss function is defined as:

[0110]

[0111] In the formula, represents the total loss value; N represents the number of samples; m represents three different output indicators: grasping confidence, grasping width, and grasping rotation angle; w m Indicates the weight of each predicted heat map loss value, the default value is 1; is the predicted value of the model, which represents the prediction result of the i-th sample under index m; is the true label data, which represents the actual result of the i-th sample under index m; ||····|| represents the norm of the vector, usually the L2 norm, which indicates the gap between the predicted value and the true value.

[0112] Step S3: Controlling the robotic arm to grasp the target object based on the grasping parameters.

[0113] Further, step S3 includes:

[0114] The grasping position with the highest grasping confidence is retrieved to obtain the target grasping position; the grasping width and grasping angle of the target grasping position are extracted to obtain the target grasping width and the target grasping angle; the robot arm is controlled to grasp the target object according to the target grasping position, the target grasping width and the target grasping angle.

[0115] In view of the problem of insufficient spatial understanding ability in the existing robot visual grasping algorithm, this embodiment proposes a robot control method based on improved Swin-Unet. This method introduces a dual-stream encoding mechanism based on sliding window attention and spatial semantic enhancement into the U-type network model. The accuracy rates on the Cornell and Jacquard datasets reached 98.1% and 95.2% respectively, which is better than the existing algorithms in terms of training efficiency and training time, reflecting that the improved algorithm has stronger learning ability in spatial understanding. After simulation testing, it was found that the model can still accurately determine the location information of the current significant object and complete the grasping when facing a multi-object cluttered scene, with an accuracy rate of 83%.

[0116] Example 2

[0117] Based on the improved Swin-Unet-based robotic arm control method provided in the embodiment, this embodiment proposes a robotic arm control system based on the improved Swin-Unet, including:

[0118] Input module, obtains RGB image and depth image of the target working scene;

[0119] A processing module converts the depth image into a normal vector angle image, and performs a splicing operation on the depth image and the normal vector angle image to obtain a spatial image;

[0120] Grasping model, using the grasping model to analyze the RGB image and the spatial image to obtain the grasping parameters;

[0121] A control module controls the robot arm to grasp the target object based on the grasping parameters;

[0122] Among them, the grasping model is built based on the improved Swin-Unet, including:

[0123] The encoding part extracts features of the RGB color image and the spatial image at multiple set scales to obtain image color features and spatial features, and fuses the image color features and spatial features at the same scale to obtain a plurality of first fused features, and then extracts features of at least one scale from the first fused features at the minimum set scale to obtain a second fused feature;

[0124] In the decoding part, the capture parameters are predicted based on the first fusion feature and the second fusion feature.

[0125] In summary, SU-Grasp uses independent encoders to extract multi-scale features of RGB color images and DI spatial images respectively, and realizes cross-modal information fusion through serial splicing. Compared with directly inputting seven-channel data, the dual-stream encoder structure can first extract the shallow semantic features of each modality separately through a similar "hierarchical management" method, and then more efficiently fuse the summarized information. At the same parameter level, it can also reduce the complexity of the algorithm, realize parallel computing and accelerate training.

[0126] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A robotic arm control method based on improved Swin-Unet, characterized in that: The improved Swin-Unet model architecture adopted in the robot control method based on the improved Swin-Unet is used to construct the grasping posture prediction model framework SU-Grasp based on the three-dimensional visual dual-stream encoding strategy based on the four-layer Swin-Unet network; the entire network architecture contains 5 modules, namely the point cloud preprocessing module, the dual-stream encoder, the bottleneck layer, the decoder and the skip connection layer; The robot arm control method based on the improved Swin-Unet includes the following steps: Step S0: Obtain an RGBD image of the target working scene, that is, data containing an RGB color image and a depth image, and use it as a model input; Step S1: converting the depth image into a normal vector angle image, and performing a splicing operation on the depth image and the normal vector angle image to obtain a spatial image; Step S2: Analyze the RGB color image and the spatial image using the grasping model to obtain grasping parameters; regard the RGB two-dimensional image as a color semantic modality, and regard the depth map D and the normal angle image I as a spatial semantic modality as inputs to the SU-Grasp network; Step S3: Controlling the robotic arm to grasp the target object based on the grasping parameters.

2. The robot arm control method based on improved Swin-Unet according to claim 1, characterized in that: In the architecture of the Swin-Unet model, the encoder, bottleneck layer, and decoder are built based on the Swin Transformer Block model; the Swin-Unet network structure divides the input image into non-overlapping image blocks, which are input to the encoder to learn deep feature representation; After the context features are extracted, they are upsampled by a decoder with an image block expansion layer and connected to the encoder at the same level through a skip connection to gradually restore the spatial resolution of the feature map and perform pixel-level segmentation prediction. The Swin-Unet model is used as the basic network framework of the SU-Grasp model to obtain local information and long-range connections at the same time; In the implementation of the SU-Grasp model, three grasping outputs are connected in parallel to the top of the decoder, representing the grasping confidence Q, grasping width W, and grasping angle Θ, respectively. Each output is a heat map of the same size as the input RGBD image.

3. The robot arm control method based on improved Swin-Unet according to claim 1, characterized in that: In step S0, the data includes a Cornell dataset and a Jacquard dataset; step S0 also includes preprocessing the depth image, and the preprocessing includes a plane segmentation operation based on a random sampling consensus algorithm RANSAC and an outlier removal operation based on a nearest neighbor search within a radius.

4. The robot arm control method based on improved Swin-Unet according to claim 1, characterized in that: In step S1, the normal vector angle image is introduced to divide the three-dimensional vision into two semantic modes: color mode and spatial mode, and the information interaction between the two is enhanced through the multimodal fusion method; in the spatial mode, during the grasping process: first, the overall position and posture of the object to be grasped is determined; secondly, the local geometric information of the object surface is provided to assist in deciding the most stable grasping point.

5. The robot arm control method based on improved Swin-Unet according to claim 1, characterized in that: In step S1, the Open3D library is called to convert the preprocessed depth map into a normal vector angle image; the calculation process of the normal vector angle image is as follows: For each pixel (u, v) in the depth image D, calculate the unit normal vector N(n x ,n y ,n z ); Determine the angle between N and the three axes of the camera coordinate system XYZ; Assume that the three axes XYZ are represented by vectors x(1,0,0), y(0,1,0), z(0,0,1), Ν(n x ,n y ,n z ) and the angles between these three axis vectors can be expressed as: a x arccos(N·x) a y arccos(N·y) a z arccos(N·z) Where N is the unit normal vector of the object surface corresponding to the current pixel; x, y, z represent the unit normal vectors in the camera coordinate system that are in the same direction as the positive direction of the three coordinate axes; a x 、a y 、a z Corresponding to the angle between N and the x, y, and z axes; the arccos function is used to calculate the cosine of the angle between two vectors. The angle (a x ,a y ,a z ) is normalized to the range of 0 to 255 and used to create a three-channel normal vector angle image I nrm , then each pixel has a normalized (a x ,a y ,a z ) as its value.

6. The robot arm control method based on improved Swin-Unet according to claim 1, characterized in that: In step S1, the normal vector angle image is introduced into the robot grasping task to explicitly express the hidden semantics of the depth information on the surface direction of the object; the depth image and the RGB image are regarded as two modalities, the former describes the depth difference and structural information between different objects, and the latter reflects the appearance information including the color, shape and boundary of a scene; a two-stream structure is used to explore the advantages of different modalities and their complementary properties; The dual-stream structure uses two encoders to process inputs of different modalities respectively, and then uses a network architecture to perform cross-modal information fusion; the depth image D and the normal vector angular image are regarded as the same modality and spliced ​​into a four-channel spatial image vector, recorded as a DI image; the color information and spatial information are first encoded separately, and then the extracted information is integrated through cross-modal fusion to enhance the semantic representation of the object position and shape of the two, and finally complete the subsequent decoding operation.

7. The robot arm control method based on improved Swin-Unet according to claim 6, characterized in that: The grasping model in step S2 is constructed based on the improved Swin-Unet, including: the encoding part extracts multiple set scale features of the RGB color image and the spatial image respectively to obtain color features and spatial features; then, the color features and spatial features of the same scale are feature-fused to obtain a number of first fused features; then, the first fused features of the minimum set scale are extracted to obtain the second fused features; the decoding part predicts the grasping parameters according to the first fused features and the second fused features; In step S2, the multi-level structure of the Swin Transformer decoder is used to transform the encoder. The specific logic of the algorithm is as follows: Set the hyperparameter C of the input→stage2 part of the original Swin-Unet encoder to 1 / 2 of the original one, and copy the structure to get two sub-encoders; The RGB color image and the spatial image obtained by concatenating the depth image and the normal vector angle image are input into the two sub-encoders respectively; after one convolution operation, the data size in the two sub-encoders becomes The two sub-encoders complete stage 1 and 2 respectively, and up-sample through two Swin Transformer Blocks. At this time, the feature dimension of the two outputs is 2C. Before performing stage 3 calculation, the data in the two sub-encoders are concatenated to restore the embedding dimension of the current feature map to 4C. Continue to complete the calculation of stage3 and bottleneck layer in the encoder; In the decoder, the jump connection operation maintains the first layer consistent with the original Swin-Unet structure, and concatenates the outputs of the two same-level sub-encoders and the decoder feature maps of the previous level in the second and third layers, as shown in the following formula: In the formula, They represent the input of the n-th level decoder and the output of the n-1-th level decoder respectively; and They represent the two-stream sub-encoder outputs corresponding to the n-th level decoder respectively; Stage3 is selected as the information fusion position of the dual-stream structure; two sub-encoders based on the Swin-Unet structure are constructed to extract features of multiple set scales of RGB color images and spatial images respectively to obtain color features and spatial features; the image block embedding vector lengths of the two sub-encoders remain the same.

8. The robot arm control method based on improved Swin-Unet according to claim 7, characterized in that: The process of grabbing parameters in step S2 includes: upsampling the second fused features of the smallest scale to the input resolution step by step; in each level of upsampling operation, fusing the first fused features or the second fused features of the corresponding scale through a jump connection, and then predicting and obtaining the grabbing parameters; The grasping parameters include the grasping position, the grasping confidence, the grasping width and the grasping angle of the grasping position; the SU-Grasp network output is a heat map (Q, W, Θ) of the grasping center confidence, the grasping width and the grasping angle; when determining the actual grasping position, the position with the highest grasping confidence is obtained by retrieving the grasping quality heat map Q, which is defined as: In the formula, represents the grasping position with the highest grasping confidence, that is, the best grasping position recommended by the model; argmax pos It means to find the independent variable that makes the following expression take the maximum value; Q represents the capture confidence heat map; Then, the predicted width w and angle θ of the corresponding position are extracted from the predicted heat map of the grasping width W and the grasping angle θ; The grasping posture estimation is regarded as a regression problem, that is, a mapping relationship is established through the SU-Grasp model. Where I represents input information, specifically RGBDI nrm Seven-channel vector, and Prediction heatmap representing the model output (Q, W, Θ); Based on minimizing the prediction distance, the loss function is defined as: In the formula, represents the total loss value; N represents the number of samples; m represents three different output indicators: grasping confidence, grasping width, and grasping rotation angle; w m Indicates the weight of each predicted heat map loss value, the default value is 1; is the predicted value of the model, which represents the prediction result of the i-th sample under index m; is the true label data, indicating the actual result of the i-th sample under the indicator m; ||....|| indicates the norm of the vector, usually the L2 norm, indicating the gap between the predicted value and the true value.

9. The robot arm control method based on improved Swin-Unet according to claim 7, characterized in that: Step S3 includes: retrieving the grasping position with the highest grasping confidence to obtain the target grasping position; extracting the grasping width and grasping angle of the target grasping position to obtain the target grasping width and target grasping angle; controlling the robot arm to grasp the target object according to the target grasping position, target grasping width and target grasping angle.

10. A robotic arm control system based on improved Swin-Unet, based on the robotic arm control method according to any one of claims 1 to 9, comprising: Input module, obtains RGB image and depth image of the target working scene; A processing module converts the depth image into a normal vector angle image, and performs a splicing operation on the depth image and the normal vector angle image to obtain a spatial image; Grasping model, using the grasping model to analyze the RGB image and the spatial image to obtain the grasping parameters; A control module controls the robot arm to grasp the target object based on the grasping parameters; Among them, the grasping model is built based on the improved Swin-Unet, including: The encoding part extracts features of multiple set scales of the RGB color image and the spatial image respectively to obtain image color features and spatial features, and fuses the image color features and spatial features of the same scale to obtain a plurality of first fused features, and then extracts features of at least one scale of the first fused features of the minimum set scale to obtain a second fused feature; the decoding part predicts and obtains the capture parameters according to the first fused features and the second fused features.

Images