Robot grasping detection method based on visual language action multi-modal alignment strategy

Abstract

This invention belongs to the field of robot grasping control technology and discloses a robot grasping detection method based on a multimodal alignment strategy of visual language and action. The method includes the following steps: inputting natural language commands into a large language model to generate three categories of descriptions—object category, task, and grasping action—along with attention masks; performing channel attention optimization on the three types of data through a text encoder to obtain text encoding features; inputting 3D point cloud data of the object to generate point cloud encoding features; inputting the point cloud data into a candidate grasping pose encoding module to extract grasping pose encoding features; achieving multimodal feature fusion through a cross-attention module to obtain fused features; inputting the fused features into a strategy module to obtain an evaluation value for each candidate grasping action; and ranking all candidate grasping actions according to the evaluation values. This invention solves the multi-action decision-making problem of task-oriented robot grasping by aligning and fusing point clouds, language text, and grasping poses.

Description

Technical Field

[0001] This invention relates to the field of robot grasping control technology, and in particular to a robot grasping detection method based on a visual-language-action multimodal alignment strategy. Background Technology

[0002] With the rapid development of robotics technology, people are placing greater expectations on home robots. They are expected not only to perform simple picking operations but also to understand and execute complex task instructions. For example, people hope that robots can assist in tasks requiring a high degree of autonomy, such as tool use, item sorting, and other tasks, based on commands. This leads to the need for robots to grasp tools in a task-oriented manner, i.e., task-oriented grasping. The core of this task is that the robot not only needs to accurately perform the grasping action but also needs to understand the purpose of the tool's use and its relationship to the task instructions.

[0003] In task-oriented grasping tasks, robots need a deep understanding of instructions and the ability to learn the relationship between tasks and objects from limited examples. Furthermore, they must generalize the learned knowledge to new tasks, placing high demands on the robot's understanding and adaptability. However, existing task-oriented grasping methods have shortcomings when dealing with real-world tasks. First, early methods often relied on carefully designed scenes or predefined action dictionaries to simulate human grasping behavior. However, these methods are only applicable to specific closed scenarios and struggle to meet the complex and varied task requirements of open worlds, significantly limiting their practical application. Inspired by human natural language processing capabilities, researchers have gradually combined language text with visual information to enhance robots' ability to perceive complex environments. However, most methods achieve visual-language fusion through modules such as CLIP, but point cloud data and motion data, crucial for grasping tasks, are often neglected, lacking truly usable multimodal fusion strategies. Second, existing methods often directly stitch point cloud and motion data together and then encode and fuse them with language data. However, this method fails to effectively address the imbalance in the distribution of point cloud and motion data, and it cannot fully explore the intrinsic relationships between multimodal data. Finally, the current method mainly employs two types of grasping strategies: single-action decision-making and grasping part segmentation. The former focuses only on a single grasping action, which is prone to task interruption due to planning failures in real-world scenarios. The latter avoids the shortcomings of single-action decision-making by first segmenting the object's grasping parts and then performing grasping detection to obtain multiple candidate actions. However, this method cannot incorporate task instructions or work scenario constraints into the grasping strategy, lacking task orientation. Summary of the Invention

[0004] To address the shortcomings of existing technologies that fail to incorporate task instructions or work scenario constraints into grasping strategies, resulting in a lack of task orientation, this invention proposes a robot grasping and detection method based on a visual-language-action multimodal alignment strategy. By constructing a multimodal feature alignment and multi-action decision framework, this method enhances the robot's generalization ability and grasping success rate in open worlds.

[0005] This invention is achieved through the following technical solution, including the following steps:

[0006] S1. Input the natural language command into the pre-trained large language model LLM for parsing, generate three types of descriptions: object category description, task description, and grasping description. Then, input the three types of descriptions and the natural language command into the pre-trained BERT model for processing to obtain high-dimensional text features and attention masks.

[0007] S2. Input the 3D point cloud data of the object, extract geometric and structural features through a pre-trained PointNet++ point cloud encoder, and generate a point cloud embedding F. p ;

[0008] S3. Input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasping poses g i ; select candidate grabbing postures g i Expand into action feature X g And extract the grasping features F through a multilayer perceptron (MLP). g ;

[0009] S4. Perform channel attention optimization on the three types of descriptions and natural language instructions obtained in step S1 using a text encoder model to obtain the language text features F. txt ;

[0010] S5. Based on the grasping characteristics F g A modal alignment module composed of a multilayer sensing mechanism is used to perform point cloud feature F p and language text features F txt Perform dimension alignment to obtain multimodal features;

[0011] S6. Based on the cross-attention mechanism, the action features F g Set as Query, point cloud feature F p Set as Key, Language Text Feature F txt Set to the value Value, and use the cross-attention module to fuse multimodal features to obtain the fused feature F. fusion ;

[0012] S7. The fusion feature F obtained in step S6 is... fusionThe input strategy module extracts important information about the grasping features through a multilayer perceptron (MLP) and obtains the evaluation value S of each candidate grasping action through a sigmoid activation function.

[0013] S8. Based on the evaluation value S of each candidate grasping action obtained in step S7, sort all candidate grasping actions, output the sorted action list, control the robot to execute the optimal grasping action, and if the grasping fails, backtrack to the suboptimal action.

[0014] As a further preferred embodiment, the pre-trained PointNet++ point cloud encoder in step S2 includes a multi-scale point cloud feature extraction layer (MSG) and a feature fusion layer (SA), and the formula for generating the point cloud embedding is F. p =SA(MSG(X) p ,dim=0)), through formula F p =Align(squeeze(F) p (dim=1), dim=1, num=N) embedding F into the point cloud p Expand upon this.

[0015] As a further preferred option, the specific steps of step S3 are as follows:

[0016] S31. For each homogeneous [R,T] matrix of size 4×4, select the candidate grasping postures g. i Expanded into one-dimensional feature X g = [x,y,z,q0,q1,q2,q3];

[0017] Where x, y, z represent the translation relationship, and x, y, z = T tr ; q0, q1, q2, q3 represent the quaternions of the rotation matrix;

[0018] S32, The unfolded grasping posture feature X g Inputting a grasping encoder composed of a multilayer perceptron (MLP) yields F. g The formula is as follows:

[0019] F g =MLP(X g dim=3)

[0020] In the formula, X g This refers to the movement characteristics after unfolding.

[0021] As a further preferred option, the specific steps of step S4 are as follows:

[0022] S41. The text encoder model encodes the three types of descriptions and natural language instructions obtained in step S1 to generate a task description paragraph. Object class description paragraph Extract description paragraphs and language instructions Word embedding;

[0023] S42. The channel attention mechanism (SE) module is used to weight the text features to obtain the language text features F. txt The formula is as follows:

[0024] F txt =SE(stack(F) td ,F od ,F gd ,F I ,dim=1))

[0025] Among them, stack will F td F od F gd F I The features are concatenated into a unified feature representation for multiple channels;

[0026] S43, Regarding the fused linguistic features F txt To perform a channel copy operation, use the following formula:

[0027] F txt =Align(squeeze(F) txt (dim=1), dim=1, num=N)

[0028] Where N represents the candidate grasping pose set g i In terms of the number of grasping gestures, squeeze removes redundant dimensions from language features before copying them by channel.

[0029] As a further preferred option, the fusion feature F obtained in step S6 is... fusion The formula is as follows:

[0030] F fusion = cross_attention(Q=F g K = F p V = F txt )

[0031] Among them, action feature F g To query Q, point cloud feature F p Let K be the key and F be the language text features. txt Let V be the value. As a further optimization, the formula for obtaining the evaluation value S for each candidate grasping action is as follows:

[0032] S = Sigmoid(MLP(F) fusion ), dim = -1).

[0033] As a further preferred option, the point cloud embedding in step S2

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] 1. This invention achieves multimodal alignment and fusion of point clouds, speech text, and grasping actions, fully explores the intrinsic relationship between modalities, and solves the problem of insufficient multimodal data fusion in existing methods;

[0036] 2. This invention designs a task-oriented grasping strategy based on multi-action sequencing, which improves the grasping efficiency and robustness of the robot in complex task scenarios and avoids task interruption caused by the failure of a single action planning.

[0037] 3. Provides a semantically enhanced data organization method, which uses the semantic knowledge generated by a large language model to guide the robot to dynamically adjust its grasping strategy and enhance task orientation. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the task-oriented capture and detection process in the method of the present invention.

[0039] Figure 2 This is a flowchart of the language feature encoding process in the method of the present invention.

[0040] Figure 3 This is a block diagram of the multimodal feature alignment and fusion module in the method of the present invention.

[0041] Figure 4 This is a schematic diagram of a real robot comparison experiment for task-oriented grasping in the method of the present invention. Detailed Implementation

[0042] The advantages and features of the present invention will be illustrated and explained by the following non-limiting description of preferred embodiments, which are given by way of example only with reference to the accompanying drawings.

[0043] like Figure 1 As shown, this invention provides a robot grasping and detection method based on a visual language action multimodal alignment strategy, comprising the following steps:

[0044] Step 1: Input the natural language command into the pre-trained Large Language Model (LLM) for parsing, generating three types of descriptions: object category description, task description, and grasping description. Then, input the three types of descriptions along with the natural language command into the pre-trained BERT model for processing to obtain high-dimensional text features and attention masks.

[0045] The process involves three parts: object category description, which extracts target object category information from natural language instructions, such as "tool" or "bottle"; task description, which parses semantic knowledge of the task objective, such as "cut" or "pour water"; and grasping description, which generates grasping action information related to the task, such as "focus on the handle of the knife". These three descriptions, along with the natural language instructions, are then fed into a pre-trained text encoder for processing, generating high-dimensional text features for subsequent multimodal fusion.

[0046] Step 2: Input the 3D point cloud data of the object, extract geometric and structural features using a pre-trained PointNet++ point cloud encoder, and generate a point cloud embedding. The F p The point cloud features are used. The pre-trained PointNet++ point cloud encoder includes a multi-scale point cloud feature extraction layer (MSG) and a feature fusion layer (SA). The formula for generating point cloud embeddings is as follows:

[0047]

[0048] Then, a point cloud alignment strategy is used to embed F. p To expand the data, it will be copied along the channel dimension to align it with the candidate grabbing pose data. The alignment formula is as follows:

[0049] F p =Align(squeeze(F) p (dim=1), dim=1, num=N)

[0050] Where N is the candidate grasping posture set g i The number of grasping gestures in the middle, squeeze is F p Adding dimensions, Align replicates by channel dimension.

[0051] Step 3: Input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasping poses g i ; select candidate grabbing postures g i Expand into action feature X g And extract the grasping features F through a multilayer perceptron (MLP). g .

[0052] Step 31: For each 4×4 homogeneous matrix [R,T], select the candidate grasping posture g. i Expanded into one-dimensional feature X g = [x,y,z,q0,q1,q2,q3];

[0053] Where x, y, z represent the translation relationship, and x, y, z = T tr ; q0, q1, q2, q3 represent the quaternions of the rotation matrix.

[0054] Step 32: Extract the unfolded grasping posture feature X g Inputting a grasping encoder composed of a multilayer perceptron (MLP) yields F. g The formula is as follows:

[0055] F g =MLP(X g dim=3)

[0056] Unlike previous methods that directly stitched together point cloud data and grasped data, this invention focuses on the grasping posture g. i Separate processing is performed to better align point cloud data and text features. Specifically, each candidate grasping pose is first expanded into a 4×4 homogeneous [R,T] matrix into a one-dimensional feature X. g = [x,y,z,q0,q1,q2,q3]. Where x,y,z represent the translation relationships, as follows:

[0057] x,y,z=T tr

[0058] Where q0, q1, q2, q3 represent the quaternions of the rotation matrix. The unfolded grasping posture features X... g The data is fed into a grasping encoder composed of a multilayer perceptron (MLP) to obtain F. g :

[0059] F g =MLP(X g dim=3)

[0060] Feature extraction using MLP ensures F g The middle channel dimension remains unchanged, which facilitates the subsequent feature fusion process.

[0061] Step 4: Perform attention masking and channel attention optimization on the three types of descriptions and natural language instructions obtained in Step 1 using a text encoder model to obtain the language text features F. txt .

[0062] Step 41: The text encoder model encodes the three types of descriptions and natural language instructions obtained in Step 1 to generate a task description paragraph. Object class description paragraph Extract description paragraphs and language instructions The word embeddings are defined by td, od, gd, and I, where td, od, gd, and I represent the maximum length of each language sequence (zero padding). The language encoder is frozen during training.

[0063] Step 42: Use the channel attention mechanism (SE) module to weight the text features to obtain the language text features F. txt The formula is as follows:

[0064] F txt =SE(stack(F) td ,F od ,F gd ,F I ,dim=1))

[0065] Among them, stack will F td F od F gd F I The features are concatenated into a unified feature representation across multiple channels. The SE module dynamically adjusts the contribution of each text feature in the fusion process by learning importance weights.

[0066] Step 43: Analyze the fused language features F txt To perform a channel copy operation, use the following formula:

[0067] F txt =Align(squeeze(F) txt (dim=1), dim=1, num=N)

[0068] Where N represents the candidate grasping pose set g i In terms of the number of grasping gestures, squeeze removes redundant dimensions from language features before copying them by channel.

[0069] like Figure 2 As shown, in order to effectively link the three concepts of task, object and grasping posture, this invention uses a pre-trained BERT model to parse natural language instructions, generate multiple semantic text features as input, and optimize feature fusion through a channel attention mechanism.

[0070] By feeding language instructions into a BERT model pre-trained on a large-scale text data corpus, there is no need to train the encoder from scratch, thus avoiding a significant consumption of time and computing resources.

[0071] To fully utilize multiple semantic text features as semantic knowledge, a channel attention mechanism (SE) module is used to weight the text features. This facilitates integration with point cloud features F. p and capturing features F g Alignment processing, for the fused language text features F txt Perform channel copying. Through the above steps, the generated language text features F txt Able to combine point cloud features F p and capturing features F gMaintain consistency to ensure the alignment and effectiveness of multimodal features in subsequent fusion processes.

[0072] Step 5: Based on the grasping feature F g A modal alignment module composed of a multilayer sensing mechanism is used to perform point cloud feature F p and language text features F txt Perform dimension alignment to obtain multimodal features.

[0073] Step 6: Based on the cross-attention mechanism, assign action features F g Set as Query, point cloud feature F p Set as Key, Language Text Feature F txt Set to the value Value, and use the cross-attention module to fuse multimodal features to obtain the fused feature F. fusion The formula is as follows:

[0074] F fusion = cross_attention(Q=F g K = F p V = F txt )

[0075] Among them, action feature F g To query Q, point cloud feature F p Let K be the key and F be the language text features. txt Let the value be V.

[0076] like Figure 3 As shown, the point cloud features F are processed through a cross-attention mechanism. p Language text features F txt and action characteristics F g Dynamic fusion is performed to achieve deep connections and feature alignment between multimodal functions. The cross-attention formula is as follows:

[0077]

[0078] First, the action feature F g Set as Query, point cloud feature F p Set as Key, Language Text Feature F txt Set to Value. These three feature types are consistent in channels and dimensions to ensure that multimodal features can be aligned accordingly.

[0079] Then, the three types of features are input into the cross-attention module to calculate the fused cross-attention feature representation:

[0080] F fusion = cross_attention(Q=F g K = Fp V = F txt )

[0081] This module aligns multiple grasping poses with the semantics of a single object and text, obtaining multimodal fusion features F. fusion .

[0082] Through the above steps, this invention realizes point cloud feature F p Language text features F txt and action characteristics F g Dynamic fusion of multimodal features provides fundamental support for task-oriented grasping action planning.

[0083] Step 7: Combine the fused features F obtained in Step 6. fusion The input strategy module extracts important information about the grasping features through a multilayer perceptron (MLP) and obtains the evaluation value S of each candidate grasping action through a sigmoid activation function.

[0084] The cross-attention features are input into the policy module, and all candidate grasping actions are evaluated based on the fused multimodal features. Specifically, the fused features F fusion The input consists of a policy module comprised of three fully connected MLP layers, each containing a one-dimensional batch normalization (BatchNorm) and ReLU activation function to extract key information from the grasping features. The evaluation value S for each candidate grasping action is calculated using the sigmoid activation function.

[0085] S = Sigmoid(MLP(F) fusion ),dim=-1)

[0086] Step 8: Based on the evaluation value S of each candidate grasping action obtained in Step 7, sort all candidate grasping actions, output the sorted action list, control the robot to execute the optimal grasping action, and if the grasping fails, backtrack to the second-best action.

[0087] Example 1

[0088] This embodiment uses the instruction "use the cup to drink." as an example to describe and explain the detection method of the present invention.

[0089] Step 1: Input the natural language command "use the cup to drink." into the pre-trained Large Language Model (LLM) for parsing. This example uses the OpenAI GPT-3.5-turbo Large Language Model to output the following three key descriptions.Object category description: "A household cup typically has a cylindrical shape with acircularbase and a uniform cross-section along its height, making it a three-dimensional object. Its geometric properties can be described in terms of its dimensions such as diameter, height, and volume, and is often used inmathematics to study concepts related to solid geometry and measurements.", Task description: "Household objects, such as cups, glasses, and mugs, provide a physical container for holding various liquids, facilitating the intake of beverages for hydration and nourishment. Additionally, items like straws and lids helpcontrol the flow of liquid and maintain the temperature of drinks to optimize the drinking experience.", Grab description: "The handle is the most suitable part for a robot hand to grasp as it is designed specifically forgripping. Additionally, gripping the rim of the cup may also provide astablegrip for the robot hand due to the relatively flat and sturdy surface." The three types of descriptions mentioned above, along with natural language instructions, are fed into the frozen BERT model to obtain descriptive features and attention masks.

[0090] Step 2: Input the RGBD image acquired by the depth camera, and output 3D point cloud data with a shape of [4095, 3] using a point cloud sampling generation algorithm. Send the point cloud data to the PointNet++ module, which consists of two multi-scale point cloud feature extraction layers (MSG) and one feature fusion layer (SA), to obtain the point cloud features F. p The shape is [1,768].

[0091] Step 3: Generate a set of candidate grasping poses g from the point cloud data using the pre-trained FGC-GraspNet module. i The shape is [25,4,4], representing 25 candidate grasping actions. The candidate grasping postures g... i Expand into action feature X g The shape is [25, 1, 7]. The input is a grasping encoder composed of a multilayer perceptron (MLP) to generate grasping features F. g The shape is [25,768].

[0092] Step 4: Using the text encoder model, process the three types of descriptions and natural language instructions obtained in Step 1 to output the following features: Task description paragraph Object class description paragraph Extract description paragraphs and language instructions The word embeddings, which consist of descriptive features and attention masks, are processed by BERT. The descriptive features and attention masks from the BERT-processed text word embeddings are then fed into a max-pooling module to remove text padding, allowing the model to focus more on relevant information. Subsequently, the four sets of text features are fed into a channel attention module to obtain the language text features F. txt The shape is [1,768].

[0093] Step 5: Based on the action characteristics F g For point cloud features F p and language text features F txt Perform dimension alignment to obtain three sets of multimodal features with shape [25, 768].

[0094] Step 6: Transfer the action feature F g Set as Query, point cloud feature F p Set as Key, Language Text Feature F txt Set to the value Value, and feed them together into the cross-attention module to obtain the fused feature F. fusion The shape is [25,768].

[0095] Step 7: Fuse the feature F fusionThe data is fed into a strategy module composed of a multilayer perceptron to obtain an evaluation value S for each candidate grasping action, with a shape of [25,1].

[0096] Step 8: Sort each candidate grabbing action according to its evaluation value S, and prioritize the action that matches the task instructions for execution.

[0097] like Figure 4 As shown, the model was trained on the public dataset LA-TaskGrasp, which contains point cloud data of common household objects and their corresponding task description text. On the hardware platform side, an Elite EC66 robot was used for experimental deployment to verify the success rate and efficiency of this method in real-world scenarios.

[0098] In addition to the above embodiments, the present invention may have other implementation methods. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A robot grasping detection method based on a visual language action multi-modal alignment strategy, characterized in that: Comprising the following steps: S1, inputting a natural language instruction into a pre-trained large language model LLM for analysis to generate three types of descriptions of object category description, task description and grabbing description, and inputting the three types of descriptions into a pre-trained BERT model together with the natural language instruction for processing to obtain high-dimensional text features; S2, input the three-dimensional point cloud data of the object, extract the geometric and structural features through the pre-trained PointNet++ point cloud encoder, and generate point cloud embedding F p ; S3, input the point cloud data into the pre-trained FGC-GraspNet module to generate a set of candidate grasp poses g i ; the candidate grasp pose g i is unfolded into action features X g , and grasp features F g are extracted through a multi-layer perception machine MLP S4, performing channel attention optimization on the three types of descriptions and natural language instructions obtained in step S1 through a text encoder model to obtain language text features F txt ; S5、According to the grasping feature F g , the modal alignment module composed of multiple layers of perception is used to perform dimension alignment operation on the point cloud feature F p and the language text feature F txt , and the multi-modal feature is obtained. S6, according to the cross attention mechanism, the action feature F g Set as query Query, point cloud feature F p Set as key Key, language text feature F txt Set as value Value, the fusion of multi-modal features is realized through the cross attention module, and the fusion feature F is obtained fusion ; S7. The fusion feature F obtained in step S6 is... fusion The input strategy module extracts important information about the grasping features through a multilayer perceptron (MLP) and obtains the evaluation value S of each candidate grasping action through a sigmoid activation function. S8, according to the evaluation value S of each candidate grabbing action obtained in step S7, sorting all candidate grabbing actions, outputting a sorted action list, and controlling the robot to execute the optimal grabbing action, and if the grabbing fails, falling back to the suboptimal action.

2. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 1, characterized in that: The pre-trained PointNet++ point cloud encoder in the step S2 includes a multi-scale point cloud feature extraction layer MSG and a feature fusion layer SA, and a formula for generating a point cloud embedding is F p = SA(MSG(X p , dim = 0) p The point cloud embedding F p is expanded by a formula F p = Align(squeeze(F p , dim = 1), dim = 1, num = N) 3. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 2, characterized in that: The specific steps of the step S3 are as follows: S31, for each candidate grasp pose g of size 4x4 [R, T] homogeneous matrix i unwound into one-dimensional feature X g = [x, y, z, q0, q1, q2, q3]; where x, y, z represent the translation relationship, x, y, z = T tr ; q0, q1, q2, q3 represent the quaternion of the rotation matrix; S32, the unfolded grasp pose feature X g Input a grasp Encoder consisting of a multi-layer perceptron, MLP, to get F g The formula is as follows: F g = MLP(X g , dim = 3) In the formula, X g is the action feature after unfolding.

4. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 3, characterized in that: The specific steps of the step S4 are as follows: S41, the text encoder model encodes the three types of descriptions and natural language instructions obtained in step S1 to generate a task description paragraph Object class description paragraph Grabbing description paragraph And language instructions Word embedding; S42, adopt channel attention mechanism SE module to weight process text features, get language text features F txt , the formula is as follows: F txt = SE(stack(F td ,F od ,F gd ,F I ,dim = 1)) wherein stack concatenates F td , F od , F gd , F I into a unified multi-channel feature representation; S43, to the fused language feature F txt The channel replication operation is performed with the following formula: F txt = Align(squeeze(F txt , dim = 1), dim = 1, num = N) where N denotes the candidate grasp pose set g i squeeze removes the extra dimensions of the language features and replicates them across the channels.

5. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 4, characterized in that: The fusion feature F obtained in the step S6 fusion The formula is as follows: F fusion = cross_attention(Q=F g , K=F p , V=F txt ) wherein the action feature F g is for the query Q, the point cloud feature F p is for the key K, and the language text feature F txt is for the value V.

6. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 5, characterized in that: The formula for obtaining the evaluation value S of each candidate grabbing action is as follows: The specific steps of the step S3 are as follows: The specific steps of the step S4 are as follows: The formula for obtaining the evaluation value S of each candidate grabbing action is as follows: S = Sigmoid(MLP(F fusion ), dim = -1).

7. The robot grasp detection method based on visual language action multi-modal alignment strategy according to claim 6, characterized in that: The point cloud embedding in the step S2

Images