A method for constructing a multi-modal skill primitive in a 3C production line unloading scene

Abstract

This invention discloses a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario, comprising: acquiring video sequences using an RGB-D vision sensor, the video sequences including the motion trajectory and depth information of an operator's hand, the operator's hand being equipped with a pressure sensor; calculating the motion features of the motion trajectory and classifying the motion features based on a preset standard motion feature set to obtain a first classification result, the first classification result including a first type of skill primitive; calculating the pressure features of the motion trajectory using tactile information and classifying the pressure features based on a preset standard tactile feature set to obtain a second classification result, the second classification result including a second type of skill primitive; and constructing the skill primitive corresponding to the motion trajectory based on the first classification result, the second classification result, depth information, and tactile information. This invention can construct skill primitives based on vision, depth, and tactile senses and form a skill primitive library, facilitating skill learning and transfer.

Description

Technical Field

[0001] This invention belongs to the field of motion planning technology for assembly tasks, specifically relating to a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario. Background Technology

[0002] While machines today possess an increasing number of motor skills, the methods used to generate these skills remain largely unchanged. They primarily rely on experts to model motion based on their understanding of the skills, adapting to the diversity of environments. Since most current methods involve setting the machine's required motion trajectory for a placed object, reprogramming is necessary if the task changes. However, teach-through learning allows machines to autonomously perform new tasks, enabling users to teach the machine the necessary skills through teach-through tasks.

[0003] In related technologies, teaching research is mostly based on modeling motion models and then imitating and learning motion trajectories. The commonly used method is DMP (Dynamic Movement Primitives). Specifically, the DMP method obtains an optimal demonstration trajectory through multiple demonstrations, performs Gaussian filtering and smoothing on multiple trajectory samples, clusters features, and finally generalizes the trajectory through Gaussian mixture regression. Thus, the trajectory is imitated and learned based on the obtained optimal trajectory, and then the corresponding trajectory is matched based on the target detection results.

[0004] However, the above methods rely solely on visual and depth information, and require multiple demonstrations and teaching sessions to transmit trajectory features to the machine for training and learning, thus failing to achieve efficient transfer. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario, comprising:

[0007] A video sequence is acquired using an RGB-D vision sensor. The video sequence includes the movement trajectory of the operator's hand and the depth information of the operator's hand. The operator's hand is equipped with a pressure sensor for acquiring tactile information.

[0008] The motion characteristics of the motion trajectory are calculated, and the motion characteristics are classified based on a preset set of standard motion characteristics to obtain a first classification result, wherein the first classification result includes a first type of skill primitive;

[0009] The pressure characteristics of the motion trajectory are calculated using the tactile information, and the pressure characteristics are classified based on a preset standard tactile feature set to obtain a second classification result, which includes a second type of skill primitive;

[0010] Based on the first classification result, the second classification result, the depth information, and the tactile information, the skill primitives corresponding to the motion trajectory are constructed.

[0011] In one embodiment of the present invention, the step of calculating the motion characteristics of the motion trajectory includes:

[0012] Multiple video segments are extracted from the video sequence at fixed time intervals;

[0013] For each video segment, calculate the motion characteristics of the motion trajectory within the corresponding fixed time interval;

[0014] The step of calculating the motion features of the motion trajectory within a fixed time interval for each video segment includes:

[0015] Calculate the change in angle Δθ of the motion trajectory within a fixed time interval of the video segment;

[0016] Calculate the product of the velocity change of the motion trajectory within a fixed time interval of the video segment and Δθ.

[0017] In one embodiment of the present invention, the change in angle Δθ of the motion trajectory within a fixed time interval of the video segment is calculated according to the following formula:

[0018]

[0019] In the formula, arccos(·) represents the arccosine function, * represents the dot product, i represents the intermediate frame within the fixed time interval, and N is half the number of frames contained in the fixed time interval. This represents the velocity of the center point of the hand in the (i+m)th frame within the fixed time interval. This represents the velocity of the center point of the hand in the (i+n)th frame within the fixed time interval.

[0020] In one embodiment of the present invention, the product of the velocity change of the motion trajectory within a fixed time interval of the video segment and Δθ is calculated according to the following formula.

[0021]

[0022] In one embodiment of the present invention, a preset standard set of motion features in, This indicates that the skill's primitive type is "take". Indicates that the skill primitive type is "release", Indicates that the skill primitive type is "move";

[0023] The step of classifying the motion feature based on a preset set of standard motion features to obtain a first classification result includes:

[0024] Based on a preset set of standard motion features, according to the following formula for Perform pre-classification:

[0025]

[0026] where x is the Pre-classification result of;

[0027] Calculate according to the following formula Confidence value value of the pre-classification result x of v :

[0028]

[0029] When Confidence value value of the pre-classification result x of v Is less than or equal to the first threshold, then The pre-classification result x of is determined as the first type of skill primitive within the fixed time interval of the video segment; where Indicates The pre-classification result of is "pick up", and the first classification result class V =[1,0,0], Indicates The pre-classification result of is "release", and the first classification result class V =[0,1,0], Indicates The pre-classification result of is "move", and the first classification result class V =[0,0,1].

[0030] In an embodiment of the present invention, the step of calculating the pressure feature of the motion trajectory using the tactile information includes:

[0031] For each video segment, determine the pressure change P within the fixed time interval according to the tactile information, and determine the tactile calculation coefficient K within the fixed time interval according to the pressure change P; where

[0032] <able> [[ID=:67]]

[0033] Obtain the pressure values at each moment within the fixed time interval according to the tactile information;

[0034] Calculate the pressure value within the fixed time interval based on the pressure values at each moment within the fixed time interval.

[0035] In one embodiment of the present invention, calculate the pressure value within the fixed time interval according to the following formula

[0036]

[0037] In the formula, P t represents the pressure value at time t within the fixed time interval, P t-1 represents the pressure value at time t - 1 within the fixed time interval, and P0 is a preset standard pressure value.

[0038] In one embodiment of the present invention, a preset standard tactile feature set wherein, represents that the skill primitive type is "pick up", represents that the skill primitive type is "put down", represents that the skill primitive type is "move";

[0039] The step of classifying the pressure feature based on the preset standard tactile feature set to obtain the second classification result of the skill primitive includes:

[0040] Perform pre-classification on according to the following formula:

[0041]

[0042] where d represents the Euclidean distance and y is the pre-classification result of;

[0043] Calculate the confidence value of the pre-classification result y of according to the following formula P :

[0044]

[0045] When the confidence value value of the v pre-classification result y of is less than or equal to the second threshold, determine the pre-classification result y of as the second type of the skill primitive within the fixed time interval of the video segment; wherein, represents that the pre-classification result of is "pick up", and the second classification result class P = [1, 0, 0], represents that the pre-classification result of is "put down", and the second classification result classP =[0,1,0], express The pre-classification result is "move", and the second classification result is class P =[0,0,1].

[0046] In one embodiment of the present invention, the step of constructing the skill primitive corresponding to the motion trajectory based on the first result, the second classification result, the depth information, and the tactile information includes:

[0047] Based on the first classification result (class) V and the second classification result class P The type of skill primitives that determine the motion trajectory within a fixed time interval of the video segment;

[0048] Based on the tactile information and the depth information, determine the attributes of the skill primitives of the motion trajectory within a fixed time interval of the video segment;

[0049] Skill primitives are constructed based on the sum and attributes of the aforementioned skill primitives.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] This invention provides a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario. For each video segment extracted from a video sequence, the motion and pressure features of the video segment are calculated. The motion features are classified based on a preset standard set of motion features, and the pressure features are classified based on a preset standard set of tactile features. Finally, the type of skill primitive is determined. After determining the attributes of the skill primitive based on depth and tactile information, a multimodal skill primitive based on vision, depth, and tactile senses can be constructed and a skill primitive library can be formed, which facilitates skill learning and transfer.

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0053] Figure 1 This is a flowchart of a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario provided by an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario provided in an embodiment of the present invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0056] Figure 1 This is a flowchart of a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario provided by an embodiment of the present invention. For example... Figure 1-2 As shown, this embodiment of the invention provides a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario, including:

[0057] S1. A video sequence is acquired using an RGB-D vision sensor. The video sequence includes the movement trajectory of the operator's hand and the depth information of the operator's hand. The operator's hand is equipped with a pressure sensor for acquiring tactile information.

[0058] S2. Calculate the motion characteristics of the motion trajectory, and classify the motion characteristics based on the preset standard motion characteristic set to obtain the first classification result, which includes the first type of skill primitive;

[0059] S3. Calculate the pressure characteristics of the motion trajectory using tactile information, and classify the pressure characteristics based on a preset standard tactile feature set to obtain a second classification result. The second classification result includes the second type of skill primitive.

[0060] S4. Based on the first classification result, the second classification result, depth information and tactile information, construct the skill primitives corresponding to the motion trajectory.

[0061] In this embodiment, a video sequence containing the movement trajectory of the operator's hand is acquired using a RealSense L515 camera to simultaneously obtain RGB video and depth information of the operator's hand. Optionally, tactile information is obtained through a pressure sensor installed on the operator's hand.

[0062] Furthermore, a skeleton point detection network is used to extract the two-dimensional motion trajectory of the operator's hand from the video sequence. Combined with depth information, the motion trajectory is mapped onto Cartesian space to extract motion features. These motion features are then classified to obtain a first type of skill primitive. For example, the first type can be one of "grab," "move," or "place," indicating that the skill primitive determined based on the motion features is "grab," "move," or "place." In step S3 above, tactile information is used to calculate the pressure features of the motion trajectory, and the second type of skill primitive is determined by classifying these pressure features. Similarly, the second type indicates that the skill primitive determined based on the pressure features is "grab," "move," or "place."

[0063] Finally, based on the first type, the second type, depth information, and tactile information, the type and attributes of the motion primitives are determined, thereby constructing skill primitives. For example, as shown in Table 1, the types of motion primitives include "grab," "move," and "place," and the attributes include: the trajectory type of the motion path, such as one-dimensional motion, two-dimensional motion, or three-dimensional motion; the pressure change type, such as continuously decreasing, remaining constant, or continuously increasing; whether the pressure is continuous or non-continuous, such as whether it is periodic, such as repeating motion according to a period or performing non-periodic motion, etc. It should be noted that in Table 1, hollow circles indicate only one type, while solid circles indicate multiple types.

[0064] Table 1

[0065]

[0066] Optionally, the step of calculating the motion characteristics of the motion trajectory in step S2 above includes:

[0067] S201. Extract multiple video segments from the video sequence at fixed time intervals;

[0068] S202. For each video segment, calculate the motion characteristics of the motion trajectory within the corresponding fixed time interval;

[0069] The step of calculating the motion features of the motion trajectory within a fixed time interval for each video segment includes:

[0070] S2021. Calculate the change in angle Δθ of the motion trajectory within a fixed time interval of a video segment;

[0071] S2022. Calculate the product of the velocity change of the motion trajectory within a fixed time interval of the video segment and Δθ.

[0072] Specifically, the change in angle Δθ of the motion trajectory within a fixed time interval of the video segment is calculated using the following formula:

[0073]

[0074] In the formula, arccos(·) represents the arccosine function, * represents the dot product, i represents the intermediate frame within the fixed time interval, and N is half the number of frames contained within the fixed time interval. This represents the velocity of the center point of the hand in the (i+m)th frame within a fixed time interval. This represents the velocity of the center point of the hand in the (i+n)th frame within a fixed time interval.

[0075] It should be noted that, This is the ratio of the distance the hand's center point moves in the (i+m)th frame to the time interval between the two frames. is the ratio of the moving distance between the center point of the hand in the (i + n)-th frame and its previous frame to the time interval between the two frames.

[0076] Further, calculate the product of the velocity change of the motion trajectory within the fixed time interval of the video segment and Δθ according to the following formula

[0077]

[0078] In this embodiment, the preset standard motion feature set where indicates that the skill primitive type is "pick up", indicates that the skill primitive type is "put down", indicates that the skill primitive type is "move".

[0079] The steps of classifying the motion features based on the preset standard motion feature set to obtain the first classification result include:

[0080] Based on the preset standard motion feature set, perform pre-classification on according to the following formula:

[0081]

[0082] where x is the pre-classification result of ;

[0083] Calculate the confidence value of the pre-classification result x of according to the following formula v :

[0084]

[0085] When the confidence value of the pre-classification result x of v is less than or equal to the first threshold, determine the pre-classification result x of as the first type of the skill primitive within the fixed time interval of the video segment; where indicates that the pre-classification result of is "pick up", and the first classification result class V = [1, 0, 0], indicates that <于 the pre-classification result of is "put down", and the first classification result class V = [0, 1, 0], indicates that the pre-classification result of is "move", and the first classification result class V = [0, 0, 1].

[0086] It should be noted that, in the context of When performing pre-classification, firstly according to the formula Solve for the set of standard motion characteristics that minimize x. The skill primitive types in the calculation are obtained. For example, if the confidence value of x is... v If it is greater than the first threshold, then it means Pre-classification results This is not accurate; further calculation is needed to find the standard motion feature set that gives x its second smallest value. The skill primitive types in the middle are or

[0087] In step S3 above, the step of calculating the pressure characteristics of the motion trajectory using tactile information includes:

[0088] S301. For each video segment, determine the pressure change P within a fixed time interval based on tactile information, and determine the tactile calculation coefficient K within the fixed time interval based on the pressure change P; wherein,

[0089]

[0090] S302. Obtain the pressure value at each moment within a fixed time interval based on tactile information;

[0091] S303. Calculate the pressure value within a fixed time interval based on the pressure value at each moment within the fixed time interval.

[0092] Specifically, the pressure value within a fixed time interval is calculated using the following formula.

[0093]

[0094] In the formula, P t P represents the pressure value at time t within a fixed time interval. t-1 P0 represents the pressure value at time t-1 within a fixed time interval, where P0 is the preset standard pressure value.

[0095] For example, a preset set of standard pressure characteristics in, This indicates that the skill's primitive type is "take". This indicates that the skill's basic type is "Release". This indicates that the skill primitive type is "Move";

[0096] The step of classifying the pressure features based on a preset standard set of tactile features to obtain a second classification result of the skill primitives includes:

[0097] Pre-classify according to the following formula as follows:

[0098]

[0099] where d represents the Euclidean distance and y is the pre-classification result;

[0100] Calculate the confidence value of the pre-classification result y of P according to the following formula:

[0101]

[0102] When the confidence value P of the pre-classification result y of is less than or equal to the second threshold, determine the pre-classification result y of as the second type of skill primitive within the fixed time interval of the video segment; where indicates that the pre-classification result of P is "pick up", and the second classification result class indicates that the pre-classification result of P is "put down", and the second classification result class indicates that the pre-classification result of P is "move", and the second classification result class

[0103] Specifically, similar to the pre-classification process of motion features, when pre-classifying , first solve the set of standard pressure feature that makes y have the minimum value according to the formula to calculate the type of skill primitive in . Taking P as an example, if the confidence value of y is greater than the second threshold, it means that the pre-classification result is not accurate. At this time, it is necessary to further solve the type of skill primitive in the set of standard motion features that makes y have the second minimum value, that is or

[0104] In the above step S4, the step of constructing the skill primitive corresponding to the motion trajectory according to the first classification result, the second classification result, the depth information, and the tactile information includes:

[0105] S401. According to the first classification result classV Second classification result class P To determine the type of skill primitives that define the motion trajectory within a fixed time interval of a video clip;

[0106] S402. Based on tactile and depth information, determine the attributes of the skill primitives of motion trajectories within fixed time intervals of a video clip;

[0107] S403. Construct skill primitives based on skill primitives and attributes.

[0108] Specifically, in step S401, the type class of the skill primitives of the motion trajectory within a fixed time interval of the video segment is determined according to the following formula:

[0109] class = K * max(class) V +class P )

[0110] With K=1, class V =[1,0,0]、class P For example, if the value is [1,0,0], then class = 2. Since class is greater than the threshold of 1.5, it can be considered that class... V and class P All the classification results are correct.

[0111] Furthermore, taking K=1 as an example, if class V =[1,0,0]、class P If the value is [0, 1, 0], then class = 1. In this case, class can be considered as [0, 1, 0]. V and class P The classification result is incorrect.

[0112] In this embodiment, the attributes of the skill primitive may include: the trajectory type of the motion trajectory, such as one-dimensional motion, two-dimensional motion or three-dimensional motion; the pressure change type, such as continuously decreasing, remaining constant or continuously increasing; whether the pressure is continuous or non-continuous, such as whether the pressure is periodic, such as repeating motion according to a period or performing non-periodic motion, etc. Therefore, the skill primitive finally constructed can be: movement is a non-periodic motion that occurs together between the hand and the object on a plane or inclined plane.

[0113] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows:

[0114] This invention provides a method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario. For each video segment extracted from a video sequence, the motion and pressure features of the video segment are calculated. The motion features are classified based on a preset standard set of motion features, and the pressure features are classified based on a preset standard set of tactile features. Finally, the type of skill primitive is determined. After determining the attributes of the skill primitive based on depth and tactile information, a multimodal skill primitive based on vision, depth, and tactile senses can be constructed and a skill primitive library can be formed, which facilitates skill learning and transfer.

[0115] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0116] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0117] Although this application has been described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the claimed application by reviewing the accompanying drawings, the disclosure, and the appended claims.

[0118] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for constructing multimodal skill primitives in a 3C production line loading and unloading scenario, characterized in that, include: A video sequence is acquired using an RGB-D vision sensor. The video sequence includes the movement trajectory of the operator's hand and the depth information of the operator's hand. The operator's hand is equipped with a pressure sensor for acquiring tactile information. The motion characteristics of the motion trajectory are calculated, and the motion characteristics are classified based on a preset set of standard motion characteristics to obtain a first classification result, wherein the first classification result includes a first type of skill primitive; The pressure characteristics of the motion trajectory are calculated using the tactile information, and the pressure characteristics are classified based on a preset standard tactile feature set to obtain a second classification result, which includes a second type of skill primitive; Based on the first classification result, the second classification result, the depth information, and the tactile information, the skill primitives corresponding to the motion trajectory are constructed.

2. The method for constructing multimodal skill primitives in the 3C production line loading and unloading scenario according to claim 1, characterized in that, The steps for calculating the motion characteristics of the motion trajectory include: Multiple video segments are extracted from the video sequence at fixed time intervals; For each video segment, calculate the motion characteristics of the motion trajectory within the corresponding fixed time interval; The step of calculating the motion features of the motion trajectory within a fixed time interval for each video segment includes: Calculate the change in angle Δθ of the motion trajectory within a fixed time interval of the video segment; Calculate the product of the velocity change of the motion trajectory within a fixed time interval of the video segment and Δθ.

3. The method for constructing multimodal skill primitives in the 3C production line loading and unloading scenario according to claim 2, characterized in that, The change in angle Δθ of the motion trajectory within a fixed time interval of the video segment is calculated using the following formula: In the formula, arccos(·) represents the arccosine function, * represents the dot product, i represents the intermediate frame within the fixed time interval, and N is half the number of frames contained in the fixed time interval. This represents the velocity of the center point of the hand in the (i+m)th frame within the fixed time interval. This represents the velocity of the center point of the hand in the (i+n)th frame within the fixed time interval.

4. The method for constructing multimodal skill primitives in the 3C production line loading and unloading scenario according to claim 3, characterized in that, Calculate the product of the velocity change of the motion trajectory within a fixed time interval of the video segment and Δθ using the following formula.

5. The method for constructing multimodal skill primitives in the 3C production line loading and unloading scenario according to claim 4, characterized in that, Preset standard motion feature set in, This indicates that the skill's primitive type is "take". This indicates that the skill's basic type is "Release". This indicates that the skill primitive type is "Move"; The step of classifying the motion features based on a preset standard set of motion features to obtain a first classification result includes: Based on a preset set of standard motion features, the following formula is used to... Perform pre-classification: Where x is The pre-classification results; Calculate according to the following formula The confidence value of the pre-classification result x v : When the confidence value of the pre-classification result x v is less than or equal to the first threshold, the pre-classification result x is determined as the first type of skill primitive within the fixed time interval of the video segment; where indicates that the pre-classification result of V is "pick up", and the first classification result class indicates that the pre-classification result of V is "put down", and the first classification result class indicates that the pre-classification result of V is "move", and the first classification result class 6. The method for constructing multimodal skill primitives in the material loading and unloading scenario of a 3C production line according to claim 2, characterized in that, The step of calculating the pressure characteristics of the motion trajectory using the tactile information includes: For each video segment, the pressure change P within the fixed time interval is determined based on tactile information, and the tactile calculation coefficient K within the fixed time interval is determined based on the pressure change P; wherein, Based on the tactile information, the pressure value at each moment within a fixed time interval is obtained; The pressure value within the fixed time interval is calculated based on the pressure value at each moment within the fixed time interval.

7. The method for constructing multimodal skill primitives in the 3C production line loading and unloading scenario according to claim 6, characterized in that, Calculate the pressure value within the fixed time interval using the following formula. In the formula, P t P represents the pressure value at time t within the fixed time interval. t-1 P0 represents the pressure value at time t-1 within the fixed time interval, where P0 is the preset standard pressure value.

8. The method for constructing multimodal skill primitives in the material loading and unloading scenario of a 3C production line according to claim 7, characterized in that, Preset standard pressure characteristic set in, This indicates that the skill's primitive type is "take". This indicates that the skill's basic type is "Release". This indicates that the skill primitive type is "Move"; The step of classifying the pressure features based on a preset standard set of tactile features to obtain a second classification result for the skill primitive includes: According to the following formula Perform pre-classification: Where d represents the Euclidean distance, and y is... The pre-classification results; Calculate according to the following formula The confidence value of the pre-classification result y P : When the confidence value of the pre-classification result y v is less than or equal to the second threshold, then the pre-classification result y is determined as the second type of skill primitive within the fixed time interval of the video segment; where means the pre-classification result of P is "pick up", and the second classification result class means the pre-classification result of P is "put down", and the second classification result class means the pre-classification result of P is "move", and the second classification result class 9. The method for constructing multimodal skill primitives in the material loading and unloading scenario of a 3C production line according to claim 2, characterized in that, The step of constructing the skill primitives corresponding to the motion trajectory based on the first classification result, the second classification result, the depth information, and the tactile information includes: Based on the first classification result (class) V and the second classification result class P The type of skill primitives that determine the motion trajectory within a fixed time interval of the video segment; Based on the tactile information and the depth information, determine the attributes of the skill primitives of the motion trajectory within a fixed time interval of the video segment; Skill primitives are constructed based on the sum and attributes of the aforementioned skill primitives.

Images