Reinforcement learning model training method, single-layer cloth separation method, device, equipment and medium

Abstract

The invention discloses a reinforcement learning model training method, a single-layer cloth separation method and device, equipment and a medium, and relates to the technical field of computers. Comprising the steps of obtaining track features in a current environment based on a current strategy in a current reinforcement learning model; the current strategy is a strategy constructed based on a material distribution state and a mechanical arm action, and the track characteristics comprise a material distribution state characteristic, a mechanical arm action characteristic and an initial reward corresponding to the mechanical arm action characteristic; adding a guide reward for accelerating a learning process to the initial reward to obtain a current total reward; constructing a corresponding dominant function based on the current total reward and generalized dominant estimation, and generating a target function based on the dominant function and a KL divergence penalty coefficient; and optimizing strategy parameters of a current strategy in the current reinforcement learning model in a mode of maximizing the target function to obtain a target reinforcement learning model for separating single-layer cloth. Therefore, separation of single-layer cloth can be realized based on reinforcement learning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a reinforcement learning model training method, a single-layer cloth separation method, a device, equipment, and a medium. Background Art

[0002] With the advancement of automation and intelligent manufacturing technologies, robotics are increasingly being used in textile processing. Deep learning, a powerful intelligent method, has demonstrated significant advantages in robotic grasping and manipulation tasks. By automatically extracting features and patterns from large amounts of data, deep learning can optimize control strategies, adapt to complex dynamic environments, and enhance the flexibility and robustness of robotic operations. In particular, achieving efficient and precise automation in operations such as layering, folding, and handling fabrics is crucial for improving production efficiency and reducing labor costs. However, due to the softness, variability, and easy deformation of fabric materials, automated processing faces numerous challenges. Traditional fabric layering methods typically rely on mechanical grippers and simple sensor feedback, which struggle to cope with the complex changes in fabric shape, resulting in unstable layering and frequent misoperation. Furthermore, while rigid grippers offer a simple structure, they often suffer from insufficient gripping force or damage when grasping soft materials, limiting their application. Flexible grippers can adapt to the shape and thickness of fabrics, achieving a gentler and more uniform grip through soft materials or deformable structures, significantly improving gripping reliability and safety. However, the control complexity of the flexible gripper is high, and traditional control strategies are difficult to cope with diverse fabric shapes and environmental interference.

[0003] From the above, it can be seen that how to separate single-layer cloth based on reinforcement learning is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a reinforcement learning model training method, a single-layer cloth separation method, device, equipment and medium, which can realize the separation of single-layer cloth based on reinforcement learning. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a reinforcement learning model training method for separating a single layer of cloth, comprising:

[0006] Acquire trajectory features in the current environment based on the current policy in the current reinforcement learning model; the current policy is a policy based on the cloth state and the robot arm motion, and the trajectory features include cloth state features, robot arm motion features, and an initial reward corresponding to the robot arm motion features;

[0007] Adding a guidance reward for accelerating the learning process to the initial reward to obtain a current total reward;

[0008] Constructing an advantage function based on the current total reward and the generalized advantage estimate, and generating an objective function based on the advantage function and a KL divergence penalty coefficient;

[0009] The policy parameters of the current policy in the current reinforcement learning model are optimized by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

[0010] Optionally, the cloth state feature includes first feature information of distance data between the robotic arm and the cloth, second feature information of cloth image data, and third feature information of force feedback data collected when the end of the robotic arm contacts the cloth.

[0011] Optionally, the reinforcement learning model training method for separating a single layer of cloth further includes:

[0012] Acquiring initial target data through a preset feature extraction network; the initial target data includes cloth image data, distance data between the robotic arm and the cloth, and force feedback data collected when the end of the robotic arm contacts the cloth;

[0013] Performing feature extraction on the initial target data through the preset feature extraction network, and sending the extracted cloth state features to the current reinforcement learning model;

[0014] Among them, the preset feature extraction network is configured with a first feature extraction branch and a second feature extraction branch; the first feature extraction branch is used to extract features from the cloth image data, and the second feature extraction branch is used to splice the distance data and the force feedback data, and perform feature extraction on the spliced ​​results.

[0015] Optionally, the first feature extraction branch includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a first linear layer connected in sequence; the second feature extraction branch includes a second linear layer, a third activation layer, and a third linear layer connected in sequence;

[0016] The preset feature extraction network is further configured with a fourth linear layer, and the fourth linear layer is used to splice the output result of the first linear layer and the output result of the third linear layer to obtain the cloth state feature.

[0017] In a second aspect, the present application provides a single-layer cloth separation method based on a reinforcement learning model, wherein the reinforcement learning model is a model trained by the aforementioned reinforcement learning model training method; wherein the method comprises:

[0018] Acquiring distance data between the end of the robotic arm and the target cloth, and controlling the robotic arm to descend based on the distance data;

[0019] When the end of the robotic arm contacts the target cloth, force feedback data collected by the end of the robotic arm is obtained;

[0020] Acquiring target cloth image data collected by the end of the robotic arm, and performing a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features;

[0021] The target cloth state characteristics are input into the reinforcement learning model, and the robot arm motion control instructions output by the reinforcement learning model are used to control the robot arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth.

[0022] Optionally, the process of controlling the robotic arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth further includes:

[0023] Detecting the target fabric using a target detection algorithm to obtain corresponding real-time detection results;

[0024] Determining whether there is a single-layer fabric grabbing failure based on the real-time detection results, and performing corresponding processing operations according to the failure type;

[0025] The failure types include a first failure type in which the end of the robotic arm fails to successfully grasp the cloth and a second failure type in which the end of the robotic arm grasps multiple layers of cloth.

[0026] In a third aspect, the present application provides a reinforcement learning model training device for separating a single layer of cloth, comprising:

[0027] A trajectory feature acquisition module is configured to acquire trajectory features in the current environment based on a current policy in the current reinforcement learning model; the current policy is a policy based on the cloth state and the robot arm motion, and the trajectory features include cloth state features, robot arm motion features, and an initial reward corresponding to the robot arm motion features;

[0028] a current total reward generating module, configured to add a guidance reward for accelerating the learning process to the initial reward to obtain a current total reward;

[0029] An objective function generation module, configured to construct a corresponding advantage function based on the current total reward and the generalized advantage estimate, and to generate an objective function based on the advantage function and a KL divergence penalty coefficient;

[0030] A strategy parameter optimization module is used to optimize the strategy parameters of the current strategy in the current reinforcement learning model by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

[0031] In a fourth aspect, the present application provides a single-layer cloth separation device based on a reinforcement learning model, wherein the reinforcement learning model is a model trained by the aforementioned reinforcement learning model training method; wherein the device comprises:

[0032] A first data acquisition module is used to acquire distance data between the end of the robotic arm and the target cloth;

[0033] A first control module, configured to control the robotic arm to descend based on the distance data;

[0034] a second data acquisition module, configured to acquire force feedback data collected by the end of the robotic arm after the end of the robotic arm contacts the target cloth;

[0035] a third data acquisition module, configured to acquire target cloth image data collected by the end of the robotic arm;

[0036] a feature extraction module, configured to perform a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features;

[0037] The second control module is used to input the target cloth state characteristics into the reinforcement learning model, and use the robot arm motion control instructions output by the reinforcement learning model to control the robot arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth.

[0038] In a fifth aspect, the present application provides an electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] The processor is configured to execute the computer program to implement the aforementioned method.

[0041] In a sixth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0042] The present application provides a reinforcement learning model training method for separating a single layer of cloth. First, based on the current policy in the current reinforcement learning model, trajectory features in the current environment are obtained; the current policy is a policy constructed based on the cloth state and the robot arm movement, and the trajectory features include cloth state features, robot arm movement features, and an initial reward corresponding to the robot arm movement features; then, a guidance reward for accelerating the learning process is added to the initial reward to obtain a current total reward; then, a corresponding advantage function is constructed based on the current total reward and the generalized advantage estimate, and an objective function is generated based on the advantage function and the KL divergence penalty coefficient; finally, the policy parameters of the current policy in the current reinforcement learning model are optimized by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

[0043] As can be seen above, this application introduces guided rewards to help accelerate the model's learning process and prevent the strategy from falling into a local optimum. The reinforcement learning model's input data and output actions can be dynamically adjusted for different physical data fabrics to capture fabrics with various physical properties. The reinforcement learning model is highly robust and generalizable, and model training can handle a variety of fabrics, resolving the problem of automated separation of single-layer fabrics requiring frequent modifications to related structures and parameters. This makes it possible to separate single-layer fabrics based on reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A roadmap for a reinforcement learning model training method for separating single-layer fabrics disclosed in this application;

[0046] Figure 2 A schematic diagram of a preset feature extraction model structure disclosed in this application;

[0047] Figure 3 This is a flow chart of a single-layer cloth separation method based on a reinforcement learning model disclosed in this application;

[0048] Figure 4 This is a schematic diagram of a reinforcement learning model training device for separating single-layer fabrics disclosed in this application;

[0049] Figure 5 This is a schematic diagram of a single-layer cloth separation device based on a reinforcement learning model disclosed in this application;

[0050] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] With the advancement of automation and intelligent manufacturing technologies, robotics are increasingly being used in textile processing. Deep learning, a powerful intelligent method, has demonstrated significant advantages in robotic grasping and manipulation tasks. By automatically extracting features and patterns from large amounts of data, deep learning can optimize control strategies, adapt to complex dynamic environments, and enhance the flexibility and robustness of robotic operations. In particular, achieving efficient and precise automation in operations such as layering, folding, and handling fabrics is crucial for improving production efficiency and reducing labor costs. However, due to the softness, variability, and easy deformation of fabric materials, automated processing faces numerous challenges. Traditional fabric layering methods typically rely on mechanical grippers and simple sensor feedback, which struggle to cope with the complex changes in fabric shape, resulting in unstable layering and frequent misoperation. Furthermore, while rigid grippers offer a simple structure, they often suffer from insufficient gripping force or damage when grasping soft materials, limiting their application. Flexible grippers can adapt to the shape and thickness of fabrics, achieving a gentler and more uniform grip through soft materials or deformable structures, significantly improving gripping reliability and safety. However, the control complexity of the flexible gripper is relatively high, and traditional control strategies are difficult to cope with the diverse fabric shapes and environmental interference. To this end, this application discloses a reinforcement learning model training and single-layer fabric separation solution, which can achieve single-layer fabric separation based on reinforcement learning.

[0053] See also Figure 1 As shown, the embodiment of the present application discloses a reinforcement learning model training method for separating a single layer of cloth, comprising:

[0054] Step S11: Acquire trajectory features in the current environment based on the current strategy in the current reinforcement learning model.

[0055] In this embodiment, the current reinforcement learning model is a model built based on the Group Relative Policy Optimization (GRPO) algorithm. GRPO belongs to the policy optimization method, the goal of which is to learn an optimal policy by maximizing the expected cumulative reward in a given environment. .in, is a policy parameter, s is the state, and a is the action. The current policy is based on the cloth state and the robot arm's motion. The trajectory features include cloth state features, robot arm motion features, and the initial reward corresponding to the robot arm's motion features. The cloth state features include first feature information of the distance data between the robot arm and the cloth, second feature information of the cloth image data, and third feature information of the force feedback data collected when the end of the robot arm contacts the cloth.

[0056] See also Figure 2 As shown, in this embodiment, to improve the robustness and generalization of the reinforcement learning model, the distance data between the manipulator and the fabric, the force feedback data collected when the end of the manipulator contacts the fabric, and the fabric image data are used as inputs to a preset feature extraction network for feature extraction. Specifically, the reinforcement learning model training method for separating a single layer of fabric may further include: obtaining initial target data through a preset feature extraction network; the initial target data includes fabric image data, the distance data between the manipulator and the fabric, and the force feedback data collected when the end of the manipulator contacts the fabric; performing feature extraction on the initial target data through the preset feature extraction network, and transmitting the extracted fabric state features to the current reinforcement learning model; wherein the preset feature extraction network is configured with a first feature extraction branch and a second feature extraction branch; the first feature extraction branch is configured to perform feature extraction on the fabric image data, and the second feature extraction branch is configured to splice the distance data and the force feedback data, and then perform feature extraction on the spliced ​​result. The first feature extraction branch includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a first linear layer connected in sequence; the second feature extraction branch includes a second linear layer, a third activation layer, and a third linear layer connected in sequence; the preset feature extraction network is further configured with a fourth linear layer, which is used to splice the output results of the first linear layer and the output results of the third linear layer to obtain the fabric state feature.

[0057] Step S12: Add a guidance reward for accelerating the learning process to the initial reward to obtain the current total reward.

[0058] In this embodiment, a set of trajectory data is collected in the environment according to the current strategy. , and combine the original reward with the guidance reward to get the total reward , the specific formula is as follows:

[0059] ;

[0060] in, For the original reward; To guide the reward, The guidance reward weight parameter is introduced to help accelerate the learning process and prevent the strategy from falling into the local optimum.

[0061] Step S13: construct a corresponding advantage function based on the current total reward and the generalized advantage estimate, and generate an objective function based on the advantage function and the KL divergence penalty coefficient.

[0062] In this embodiment, a corresponding advantage function is constructed based on the current total reward and the generalized advantage estimate. The advantage function The specific formula is as follows:

[0063] ;

[0064] in, Discount Factor, the value range is [0,1]; is the adjustment parameter of the generalized advantage estimation (GAE) with a value range of [0,1]; l is the span of future time steps (the lth step starting from the current time step t); is the time step The timing difference error.

[0065] Furthermore, the policy parameters are updated by maximizing the objective function, which is as follows:

[0066] ;

[0067] in, is the objective function to be maximized, which includes two parts: strategy performance improvement and strategy update constraints; is the current strategy (parameters are ) in the state Select Action probability; For the old strategy (parameters are ) in the state Select Action probability; is the KL divergence, which is used to measure the difference between the new and old strategies; It is used to control the KL divergence penalty, limit the range of strategy changes, and ensure training stability.

[0068] Step S14: Optimizing the policy parameters of the current policy in the current reinforcement learning model by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

[0069] In this embodiment, by updating the policy parameters The value of and the aforementioned model training process are repeated to obtain the maximized objective function, thereby obtaining a target reinforcement learning model for separating a single layer of cloth. It should be noted that the input data and output actions of the preset reinforcement learning model can be dynamically adjusted to adapt to cloth with different physical data.

[0070] As can be seen from the above, the embodiments of the present application introduce guided rewards to help accelerate the model's learning process and prevent the strategy from falling into a local optimum. The reinforcement learning model's input data and output actions can be dynamically adjusted for different physical data fabrics to capture fabrics with different physical properties. By obtaining initial target data through a pre-set feature extraction network and performing feature extraction on this initial target data, the model's robustness and generalization can be improved. Model training can handle a variety of fabrics, resolving the issue of automated separation of single-layer fabrics requiring frequent modifications to related structures and parameters. This enables the separation of single-layer fabrics based on reinforcement learning.

[0071] See also Figure 3 As shown, the embodiment of the present application discloses a single-layer cloth separation method based on a reinforcement learning model, comprising:

[0072] Step S21: Acquire distance data between the end of the robotic arm and the target cloth, and control the robotic arm to descend based on the distance data.

[0073] In this embodiment, by reading the distance between the end workpiece and the fabric, the robotic arm is controlled to descend to a specified height for contact. The end of the robotic arm includes a laser displacement sensor, a visual-tactile sensor, a six-dimensional force sensor, and a flexible gripper. The laser displacement sensor is a high-precision measuring instrument that uses a laser beam to measure the distance between an object and a sensor. Distance measurement is performed by triangulation. The laser emits a laser beam to illuminate the surface of the object to be measured. The reflected light is projected onto the photosensitive element of the photodetector through the optical system. The distance is calculated using the geometric relationship of the triangle according to the position offset of the reflected light point on the detector. Assume that the laser emission point is O, the plane of the photodetector is the photosensitive surface, and the angle between the laser line and the photosensitive surface is a known angle. The surface of the object to be measured positions the reflected light point as point P on the photosensitive surface. Using the trigonometric function relationship, the relationship between the distance D and x is as shown in the formula:

[0074] ;

[0075] Where L is the baseline length (the known distance from the emission point to the detector position), x is the position offset of the light spot on the photosensitive surface, is the angle between the sensor and the light beam, and f is the focal length.

[0076] Furthermore, after the laser displacement sensor obtains the descending height, the robotic arm is controlled to descend until the six-dimensional force sensor and the visual-tactile sensor are in complete contact with the target fabric.

[0077] Step S22: After the end of the robotic arm contacts the target cloth, force feedback data collected by the end of the robotic arm is obtained.

[0078] In this embodiment, when the visual-tactile sensor at the end of the robotic arm and the six-dimensional force sensor are in full contact with the target fabric, the rotation degree and inflation volume data of the flexible gripper are adjusted to sense the fabric properties, and the physical properties of the fabric are indirectly obtained by reading the six-dimensional force sensor values. Specifically, the adjustment of the control parameters of the flexible gripper and the acquisition of the six-dimensional force data collected by the six-dimensional force sensor may include: adjusting the rotation of the end joint of the robotic arm to 5 degrees and the inflation volume of the flexible gripper to 20; and reading the current six-dimensional force data acquired by the six-dimensional force sensor via the Modbus RTU protocol. It is understandable that the specific values ​​of the rotation degree and inflation volume data of the flexible gripper can be determined according to the actual application situation and are not specifically limited here.

[0079] Step S23: acquiring the target cloth image data collected by the end of the robotic arm, and performing a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features.

[0080] In this embodiment, image data is acquired using a visual-tactile sensor at the end of the robotic arm. This sensor combines visual and tactile sensing to simultaneously capture an object's shape, texture, and contact force information. The tactile sensor incorporates a flexible membrane or elastic material. When an object contacts and deforms the membrane, the deformation is recorded through visual imaging. The sensing surface typically features a known pattern (such as a dot matrix or grid). Touching distorts the pattern, and a camera captures this distorted image. Image processing algorithms are then used to calculate contact force and shape information. It's worth noting that OpenCV can also be used to read image data from the visual-tactile sensor to perceive fabric texture information.

[0081] In this embodiment, feature extraction operations are performed on the laser displacement sensor data, the six-dimensional force sensor data, and the visual-tactile sensor image data to obtain the target fabric state features. Specifically, the visual-tactile sensor image data dimension is 700*400*3, and the image is downsampled to 84*84*3, retaining the texture feature information while reducing computational consumption. The 84*84*3 image data undergoes two convolutions for feature extraction. The first convolution layer changes the number of image data channels from 3 to 32, and the second convolution layer changes the image data channels from 32 to 64. After passing through a ReLU activation layer, the image data is output as 256-dimensional features through a linear layer. The distance value of the laser displacement sensor is spliced ​​with the numerical value of the six-dimensional force sensor, and the seven-dimensional data is expanded to 64 dimensions through a linear layer. After passing through a normalization layer and a ReLU activation layer, a linear layer is used to expand the 64 dimensions to 128 dimensions. The 256-dimensional visual-tactile sensor data, the 128-dimensional distance and the six-dimensional force sensor data are concatenated in the first dimension and then a linear layer is used to output 256-dimensional data.

[0082] Step S24: input the target cloth state characteristics into the reinforcement learning model, and use the robot arm motion control instructions output by the reinforcement learning model to control the robot arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth.

[0083] In this embodiment, the target fabric state characteristics are input into the reinforcement learning model, and the robot arm motion control instructions output by the reinforcement learning model are utilized. Specifically, the flexible gripper is controlled based on the robot arm motion control instructions, and the spatial RPY angle is adjusted according to the reinforcement learning output action to perform a fabric grasping operation on the target fabric. The input data and output actions of the reinforcement learning model can be dynamically adjusted to accommodate fabrics with different physical data.

[0084] Furthermore, when grasping a single layer of fabric, a target detection algorithm is used to detect the current state of the gripped fabric, and error handling is performed based on the detection result. Specifically, in controlling the robotic arm to perform a fabric grasping operation on the target fabric to complete the separation of the single layer of fabric, the process may also include: using a target detection algorithm to detect the current target fabric to obtain a corresponding real-time detection result; based on the real-time detection result, determining whether there is a single layer of fabric grasping failure, and performing corresponding processing operations based on the failure type; wherein the failure type includes a first failure type in which the end of the robotic arm fails to grasp the fabric successfully, and a second failure type in which the end of the robotic arm grasps multiple layers of fabric.

[0085] As can be seen from the above, the embodiments of the present application acquire the distance data, the force feedback data, and the image data, and perform feature extraction on the distance data, the force feedback data, and the image data using a preset reinforcement learning model. This allows for dynamic adjustment of the input data and output actions of the reinforcement learning model for fabrics with different physical data. Furthermore, after performing a fabric grabbing operation, the current fabric state is detected to facilitate abnormality detection and dynamic adjustment. This enables the separation of single-layer fabrics based on reinforcement learning.

[0086] See also Figure 4 As shown, the embodiment of the present application discloses a reinforcement learning model training device for separating a single layer of cloth, comprising:

[0087] A trajectory feature acquisition module 11 is configured to acquire trajectory features in the current environment based on a current policy in the current reinforcement learning model; the current policy is a policy based on the cloth state and the robot arm motion, the trajectory features including cloth state features, robot arm motion features, and an initial reward corresponding to the robot arm motion features; the cloth state features including first feature information of the distance data between the robot arm and the cloth, second feature information of the cloth image data, and third feature information of force feedback data collected when the end of the robot arm contacts the cloth;

[0088] a current total reward generating module 12, configured to add a guidance reward for accelerating the learning process to the initial reward to obtain a current total reward;

[0089] An objective function generation module 13 is configured to construct a corresponding advantage function based on the current total reward and the generalized advantage estimate, and to generate an objective function based on the advantage function and a KL divergence penalty coefficient;

[0090] The strategy parameter optimization module 14 is used to optimize the strategy parameters of the current strategy in the current reinforcement learning model by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

[0091] As can be seen from the above, the embodiments of this application introduce guided rewards to help accelerate the model's learning process and prevent the strategy from falling into a local optimum. The reinforcement learning model's input data and output actions can be dynamically adjusted for different physical data fabrics to capture fabrics with various physical properties. The reinforcement learning model is highly robust and generalizable, and model training can handle a variety of fabrics, resolving the problem of automated separation of single-layer fabrics requiring frequent modifications to related structures and parameters. This makes it possible to separate single-layer fabrics based on reinforcement learning.

[0092] In some specific embodiments, the reinforcement learning model training device for separating a single layer of cloth may further include:

[0093] an initial target data acquisition unit, configured to acquire initial target data through a preset feature extraction network; the initial target data including cloth image data, distance data between the robotic arm and the cloth, and force feedback data collected when the end of the robotic arm contacts the cloth;

[0094] a feature extraction unit, configured to extract features from the initial target data through the preset feature extraction network, and send the extracted cloth state features to the current reinforcement learning model;

[0095] Among them, the preset feature extraction network is configured with a first feature extraction branch and a second feature extraction branch; the first feature extraction branch is used to extract features from the cloth image data, and the second feature extraction branch is used to splice the distance data and the force feedback data, and perform feature extraction on the spliced ​​result; the first feature extraction branch includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer and a first linear layer connected in sequence; the second feature extraction branch includes a second linear layer, a third activation layer and a third linear layer connected in sequence; the preset feature extraction network is also configured with a fourth linear layer, and the fourth linear layer is used to splice the output result of the first linear layer and the output result of the third linear layer to obtain the cloth state feature.

[0096] See also Figure 5 As shown, the embodiment of the present application discloses a single-layer cloth separation device based on a reinforcement learning model, comprising:

[0097] A first data acquisition module 21 is used to acquire distance data between the end of the robotic arm and the target cloth;

[0098] A first control module 22, configured to control the robotic arm to descend based on the distance data;

[0099] a second data acquisition module 23, configured to acquire force feedback data collected by the end of the robotic arm after the end of the robotic arm contacts the target cloth;

[0100] A third data acquisition module 24 is used to acquire the target cloth image data collected by the end of the robotic arm;

[0101] a feature extraction module 25 for performing a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features;

[0102] a second control module 26 for inputting the target cloth state characteristics into the reinforcement learning model and controlling the robot arm to perform a cloth grabbing operation on the target cloth using the robot arm motion control instructions output by the reinforcement learning model, thereby completing the separation of the single layer of cloth;

[0103] Among them, the reinforcement learning model is a model trained by the aforementioned reinforcement learning model training method.

[0104] As can be seen from the above, the embodiments of the present application acquire the distance data, the force feedback data, and the image data, and perform feature extraction on the distance data, the force feedback data, and the image data using a preset reinforcement learning model. This allows for dynamic adjustment of the input data and output actions of the reinforcement learning model for fabrics with different physical data. This enables the separation of single layers of fabric based on reinforcement learning.

[0105] In some specific embodiments, the single-layer cloth separation device based on the reinforcement learning model may further include:

[0106] A cloth detection unit, configured to detect the target cloth using a target detection algorithm to obtain corresponding real-time detection results;

[0107] a detection result judgment unit, configured to judge whether there is a single-layer fabric grabbing failure based on the real-time detection result, and perform corresponding processing operations according to the failure type;

[0108] The failure types include a first failure type in which the end of the robotic arm fails to successfully grasp the cloth and a second failure type in which the end of the robotic arm grasps multiple layers of cloth.

[0109] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 6 This is a structural diagram of an electronic device 30 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device 30 may specifically include: at least one processor 31, at least one memory 32, a power supply 33, a communication interface 34, an input / output interface 35, and a communication bus 36. The memory 32 is used to store a computer program, which is loaded and executed by the processor 31 to implement the relevant steps of the method disclosed in any of the aforementioned embodiments. In addition, the electronic device 30 in this embodiment may specifically be an electronic computer.

[0110] In this embodiment, the power supply 33 is used to provide operating voltage for each hardware device on the electronic device 30; the communication interface 34 can create a data transmission channel between the electronic device 30 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 35 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0111] In addition, the memory 32, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 321, a computer program 322, etc., and the storage method can be temporary storage or permanent storage.

[0112] The operating system 321 is used to manage and control the hardware devices on the electronic device 30 and the computer program 322, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to implement the method performed by the electronic device 30 disclosed in any of the aforementioned embodiments, the computer program 322 can further include computer programs that can be used to implement other specific tasks.

[0113] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0115] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0117] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0118] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A reinforcement learning model training method for separating single-layer fabrics, characterized in that: include: Obtain trajectory features in the current environment based on the current policy in the current reinforcement learning model; The current strategy is a strategy based on the cloth state and the robot arm motion, and the trajectory features include the cloth state features, the robot arm motion features, and the initial reward corresponding to the robot arm motion features; Adding a guidance reward for accelerating the learning process to the initial reward to obtain a current total reward; Constructing an advantage function based on the current total reward and the generalized advantage estimate, and generating an objective function based on the advantage function and a KL divergence penalty coefficient; The policy parameters of the current policy in the current reinforcement learning model are optimized by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

2. The reinforcement learning model training method for separating single-layer fabrics according to claim 1, characterized in that: The cloth state features include first feature information of distance data between the robotic arm and the cloth, second feature information of cloth image data, and third feature information of force feedback data collected when the end of the robotic arm contacts the cloth.

3. The reinforcement learning model training method for separating a single layer of cloth according to claim 2, characterized in that: Also includes: Obtain initial target data through a preset feature extraction network; The initial target data includes cloth image data, distance data between the robotic arm and the cloth, and force feedback data collected when the end of the robotic arm contacts the cloth; Performing feature extraction on the initial target data through the preset feature extraction network, and sending the extracted cloth state features to the current reinforcement learning model; Among them, the preset feature extraction network is configured with a first feature extraction branch and a second feature extraction branch; the first feature extraction branch is used to extract features from the cloth image data, and the second feature extraction branch is used to splice the distance data and the force feedback data, and perform feature extraction on the spliced ​​results.

4. The reinforcement learning model training method for separating a single layer of cloth according to claim 3, characterized in that: The first feature extraction branch includes a first convolutional layer, a first activation layer, a second convolutional layer, a second activation layer, and a first linear layer connected in sequence; the second feature extraction branch includes a second linear layer, a third activation layer, and a third linear layer connected in sequence; The preset feature extraction network is further configured with a fourth linear layer, and the fourth linear layer is used to splice the output result of the first linear layer and the output result of the third linear layer to obtain the cloth state feature.

5. A single-layer cloth separation method based on reinforcement learning model, characterized in that: The reinforcement learning model is a model trained by the reinforcement learning model training method according to any one of claims 1 to 4; wherein the method comprises: Acquiring distance data between the end of the robotic arm and the target cloth, and controlling the robotic arm to descend based on the distance data; When the end of the robotic arm contacts the target cloth, force feedback data collected by the end of the robotic arm is obtained; Acquiring target cloth image data collected by the end of the robotic arm, and performing a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features; The target cloth state characteristics are input into the reinforcement learning model, and the robot arm motion control instructions output by the reinforcement learning model are used to control the robot arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth.

6. The single-layer cloth separation method based on the reinforcement learning model according to claim 5 is characterized in that: The process of controlling the robotic arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth further includes: Detecting the target fabric using a target detection algorithm to obtain corresponding real-time detection results; Determining whether there is a single-layer fabric grabbing failure based on the real-time detection results, and performing corresponding processing operations according to the failure type; The failure types include a first failure type in which the end of the robotic arm fails to successfully grasp the cloth and a second failure type in which the end of the robotic arm grasps multiple layers of cloth.

7. A reinforcement learning model training device for separating single-layer fabrics, characterized in that: include: The trajectory feature acquisition module is used to obtain the trajectory features in the current environment based on the current strategy in the current reinforcement learning model; The current strategy is a strategy based on the cloth state and the robot arm motion, and the trajectory features include the cloth state features, the robot arm motion features, and the initial reward corresponding to the robot arm motion features; a current total reward generating module, configured to add a guidance reward for accelerating the learning process to the initial reward to obtain a current total reward; An objective function generation module, configured to construct a corresponding advantage function based on the current total reward and the generalized advantage estimate, and to generate an objective function based on the advantage function and a KL divergence penalty coefficient; A strategy parameter optimization module is used to optimize the strategy parameters of the current strategy in the current reinforcement learning model by maximizing the objective function to obtain a target reinforcement learning model for separating a single layer of cloth.

8. A single-layer cloth separation device based on a reinforcement learning model, characterized in that: The reinforcement learning model is a model trained by the reinforcement learning model training method according to any one of claims 1 to 4; wherein the device comprises: A first data acquisition module is used to acquire distance data between the end of the robotic arm and the target cloth; A first control module, configured to control the robotic arm to descend based on the distance data; a second data acquisition module, configured to acquire force feedback data collected by the end of the robotic arm after the end of the robotic arm contacts the target cloth; a third data acquisition module, configured to acquire target cloth image data collected by the end of the robotic arm; a feature extraction module, configured to perform a feature extraction operation on the distance data, the force feedback data, and the target cloth image data to obtain target cloth state features; The second control module is used to input the target cloth state characteristics into the reinforcement learning model, and use the robot arm motion control instructions output by the reinforcement learning model to control the robot arm to perform a cloth grabbing operation on the target cloth to complete the separation of the single layer of cloth.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images