A self-learning metal plate bending method based on multi-modal fusion

Abstract

The application discloses a kind of self-learning touch bionic sheet metal bending method based on multi-modal fusion, it is related to sheet metal bending technical field, the method includes: using each minimum loss function value corresponding model parameter to carry out parameter updating to preset learning network model, obtain the sheet metal bending model corresponding to training completion and preset learning network model;First multi-modal data in the initial moment of the process that sheet metal is bent by bending robot is acquired, and first multi-modal data is input into sheet metal bending model and is handled, and the bending action of initial moment is obtained;Control bending robot executes bending action, obtain second multi-modal data after executing bending action, and second multi-modal data is input into sheet metal bending model and is handled, until sheet metal completes bending process, introduce force touch system and visual system are combined, to reach the purpose of adapting environmental complexity, perceiving material own characteristic and adapting extensive range etc.

Description

Technical Field

[0001] This invention relates to the field of sheet metal bending technology, and more specifically, to a self-learning tactile bionic sheet metal bending method based on multimodal fusion. Background Technology

[0002] Currently, sheet metal parts are widely used in various fields such as electronics, automotive, and medical due to their advantages of good electrical conductivity, low cost, and high strength. With manufacturing enterprises placing increasing emphasis on high quality, high efficiency, and high standards, digital and intelligent production robots have emerged. To increase sheet metal part output faster and better, a large number of sheet metal bending robots have entered the market, replacing workers in time-consuming tasks.

[0003] To address the inherent springback issue in sheet metal materials during bending, traditional engineering practices rely on operators' tactile experience and multiple trial bends to eliminate or reduce the impact of springback deformation. This repeated experimentation helps control bending precision. Currently, robot sensing technologies primarily employ vision and force sensors, which suffer from drawbacks such as high computational demands, low robustness, and sensitivity to environmental and operational conditions. Research on the application of tactile sensors and their multimodal integration with vision and force sensors is still limited. Furthermore, with the increasing variety of sheet metal parts, most bending robots today struggle to achieve efficient flexible processing, and the repeated trial-and-error processes can lead to machine and material damage and waste. Summary of the Invention

[0004] The purpose of this invention is to provide a self-learning tactile bionic sheet metal bending method based on multimodal fusion. It combines vision, end effector force-torque, and tactile sensors. The vision sensor determines the thickness and shape of the sheet metal part, and the tactile sensor determines the material's inherent properties and hardening degree. The end effector force-torque sensor then determines the appropriate bending force and bending angle, enabling the robot to acquire the same level of experience as a worker bending the material by touch, thus achieving more standardized processing accuracy.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] Firstly, this application provides a self-learning tactile bionic sheet metal bending method based on multimodal fusion, including the following specific steps:

[0007] Acquire teaching multimodal data during the bending process of sheet metal parts under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data.

[0008] The pre-built learning network model is trained using multiple transition tuples in the teaching multimodal data, and the minimum loss function value of each transition tuple in the pre-built learning network model is calculated.

[0009] The parameters of the pre-learning network model are updated using the model parameters corresponding to each minimum loss function value, resulting in a trained sheet metal bending model that corresponds to the pre-learning network model.

[0010] The first multimodal data of the bending robot at the initial moment during the bending of sheet metal parts is obtained, and the first multimodal data is input into the sheet metal bending model for processing to obtain the bending action at the initial moment.

[0011] The bending robot is controlled to perform bending actions, and the second multimodal data after the bending action is performed is obtained. The second multimodal data is then input into the sheet metal bending model for processing until the sheet metal part completes the bending process.

[0012] The beneficial effects of this invention are as follows: In this solution, firstly, multimodal data of the sheet metal bending process is acquired through manual instruction. Then, the pre-set learning network model is trained using the multimodal data until training is complete. The trained sheet metal bending model is then used to perform sheet metal bending. When bending the sheet metal using the bending model, first or second multimodal data is continuously acquired during the bending process, and this acquired first or second multimodal data is input into the sheet metal bending model for processing. The bending action corresponding to the first or second multimodal data is obtained and repeated. That is, after obtaining the first multimodal data at the initial moment during the bending process of the sheet metal part, the bending action corresponding to the first multimodal data is executed. Then, the second multimodal data after the bending action corresponding to the first multimodal data is obtained, and the bending action corresponding to the second multimodal data is executed. Then, new second multimodal data is obtained, and the bending action corresponding to the new second multimodal data is executed. This process is repeated until the bending process of the sheet metal part is completed.

[0013] In this solution, vision, currently the primary means of perception for robots, cannot comprehensively acquire relevant information when faced with diverse environmental information. Furthermore, in high-precision controlled processing experiments, visual perception alone is insufficient for judging the inherent properties and mechanical performance of materials. Therefore, this solution combines a force-tactile system with a vision system to achieve adaptability to environmental complexity, perception of material properties, and a wide range of applicability. A worker-led preprocessing approach is adopted, enabling the robot to calculate the loss value of the learning network using acquired modal data and reward values ​​during the initial bending exploration phase. This allows for further optimization of the strategy and value model parameters through backpropagation. This preprocessing method also reduces the risk of machine and material damage caused by trial and error compared to random initialization. In addition, the entropy regularization coefficient in the algorithm is dynamically improved so that each transition tuple corresponds to an entropy regularization coefficient, which changes continuously according to the training situation, improving the randomness of the training process and preventing the optimization result from getting trapped in a local optimum too early. As training continues, the experience database will also be continuously optimized and iterated, and better solutions will gradually replace old data, thereby realizing the robot's autonomous learning function.

[0014] Based on the above technical solution, the present invention can be further improved as follows.

[0015] Furthermore, the above applies to the teaching multimodal data:

[0016] The teaching visual data is acquired through a visual sensor, and the teaching visual data characterizes the thickness and deformation trend of the sheet metal part.

[0017] The teaching force data is acquired through a force sensor, and the teaching force data characterizes the pressure value fed back by the sheet metal part at the end of the bending mold during the deformation process;

[0018] The teaching tactile data is acquired through tactile sensors, and the teaching tactile data characterizes the texture and change process of the outer wall of the sheet metal part during the deformation process.

[0019] The beneficial effects of adopting the above-mentioned further solutions are: to combine the force-tactile system with the visual system in order to achieve the goals of adapting to environmental complexity, perceiving the characteristics of the materials themselves, and having a wide range of applicability.

[0020] Furthermore, the aforementioned tactile sensors include variable dielectric flexible capacitive tactile sensors and variable electrode spacing flexible capacitive tactile sensors. Both the variable dielectric flexible capacitive tactile sensors and the variable electrode spacing flexible capacitive tactile sensors reflect the texture and changes of the outer wall of the sheet metal part during the deformation process through the total capacitance value and its change process.

[0021] The beneficial effects of adopting the above-mentioned further solution are as follows: For tactile sensing, which requires high sensitivity, simple results, and good dynamic response, as well as the requirements of material property judgment, high pressure, and high precision in bending operations, a flexible capacitive tactile sensor is selected. When operators perform bending operations, they mainly rely on their experience with the surface texture or material type of the sheet metal and the processing pressure. To better enable the robot to achieve the same level of bending experience as the operator by introducing a tactile sensor, this solution mainly involves inputting data into the experience database from two aspects: the variable dielectric and the variable electrode spacing of the capacitive tactile sensor.

[0022] Furthermore, the total capacitance value of the aforementioned variable dielectric flexible capacitive tactile sensor is specifically as follows:

[0023]

[0024] In the formula, C represents the total capacitance value; C1 represents the capacitance value from the upper electrode of the variable dielectric flexible capacitive tactile sensor to the top of the sheet metal part; C2 represents the capacitance value from the lower electrode of the variable dielectric flexible capacitive tactile sensor to the bottom of the sheet metal part; A represents the coverage area between the two electrodes in the variable dielectric flexible capacitive tactile sensor; ε0 represents the dielectric constant of vacuum, ε0=8.854*10 -12 F / m;ε r ε represents the relative permittivity of the medium between the plates. For air, ε is... r =1; δ represents the distance between the plates; d represents the thickness of the sheet metal part;

[0025] The total capacitance value of the variable plate spacing type flexible capacitive tactile sensor is as follows:

[0026]

[0027] In the formula, C represents the total capacitance, A represents the coverage area between the two plates in the variable plate spacing flexible capacitive tactile sensor, C0 represents the initial capacitance of the variable plate spacing flexible capacitive tactile sensor, d0 represents the initial plate spacing of the variable plate spacing flexible capacitive tactile sensor, and ε0 represents the dielectric constant of vacuum, ε0=8.854*10 -12 F / m;ε r ε represents the relative permittivity of the medium between the plates. For air, ε is... r =1; when the distance between the plates decreases by Δd, d = d0 - Δd.

[0028] The beneficial effects of adopting the above-mentioned further solution are as follows: This solution utilizes the principle of variable medium and variable electrode spacing of capacitive tactile sensors. Based on the texture or type and thickness of the material gripped by the robot during bending, the capacitance value under different conditions can be calculated, and the output information is input into the bending springback experience library. The bending springback experience library is used to store teaching multimodal data. Through multiple manual teachings, the robot can continuously optimize and adjust its processing state in future processing by using feedback from the bending springback experience library. This solution introduces a bionic tactile sensor, combined with the high precision and high sensitivity of flexible capacitive sensors, and utilizes its two major principles of variable medium and variable electrode spacing to make the bending robot approach "skin-like", further matching the bending experience of bending operators.

[0029] Furthermore, the aforementioned pre-built learning network model includes two Critic networks and one Actor network. The specific method for calculating the minimum loss function value of each transition tuple in the pre-built learning network model is as follows: The target network value is calculated using the target network, and then the minimum loss function value of each transition tuple is calculated using the target network value. Each transition tuple is represented as: (s t ,a t ,r t ,s t+1 );

[0030] In the formula, s t This indicates the acquired teaching multimodal data, or the first multimodal data, or the second multimodal data, a. t Indicates a bending action, r t This indicates that a reward is given for performing the bending action, s t+1 This indicates that the bending action a is performed at time step t. t The subsequent teaching multimodal data or second multimodal data.

[0031] The beneficial effects of adopting the above-mentioned further scheme are as follows: The deep learning algorithm SAC is used for interactive cognition with the environment. The SAC algorithm can effectively solve reinforcement learning problems in discrete or continuous action spaces, i.e., the training process using two Critic networks and one Actor network. The introduction of SAC's dual evaluation function and maximum entropy can effectively solve the problems of overestimation value of a single evaluation function and premature trapping in local optima. Furthermore, to avoid excessive trial-and-error in the early stages of bending, this scheme adopts a worker-led teaching method, acquiring relevant modal data through sensors to pre-initialize the springback experience database, thereby improving the exploration rate in the early stages of bending. Specifically, the Actor network makes decisions based on three modalities (vision, touch, and action) through sensors to take actions, while the Critic network evaluates and predicts future situations based on the current state and the upcoming action. By continuously optimizing decisions and evaluations, the optimal strategy is learned.

[0032] Furthermore, the target network value mentioned above is specifically as follows:

[0033] y i =r i +γmin j＝1，2 Q θj (s i+1 a i+1 )-αlogπ(a i+1 |s i+1 );

[0034] In the formula, y i Represents the target network value, r i Let represent the reward for the i-th transition tuple performing the bending action, γ represent the discount factor, j represent the index of the Critic network, and Q represent the reward. θj (s i+1 a i+1 ) represents the Critic network evaluation value of the (i+1)th transition tuple, α represents the entropy regularization coefficient, and π(a i+1 |s i+1 ) represents the strategy preferred by the state in the (i+1)th transition tuple, and θ represents the network parameters of the Critic network.

[0035] Furthermore, the specific value of the aforementioned minimum loss function is as follows:

[0036]

[0037] In the formula, y i Let minL represent the target network value of the i-th transition tuple, minL represent the minimum loss function value, N represent the number of transition tuples, and Q represent the target network value of the i-th transition tuple. θj (s i a i ) represents the Critic network evaluation value of the i-th transition tuple, θ represents the network parameters of the Critic network, and j represents the index of the Critic network.

[0038] Secondly, embodiments of this application provide a self-learning tactile bionic sheet metal bending system based on multimodal fusion, applied to any of the self-learning tactile bionic sheet metal bending methods based on multimodal fusion in the first aspect, comprising:

[0039] The first module is used to acquire teaching multimodal data during the bending process of sheet metal parts under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data.

[0040] The second module is used to train a pre-built learning network model using multiple transition tuples in the teaching multimodal data, and to calculate the minimum loss function value of each transition tuple in the pre-built learning network model.

[0041] The third module is used to update the parameters of the pre-learning network model using the model parameters corresponding to each minimum loss function value, so as to obtain the sheet metal bending model that has been trained and corresponds to the pre-learning network model.

[0042] The fourth module is used to acquire the first multimodal data of the bending robot at the initial moment during the bending process of the sheet metal part, and input the first multimodal data into the sheet metal part bending model for processing to obtain the bending action at the initial moment;

[0043] The fifth module is used to control the bending robot to perform bending actions, acquire the second multimodal data after the bending action is performed, and input the second multimodal data into the sheet metal bending model for processing until the sheet metal part completes the bending process.

[0044] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of the first aspects.

[0045] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to perform any of the methods in the first aspect.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects:

[0047] In this application, firstly, multimodal teaching data of the sheet metal bending process is acquired through manual instruction. Then, a pre-set learning network model is trained using this multimodal teaching data until training is complete. The trained sheet metal bending model is then used to perform sheet metal bending. When bending the sheet metal using the bending model, first or second multimodal data is continuously acquired during the bending process. This acquired first or second multimodal data is then input into the sheet metal bending model for processing, thus obtaining data similar to the first multimodal data. The bending action corresponding to the multimodal data or the second multimodal data is performed in a loop. That is, after obtaining the first multimodal data at the initial moment during the bending process of the sheet metal part, the bending action corresponding to the first multimodal data is executed. Then, the second multimodal data after the bending action corresponding to the first multimodal data is obtained, and the bending action corresponding to the second multimodal data is executed. Then, new second multimodal data is obtained, and the bending action corresponding to the new second multimodal data is executed. This process is repeated until the bending process of the sheet metal part is completed.

[0048] In this application, vision, as the primary perception method for robots, cannot comprehensively acquire relevant information when faced with diverse environmental information. Furthermore, in high-precision controlled processing experiments, visual perception alone is insufficient for judging the material's inherent characteristics and mechanical properties. Therefore, this solution combines a force-tactile system with a vision system to achieve adaptability to environmental complexity, perception of material properties, and a wide range of applicability. A worker-led preprocessing method is adopted, enabling the robot to calculate the loss value of the learning network using acquired modal data and reward values ​​during the initial bending exploration phase, and further optimize the strategy and value model parameters through backpropagation. This preprocessing method also reduces the risk of machine and material damage caused by trial and error compared to random initialization. In addition, the entropy regularization coefficient α in the algorithm is dynamically improved so that each transition tuple corresponds to an entropy regularization coefficient α, which changes continuously according to the training situation, improving the randomness of the training process and preventing the optimization result from getting trapped in a local optimum too early. As training continues, the experience database will also be continuously optimized and iterated, and better solutions will gradually replace old data, thereby realizing the robot's autonomous learning function.

[0049] In this application, a flexible capacitive tactile sensor is selected to address the requirements of high sensitivity, simple results, good dynamic response, and the need for material property judgment, high pressure, and high precision in bending operations. Since operators rely primarily on their experience with the surface texture or material type of sheet metal and processing pressure during bending operations, this solution aims to better enable the robot to match the bending experience of operators by introducing a tactile sensor. The solution primarily focuses on inputting data into the experience database from two aspects: variable dielectric and variable electrode spacing of the capacitive tactile sensor. By improving existing vision-force bending robots and introducing a biomimetic flexible capacitive tactile sensor, the multimodal signals of vision-force-tactile sensing are initialized into the springback experience database through manual teaching. This allows for continuous pose updates and iterative learning between the experience database and the robot, ultimately enabling the robot to independently complete multiple bending tasks and match the bending experience of skilled operators, thereby improving the processing efficiency and yield of military electronic sheet metal parts.

[0050] In this application, the proposed solution utilizes the principles of variable dielectric and variable electrode spacing of capacitive tactile sensors. Based on the texture or type and thickness of the material gripped by the robot during bending, the capacitance value under different conditions can be calculated, and the output information is input into the bending springback experience library. The bending springback experience library is used to store multimodal teaching data. Through multiple manual teaching sessions, the robot can continuously optimize and adjust its processing state in future processing based on feedback from the bending springback experience library. This solution introduces a bionic tactile sensor, combining the high precision and high sensitivity of flexible capacitive sensors, and utilizes its two main principles of variable dielectric and variable electrode spacing to make the bending robot approach "skin-like" and further match the bending experience of bending operators.

[0051] In this application, the deep learning algorithm SAC is used to interact and cognize with the environment. The SAC algorithm can effectively solve the reinforcement learning problem in discrete or continuous action spaces, that is, the training process of two Critic networks and one Actor network. The introduction of dual evaluation functions and maximum entropy in SAC can effectively solve the problems of overestimation value of single evaluation function and premature getting trapped in local optima. Furthermore, in order to avoid excessive trial and error in the early stage of bending, this scheme adopts the worker teaching method, and uses sensors to acquire relevant modal data to initialize the springback experience database in advance, thereby improving the exploration rate in the early stage of bending. Attached Figure Description

[0052] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0053] Figure 1This is a flowchart of the sheet metal bending method in an embodiment of the present invention;

[0054] Figure 2 This is a connection diagram of the sheet metal bending system in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the bending process of the bending robot in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram illustrating the principle of the variable dielectric flexible capacitive tactile sensor in an embodiment of the present invention.

[0057] Figure 5 This is a schematic diagram illustrating the principle of a variable electrode spacing type flexible capacitive tactile sensor in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] In the description of the embodiments of the present invention, "multiple" means at least two.

[0062] Example 1:

[0063] To enable robots to acquire the tactile feedback of workers during bending, thus achieving more standardized processing accuracy, a combination of vision, end-effector force-torque, and tactile sensors is used. The vision sensor determines the thickness and shape of the sheet metal part, while the tactile sensor assesses the material's inherent properties and hardening degree. This, in turn, allows the end-effector force-torque sensor to determine the appropriate bending force and angle. This embodiment provides a self-learning tactile bionic sheet metal bending method based on multimodal fusion. Figure 1 As shown, the specific steps include the following:

[0064] S1, acquire teaching multimodal data during the bending process of sheet metal parts under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data.

[0065] The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data, all of which are acquired through corresponding sensors, such as... Figure 3 As shown, the teaching visual data is acquired through a vision sensor. This data characterizes the material type, thickness, and deformation trend of the sheet metal part. The location of the vision sensor is shown in [reference needed]. Figure 3 ,exist Figure 3 In the diagram, the black bar at the bottom of the vision sensor represents the sheet metal part before bending. The vision sensor can acquire parameters such as the deformation trend of the upper surface of the sheet metal part before bending. The teaching force data is acquired through a force sensor, which characterizes the pressure value fed back by the sheet metal part at the end of the bending mold during the deformation process. Figure 3 As shown, in Figure 3 In the bending die on the right, the upper and lower parts constitute the bending die. It can be seen that the sheet metal part formed by this die is "V" shaped. During the bending process, that is, during the process of gradually forming the "V" sheet metal part, the two ends of the upper part of the "V" sheet metal part will have a reaction force, that is, the pressure value fed back by the sheet metal part at the end of the bending die. When bending, the operator will apply an opposite force to this reaction force. This is to make the formed sheet metal part more accurate. Therefore, this reaction force is obtained by force sensor.

[0066] The teaching tactile data is acquired through tactile sensors, which characterize the texture and changes of the outer wall of the sheet metal part during deformation. Specifically, utilizing the principle of variable dielectric and variable electrode spacing of capacitive tactile sensors, the capacitance value for different situations can be calculated based on the texture or type and thickness of the material gripped by the robot during bending. The output information is then input into the bending springback experience library. Through multiple manual teaching sessions, the robot can continuously optimize and adjust its processing state, i.e., execute actions, based on feedback from the springback experience library during future processing.

[0067] Optionally, the aforementioned tactile sensors include variable dielectric flexible capacitive tactile sensors and variable electrode spacing flexible capacitive tactile sensors. Both the variable dielectric flexible capacitive tactile sensors and the variable electrode spacing flexible capacitive tactile sensors reflect the texture and changes of the outer wall of the sheet metal part during the deformation process through the total capacitance value and its change process.

[0068] Furthermore, the schematic diagram of the aforementioned variable dielectric flexible capacitive tactile sensor can be found here. Figure 4The total capacitance value of the variable dielectric flexible capacitive tactile sensor is as follows:

[0069]

[0070] In the formula, C represents the total capacitance value; C1 represents the capacitance value from the upper electrode of the variable dielectric flexible capacitive tactile sensor to the top of the sheet metal part; C2 represents the capacitance value from the lower electrode of the variable dielectric flexible capacitive tactile sensor to the bottom of the sheet metal part; A represents the coverage area between the two electrodes in the variable dielectric flexible capacitive tactile sensor; ε0 represents the dielectric constant of vacuum, ε0=8.854*10 -12 F / m;ε r ε represents the relative permittivity of the medium between the plates. For air, ε is... r =1; δ represents the distance between the plates; d represents the thickness of the sheet metal part.

[0071] Among them, such as Figure 4 The parameters shown are: A is the coverage area between the two plates; ε0 is the dielectric constant of vacuum, ε0 = 8.854 * 10⁻⁶. -12 F / m;ε r ε is the relative permittivity of the medium between the plates. For air, ε r =1; δ is the distance between the plates; d is the thickness of the sheet metal part; Assume: C1 is the capacitance from the upper plate to the top of the sheet metal part; C2 is the capacitance from the lower plate to the bottom of the sheet metal part; C is the total capacitance after the two capacitances are connected in series; then the capacitance from the upper plate to the top of the sheet metal part is: The capacitance value from the lower electrode plate to the bottom of the sheet metal part: The total capacitance value is:

[0072]

[0073] Furthermore, the schematic diagram of the aforementioned variable electrode spacing type flexible capacitive tactile sensor can be found here. Figure 5 The total capacitance value of the variable electrode spacing type flexible capacitive tactile sensor is as follows:

[0074]

[0075] In the formula, C represents the total capacitance, A represents the coverage area between the two plates in the variable plate spacing flexible capacitive tactile sensor, C0 represents the initial capacitance of the variable plate spacing flexible capacitive tactile sensor, d0 represents the initial plate spacing of the variable plate spacing flexible capacitive tactile sensor, and ε0 represents the dielectric constant of vacuum, ε0=8.854*10 -12 F / m;ε r ε represents the relative permittivity of the medium between the plates. For air, ε is... r =1; when the distance between the plates decreases by Δd, d = d0 - Δd.

[0076] Among them, such as Figure 5 The parameters shown are: the coverage area A between the two plates, the dielectric constant ε0 of the vacuum, and the relative dielectric constant ε of the medium between the plates. r Similar to the variable dielectric type flexible capacitive tactile sensor, d is the distance between the plates; assuming: the initial capacitance is C0, and the initial plate spacing is d0, then the initial capacitance value is: When the distance between the plates decreases by Δd, d = d0 - Δd; then the total capacitance is: C = C0 + ΔC; Additionally... when When (generally taken as 0.02-0.1), higher-order terms are ignored. Sensor sensitivity: And nonlinear error:

[0077] Specifically, utilizing the principles of variable dielectric and variable electrode spacing of capacitive tactile sensors, the capacitance value under different conditions can be calculated based on the texture or type and thickness of the material gripped by the robot during bending. The output information is then input into the bending springback experience library, which stores multimodal teaching data. Through repeated manual teaching, the robot can continuously optimize and adjust its processing state based on feedback from the bending springback experience library during future processing. This solution introduces a biomimetic tactile sensor, combining the high precision and high sensitivity of flexible capacitive sensors, and utilizes its two main principles of variable dielectric and variable electrode spacing to make the bending robot approach "skin-like" and further match the bending experience of the bending operator.

[0078] S2 trains the pre-built learning network model using multiple transition tuples in the teaching multimodal data and calculates the minimum loss function value for each transition tuple in the pre-built learning network model.

[0079] Optionally, the above-mentioned pre-built learning network model includes two Critic networks and one Actor network. The specific method for calculating the minimum loss function value of each transition tuple in the pre-built learning network model is as follows: The target network value is calculated using the target network, and the minimum loss function value of each transition tuple is then calculated using the target network value, where each transition tuple is represented as: (s t ,a t ,r t ,s t+1 );

[0080] In the formula, s t This indicates the acquired teaching multimodal data, or the first multimodal data, or the second multimodal data, a. t Indicates a bending action, r t This indicates that a reward is given for performing the bending action, st+1 This indicates that the bending action a is performed at time step t. t The subsequent teaching multimodal data or second multimodal data.

[0081] The solution employs the deep learning algorithm SAC to interact and cognize with the environment. SAC effectively solves reinforcement learning problems in discrete or continuous action spaces, using a training process involving two Critic networks and one Actor network. The introduction of dual evaluation functions and maximum entropy in SAC effectively addresses the issues of overestimation by a single evaluation function and premature entrapment in local optima. Furthermore, to avoid excessive trial-and-error in the initial bending phase, this solution uses worker-led instruction, acquiring relevant modal data through sensors to pre-initialize the springback experience database, thereby improving the exploration rate in the early bending stages. The Actor network makes decisions based on three modalities (vision, touch, and action) via sensors, while the Critic network evaluates and predicts future actions based on the current state and the impending action. Through continuous optimization of decisions and evaluations, the optimal strategy is learned.

[0082] Specifically, the steps can be as follows: 1. Initialize the Critic network Q using random network parameters θ1, θ2, and φ respectively. θ1 (s,a), Q θ2 (s,a) and Actor network π φ (s); 2. Copy the same parameters Initialize the objective function Q respectively θ1 (s,a), Q θ2 (s,a); 3. Initialize the experience database R and the worker teaching database R` to be empty. The worker demonstrates bending, and the sensor continuously acquires modal data and stores it in the worker teaching database R`; 4. Copy the worker teaching database R` into the experience database R: R←R`; 5. Acquire the environmental state s through the sensor. t Select action a based on the current strategy t =π φ (s t ), perform action a t Receive reward r t The environmental state becomes s t+1 6. Sample N tuples {(s) from R i ,a i ,r i ,s i+1 )} i＝1,...,N For each tuple, compute using the target network:

[0083] y i =r i +γmin j＝1，2 Qθj (s i+1 a i+1 )-αlogπ(a i+1 |s i+1 ), where a i+1 ~π φ (·|s i+1 7. Update the two Critic networks as follows: For j=1,2, minimize the loss function.

[0084] S3. Update the parameters of the pre-learning network model using the model parameters corresponding to each minimum loss function value to obtain the trained sheet metal bending model that corresponds to the pre-learning network model.

[0085] Specifically, by using the model parameters corresponding to each transition tuple when calculating the minimum loss function value, the network parameters of the two Critic networks and one Actor network in the pre-learned network model are updated, so that each transition tuple will correspond to a set of model parameters.

[0086] Optionally, the target network value mentioned above is specifically:

[0087] y i =r i +γmin j＝1，2 Q θj (s i+1 a i+1 )-αlogπ(a i+1 |s i+1 );

[0088] In the formula, y i Represents the target network value, r i Let represent the reward for the i-th transition tuple performing the bending action, γ represent the discount factor, j represent the index of the Critic network, and Q represent the reward. θj (s i+1 a i+1 ) represents the Critic network evaluation value of the (i+1)th transition tuple, α represents the entropy regularization coefficient, and π(a i+1 |s i+1 ) represents the strategy preferred by the state in the (i+1)th transition tuple, and θ represents the network parameters of the Critic network.

[0089] Optionally, the minimum loss function value mentioned above is specifically:

[0090]

[0091] In the formula, y iLet minL represent the target network value of the i-th transition tuple, minL represent the minimum loss function value, N represent the number of transition tuples, and Q represent the target network value of the i-th transition tuple. θj (s i a i ) represents the Critic network evaluation value of the i-th transition tuple, θ represents the network parameters of the Critic network, and j represents the index of the Critic network.

[0092] S4: Acquire the first multimodal data of the bending robot at the initial moment during the bending process of the sheet metal part, and input the first multimodal data into the sheet metal part bending model for processing to obtain the bending action at the initial moment; the bending action can include orientation control and force application control, and the orientation includes up, down, left, and right, for example: Figure 3 During the bending process shown by the bending robot, Figure 3 The robot in the image is a robotic arm, with various sensors installed at the end of the arm. Based on this type of robot, bending actions can be the rotation angle and bending angle of the first joint, the rotation angle and bending angle of the second joint, the rotation angle and bending angle of the third joint, etc.

[0093] S5 controls the bending robot to perform bending actions, acquires the second multimodal data after the bending action, and inputs the second multimodal data into the sheet metal bending model for processing until the sheet metal part completes the bending process.

[0094] Specifically, after acquiring the first multimodal data at the initial moment during the bending process of the sheet metal part, the bending action corresponding to the first multimodal data is executed. Then, the second multimodal data after the bending action corresponding to the first multimodal data is acquired, and the bending action corresponding to the second multimodal data is executed. Then, new second multimodal data is acquired, and the bending action corresponding to the new second multimodal data is executed. This process is repeated until the bending process of the sheet metal part is completed.

[0095] Example 2:

[0096] This application provides a self-learning tactile bionic sheet metal bending system based on multimodal fusion, applicable to any of the self-learning tactile bionic sheet metal bending methods based on multimodal fusion in Embodiment 1, such as... Figure 2 As shown, it includes:

[0097] The first module is used to acquire teaching multimodal data during the bending process of sheet metal parts under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data.

[0098] The second module is used to train a pre-built learning network model using multiple transition tuples in the teaching multimodal data, and to calculate the minimum loss function value of each transition tuple in the pre-built learning network model.

[0099] The third module is used to update the parameters of the pre-learning network model using the model parameters corresponding to each minimum loss function value, so as to obtain the trained sheet metal bending model that corresponds to the pre-learning network model.

[0100] The fourth module is used to acquire the first multimodal data of the bending robot at the initial moment during the bending process of the sheet metal part, and input the first multimodal data into the sheet metal part bending model for processing to obtain the bending action at the initial moment.

[0101] The fifth module is used to control the bending robot to perform bending actions, acquire the second multimodal data after the bending action is performed, and input the second multimodal data into the sheet metal bending model for processing until the sheet metal part completes the bending process.

[0102] Example 3:

[0103] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method of any one of Embodiment 1.

[0104] Example 4:

[0105] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to perform any of the methods in Embodiment 1.

[0106] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A self-learning tactile bionic sheet metal bending method based on multimodal fusion, characterized in that, The specific steps include the following: Acquire teaching multimodal data during the bending process of a sheet metal part under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data. Among the teaching multimodal data: the teaching visual data is acquired through a visual sensor, and the teaching visual data characterizes the thickness and deformation trend of the sheet metal part. The teaching force data is acquired through a force sensor, which characterizes the pressure value fed back by the sheet metal part at the end of the bending die during the deformation process; the teaching tactile data is acquired through a tactile sensor, which characterizes the texture and its change process of the outer wall of the sheet metal part during the deformation process. The pre-built learning network model is trained using multiple transition tuples in the teaching multimodal data, and the minimum loss function value of each transition tuple in the pre-built learning network model is calculated. The model parameters corresponding to each of the minimum loss function values ​​are used to update the parameters of the pre-set learning network model to obtain a sheet metal bending model that has been trained and corresponds to the pre-set learning network model. The first multimodal data of the bending robot at the initial moment during the bending process of the sheet metal part is obtained, and the first multimodal data is input into the bending model of the sheet metal part for processing to obtain the bending action at the initial moment; The bending robot is controlled to perform the bending action, and the second multimodal data after the bending action is performed is obtained. The second multimodal data is then input into the sheet metal bending model for processing until the sheet metal part completes the bending process.

2. The self-learning tactile bionic sheet metal bending method based on multimodal fusion according to claim 1, characterized in that, The tactile sensors include a variable dielectric flexible capacitive tactile sensor and a variable electrode spacing flexible capacitive tactile sensor. Both the variable dielectric flexible capacitive tactile sensor and the variable electrode spacing flexible capacitive tactile sensor reflect the texture and change process of the outer wall of the sheet metal part during the deformation process through the total capacitance value and its change process.

3. The self-learning tactile bionic sheet metal bending method based on multimodal fusion according to claim 2, characterized in that, The total capacitance value of the variable dielectric flexible capacitive tactile sensor is specifically as follows: ； In the formula, C represents the total capacitance value, C1 represents the capacitance value from the upper electrode of the variable dielectric flexible capacitive tactile sensor to the top of the sheet metal part, C2 represents the capacitance value from the lower electrode of the variable dielectric flexible capacitive tactile sensor to the bottom of the sheet metal part, and A represents the coverage area between the two electrodes in the variable dielectric flexible capacitive tactile sensor. The dielectric constant of vacuum. ; This represents the relative permittivity of the medium between the plates. For air, ; d represents the distance between the plates; d represents the thickness of the sheet metal part. The total capacitance value of the variable electrode spacing type flexible capacitive tactile sensor is specifically as follows: ； In the formula, C represents the total capacitance value, A represents the coverage area between the two plates in the variable plate spacing flexible capacitive tactile sensor, C0 represents the initial capacitance of the variable plate spacing flexible capacitive tactile sensor, and d0 represents the initial plate spacing of the variable plate spacing flexible capacitive tactile sensor. The dielectric constant of vacuum. ; This represents the relative permittivity of the medium between the plates. For air, When the distance between the plates decreases by Δd, d = d0 - Δd.

4. The self-learning tactile bionic sheet metal bending method based on multimodal fusion according to claim 1, characterized in that, The pre-built learning network model includes two Critic networks and one Actor network. The calculation of the minimum loss function value for each transition tuple in the pre-built learning network model specifically involves: calculating the target network value using the target network, and then calculating the minimum loss function value for each transition tuple using the target network value, wherein each transition tuple is represented as: ; In the formula, This indicates the acquired teaching multimodal data, or the first multimodal data, or the second multimodal data. Indicates a bending action. This indicates that a reward is given for performing the bending action. This indicates that the bending action is performed at time step t. The subsequent teaching multimodal data or second multimodal data.

5. The self-learning tactile bionic sheet metal bending method based on multimodal fusion according to claim 4, characterized in that, The target network value is specifically: ； In the formula, Indicates the target network value. This represents the reward received by the i-th transition tuple for performing the bending action. denoted by the discount factor, and j represents the index of the Critic network. This represents the Critic network evaluation value of the (i+1)th transition tuple. This represents the coefficient of the entropy regularization term. This represents the preferred strategy for the state in the (i+1)th transition tuple. This represents the network parameters of the Critic network.

6. The self-learning tactile bionic sheet metal bending method based on multimodal fusion according to claim 5, characterized in that, The specific value of the minimum loss function is: ； In the formula, Let L represent the target network value of the i-th transition tuple, and minL represent the minimum loss function value. Indicates the number of transition tuples. This represents the Critic network evaluation value of the i-th transition tuple. This represents the network parameters of the Critic network, and j represents the index of the Critic network.

7. A self-learning tactile bionic sheet metal bending system based on multimodal fusion, applied to the self-learning tactile bionic sheet metal bending method based on multimodal fusion as described in any one of claims 1-6, characterized in that, include: The first module is used to acquire teaching multimodal data during the bending process of sheet metal parts under manual instruction. The teaching multimodal data includes teaching visual data, teaching force data, and teaching tactile data. The second module is used to train a pre-set learning network model using multiple transition tuples in the teaching multimodal data, and to calculate the minimum loss function value of each transition tuple in the pre-set learning network model. The third module is used to update the parameters of the pre-set learning network model using the model parameters corresponding to each of the minimum loss function values, so as to obtain a sheet metal bending model that has been trained and corresponds to the pre-set learning network model. The fourth module is used to acquire the first multimodal data of the bending robot at the initial moment during the bending process of the sheet metal part, and input the first multimodal data into the sheet metal part bending model for processing to obtain the bending action at the initial moment; The fifth module is used to control the bending robot to perform the bending action, acquire the second multimodal data after the bending action is performed, and input the second multimodal data into the sheet metal bending model for processing until the sheet metal part completes the bending process.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of any one of claims 1-6.

Images