Flexible multifunctional touch sensing electronic skin and application

Abstract

The invention relates to the technical field of sensors, in particular to a flexible multifunctional touch sensing electronic skin and application, and the flexible multifunctional touch sensing electronic skin comprises a deformation module, an imaging module and a deep learning algorithm processing module; the deformation module comprises a mark layer provided with a mark pattern; the deformation module is used for capturing deformation information of the marking layer caused by external stimulation in real time; the imaging module comprises an embedded camera; the imaging module is used for acquiring a real-time image of the marking layer of the deformation module and reflecting the characteristics of tactile stimulation through image data; and the deep learning algorithm processing module is used for processing the image data captured by the imaging module in real time through a deep learning algorithm, extracting feature information in the tactile signal and further analyzing the property and the distribution of the external stimulation. According to the method, the deep learning algorithm and the visual tactile perception are combined, so that the tactile perception precision is improved, breakthrough is realized in the aspects of three-dimensional force decoupling and real-time recognition of complex actions, and the method has higher application potential and wide prospects.

Description

Technical Field

[0001] This application relates to the technical field of sensors, and particularly relates to a flexible multi-functional tactile sensing electronic skin and its applications. Background Art

[0002] In the field of human-computer interaction, tactile sensing technology is gradually becoming a research hotspot. Traditional human-computer interaction methods mainly rely on visual and auditory signals, and the lack of tactile sensing technology greatly restricts the naturalness and richness of the interaction experience. As one of the important ways for humans to perceive the world, touch can enhance the immersion and emotional feedback in the interaction, making human-computer interaction more intuitive and natural. To make up for this gap, it is crucial to develop electronic skin technology that can simulate human tactile perception. This technology can not only significantly improve the human-computer interaction experience, but also enable robots to exhibit more anthropomorphic behaviors in emotional interaction, thus promoting the development of the human-computer interaction field.

[0003] With the progress of flexible electronic skin technology, significant achievements have been made in tactile sensing in recent years. Existing research captures the real-time deformation information of the surface marking layer under external stimuli through an embedded camera, and combines deep learning algorithms to accurately analyze this information. This method can not only identify conventional pressures, but also effectively detect complex actions with a tendency of slippage, such as stroking and pinching. Experimental results show that this technology is much superior to traditional resistive or capacitive thin film sensors in terms of response speed, recognition accuracy, and three-dimensional force resolution.

[0004] However, the existing tactile sensors have the following technical problems: (1) Unable to effectively detect complex tactile actions: Existing tactile sensors, especially resistive or capacitive sensors, usually can only sense simple pressures or contact forces, and it is difficult to identify complex tactile actions such as stroking, pinching, or sliding, which are very common in actual emotional interaction or fine operations. This limits the application of traditional sensors in high-precision tactile perception. (2) Lack of three-dimensional force perception ability: Many existing electronic skins and tactile sensing technologies can only provide two-dimensional force data, and cannot accurately measure and decompose complex three-dimensional force fields. The limitation of this two-dimensional force perception cannot meet the requirements in applications that require high-precision force decoupling. (3) Insufficient response speed and accuracy: Traditional tactile sensing systems often face bottlenecks in response speed and perception accuracy, and it is difficult to provide fast and high-precision real-time feedback, which is not ideal for application scenarios that require high-speed interaction (such as robot emotional interaction and real-time human-computer interaction). (4) Insufficient flexibility and stability: Although flexible electronic skins have a certain degree of adaptability, the balance between flexibility and stability is often not the best. Traditional systems are prone to problems such as external interference or insufficient system stability in dynamic interaction. Summary of the Invention

[0005] The embodiments of the present application provide a flexible multi-functional tactile sensing electronic skin and its application, aiming to solve problems such as complex tactile motion recognition, three-dimensional force decoupling, response speed, and recognition accuracy in existing tactile sensing technologies.

[0006] To solve the above technical problems, in a first aspect, the embodiments of the present application provide a flexible multi-functional tactile sensing electronic skin, including: a deformation module, an imaging module, and a deep learning algorithm processing module; the deformation module includes a marking layer with a marking pattern on its surface; the deformation module is used to capture in real time the deformation information of the marking layer caused by external stimuli; the imaging module includes an embedded camera; the imaging module is used to obtain a real-time image of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through image data; the deep learning algorithm processing module: performs real-time processing on the image data captured by the imaging module through a deep learning algorithm, extracts the characteristic information in the tactile signal, and further analyzes the nature and distribution of external stimuli.

[0007] In some exemplary embodiments, when an external tactile stimulus acts on the electronic skin, the marking pattern of the marking layer deforms, the endoscopic camera captures the change of the marking pattern, and extracts the pattern features; then the image features are transmitted to the deep learning algorithm processing module for analysis to identify the type and spatial distribution of external stimuli.

[0008] In some exemplary embodiments, the marking layer is used to enhance the visualization of the deformation caused by tactile stimuli, so that the imaging module can clearly capture the changes; The marking layer includes a coating film layer and an acrylic plate layer, and an elastomeric material is filled between the coating film layer and the acrylic plate layer; the coating film layer includes multiple layers of silicone coatings, and a spiral pattern is provided on the top silicone coating.

[0009] In some exemplary embodiments, the marking layer includes three layers of silicone coatings, the material of the top silicone coating is white silicone, and a spiral pattern is created on the surface of the white silicone by a UV marking machine; the outer diameter of the spiral pattern is 10 mm, the line width is 20 μm, and the distance between adjacent lines is 200 μm; when the deformation module is subjected to external stimuli, the change of the spiral pattern reflects the surface deformation caused by tactile stimuli.

[0010] In some exemplary embodiments, the silicone coating is uniformly coated on the acrylic plate layer by a spin coating method to form a hierarchical structure of a marking layer, a light shielding layer, and an analog skin appearance; the light shielding layer is the middle layer of the three layers of silicone coatings.

[0011] In some exemplary embodiments, the acrylic plate layer is of type I structure and is used to support the deformation module.

[0012] In some exemplary embodiments, the elastomeric material is made by mixing silicone rubber and silicone oil in proportion, and the elastomeric material is fixed between the coating film layer and the acrylic plate layer through a casting process.

[0013] In some exemplary embodiments, the imaging module further includes an LED strip for providing an illumination source; the LED strip is a self-adhesive COB strip, powered by 12V, and adopts a flicker-free design.

[0014] In some exemplary embodiments, the deep learning algorithm processing module includes a deep learning model; the deep learning model includes an action interaction recognition model and a three-dimensional force perception model; the action interaction recognition model adopts a ResNeXt model and an R(2+1)D convolutional network to classify and recognize image data; image features are extracted through deep learning to identify the type of tactile interaction; the three-dimensional force perception model uses a regression model to predict the distribution of three-dimensional forces, adopts an Adam optimizer and mean squared error as the loss function, and optimizes the model through training data to finally achieve accurate three-dimensional force perception.

[0015] In a second aspect, the embodiments of the present application also provide an application of the flexible multi-functional tactile perception electronic skin as described in the above embodiments in the fields of robot emotional interaction, intelligent wearable devices, intelligent medical devices, and virtual reality / augmented reality.

[0016] The technical solutions provided by the embodiments of the present application have at least the following advantages: The embodiments of the present application provide a flexible multi-functional tactile perception electronic skin, including: a deformation module, an imaging module, and a deep learning algorithm processing module; the deformation module includes a marking layer with a marking pattern on the surface; the deformation module is used to capture the deformation information of the marking layer caused by external stimuli in real time; the imaging module includes an embedded camera; the imaging module is used to obtain a real-time image of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through image data; the deep learning algorithm processing module: performs real-time processing on the image data captured by the imaging module through a deep learning algorithm, extracts the characteristic information in the tactile signal, and then analyzes the nature and distribution of external stimuli.

[0017] The flexible multi-functional tactile perception electronic skin provided by the embodiments of the present application can perform tactile perception in a more efficient and accurate manner by combining deep learning algorithms and visual tactile perception. Especially when dealing with complex tactile interactions (such as slipping, pinching, etc.), it shows great advantages. Compared with traditional flexible electronic skins, the present application not only improves the accuracy of tactile perception, but also achieves breakthroughs in three-dimensional force decoupling and real-time recognition of complex actions, and has stronger application potential and broad prospects. Description of the Drawings

[0018] One or more embodiments are illustrated by way of example in the corresponding accompanying drawings. These illustrative descriptions do not constitute a limitation on the embodiments. Unless otherwise stated, the figures in the accompanying drawings do not constitute a scale limitation.

[0019] Figure 1 This is a schematic diagram of the implementation process of a flexible multi-functional tactile sensing e-skin provided by an embodiment of the present application.

[0020] Figure 2 This is a schematic diagram of the image data enhancement processing flow chart and the tactile action data set constructed by the robotic arm provided by an embodiment of the present application. Detailed implementation manners

[0021] As can be seen from the background art, existing tactile sensors have technical problems such as being unable to effectively detect complex tactile actions, lacking three-dimensional force perception ability, insufficient response speed and accuracy, and deficiencies in flexibility and stability.

[0022] Existing research captures the real-time deformation information of the surface marking layer under external stimuli through an embedded camera, and precisely analyzes this information in combination with deep learning algorithms. In addition, to further verify the performance, existing research integrates the e-skin into a toy dog for interactive motion testing. The results show that the system can identify complex interactive actions in real time and accurately decouple the distribution of three-dimensional forces. A comparative experiment with a traditional rigid six-axis force sensor further proves the excellent performance of this e-skin in real-time three-dimensional force decoupling.

[0023] This application aims to address the limitations existing in traditional sensing technologies, especially in the field of human-computer interaction. Existing piezoelectric films (such as resistive or capacitive film sensors) are difficult to accurately detect complex movements and slip tendencies, such as touch and pinch interactions. Existing technologies usually cannot achieve real-time, large-scale force sensing and accurate motion interaction recognition, and lack the ability to efficiently decouple multiple senses. To solve the above technical problems, the embodiments of this application provide a flexible multifunctional tactile sensing e-skin and its application, including: a deformation module, an imaging module, and a deep learning algorithm processing module; the deformation module includes a marking layer with a marking pattern on its surface; the deformation module is used to capture in real time the deformation information of the marking layer caused by external stimuli; the imaging module includes an embedded camera; the imaging module is used to obtain the real-time image of the marking layer of the deformation module and reflect the characteristics of the tactile stimulus through the image data; the deep learning algorithm processing module: performs real-time processing on the image data captured by the imaging module through the deep learning algorithm, extracts the characteristic information in the tactile signal, and then analyzes the nature and distribution of the external stimulus. This application provides a flexible multifunctional tactile sensing e-skin, which, combined with the deep learning algorithm, can capture and analyze the real-time deformation information of external stimuli, accurately identify complex decision-making interactions, and provide three-dimensional force decoupling ability, thus providing an innovation in robot emotional interaction and human-computer interaction.

[0024] Aiming at the defects of the existing technology, the flexible multifunctional tactile sensing e-skin provided by this application aims to: (1) Achieve accurate perception of complex tactile actions. This application uses visual-tactile sensing technology, combines the embedded camera to capture the deformation information of the marking layer of the deformation module, and combines the deep learning algorithm to analyze the data, and can accurately identify complex tactile interaction actions such as stroking, pinching, and sliding. Compared with traditional sensors, this application significantly improves the detection ability of complex tactile actions and meets the more diverse emotional interaction needs.

[0025] (2) Provide accurate three-dimensional force decoupling ability: This application realizes the accurate perception and decoupling of the three-dimensional force field through visual perception and deep learning analysis technology, making up for the deficiency that the existing technology can only provide two-dimensional perception. Through multi-dimensional analysis of force, the system can more accurately capture the spatial distribution of tactile stimuli and improve the response accuracy of the system to complex actions. (3) Improve the response speed and recognition accuracy of the system: By using a high-speed camera and a deep learning model, this application can quickly respond in real-time interaction and provide high-precision tactile perception. This advantage makes it have obvious technical leadership in robot emotional interaction and complex human-computer interaction scenarios.

[0026] (4) Enhance the integration and stability of the system: By simplifying the hardware design and integrating the visual perception module, this application improves the integration of the electronic skin system and reduces the dependence on complex sensor arrays, thereby enhancing the stability and reliability of the system during long-term use.

[0027] The following will elaborate on each embodiment of this application in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that in each embodiment of this application, many technical details are presented to help readers better understand this application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can still be implemented.

[0028] An embodiment of this application provides a flexible multi-functional tactile perception electronic skin, including: a deformation module, an imaging module, and a deep learning algorithm processing module; the deformation module includes a marking layer with a marking pattern on the surface; the deformation module is used to capture in real time the deformation information of the marking layer caused by external stimuli; the imaging module includes an embedded camera; the imaging module is used to obtain a real-time image of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through image data; the deep learning algorithm processing module: processes the image data captured by the imaging module in real time through a deep learning algorithm, extracts the characteristic information in the tactile signal, and then analyzes the nature and distribution of external stimuli.

[0029] This application captures the deformation of the surface marking layer through an embedded camera, combines a deep learning algorithm to accurately identify tactile stimuli, and decouples the three-dimensional force distribution through visual information. This vision-based tactile perception scheme is an innovative technology in the field of flexible electronic skin. It obtains tactile information through visual perception rather than traditional resistive or capacitive sensors, achieving higher sensitivity and the recognition of complex tactile actions. Moreover, this electronic skin adopts a design that combines a deformation module and an imaging module to form an integrated system with tactile perception and three-dimensional force decoupling capabilities. Through the combination of specific coatings, acrylic plate layers, and elastomer materials, the mechanical response performance of the electronic skin and the pattern recognition effect of the surface marking layer are optimized.

[0030] The flexible multifunctional tactile sensing e-skin based on visual tactile perception and deep learning algorithms provided by this application aims to solve problems such as complex tactile motion recognition, three-dimensional force decoupling, response speed, and recognition accuracy in existing tactile sensing technologies. This e-skin can achieve high-precision tactile motion recognition and three-dimensional force perception in complex human-machine interactions. Compared with existing technologies, by combining deep learning algorithms and visual tactile perception, this application can perform tactile perception in a more efficient and accurate manner, especially showing great advantages when dealing with complex tactile interactions (such as slipping, pinching, etc.). Compared with traditional flexible e-skins, this application not only improves the accuracy of tactile perception but also achieves breakthroughs in three-dimensional force decoupling and real-time recognition of complex actions, with stronger application potential and broad prospects.

[0031] The system architecture of the e-skin in this application consists of three key components: a deformation module (marking layer), an imaging module (embedded camera), and a deep learning algorithm processing module. Each component plays an important role in tactile perception and three-dimensional force decoupling. Among them, the marking layer of the deformation module is used to enhance the visualization of the deformation caused by tactile stimuli, enabling the camera to clearly capture the changes. The marking layer reflects the tactile force through color or texture changes. The endoscope camera of the imaging module integrates a small camera for real-time capturing of the deformation information of the marking layer on the flexible skin surface. This camera can acquire image data at a high frequency and record the deformation of the skin surface caused by external tactile stimuli (such as pressure, friction, sliding, etc.). The deep learning algorithm processing module then processes the image data captured by the camera in real time through deep learning algorithms, extracts the feature information in the tactile signals, and further analyzes the nature and distribution of the external stimuli. These information include the type of stimuli (such as pressure, sliding, stroking, etc.) and the force distribution in three-dimensional space. In some embodiments, when an external tactile stimulus acts on the e-skin, the marking pattern of the marking layer is deformed, the endoscope camera captures the change of the marking pattern, and extracts the pattern features; then the image features are transmitted to the deep learning algorithm processing module for analysis to identify the type and spatial distribution of the external stimuli.

[0032] The working principle of the electronic skin based on visual tactile perception of this application can be divided into four steps: tactile stimulation, image data processing, deep learning analysis, and force decoupling and feedback. In the tactile stimulation step, when the surface of the electronic skin is subjected to external tactile stimulation, the surface marking layer deforms, and the camera captures this deformation information in real time. Then, the image data captured by the camera is processed to extract the features of the deformation. Through the changes in the marking layer, the image data can reflect the action and distribution of external forces. Next, after the image data is processed, the deep learning algorithm further analyzes the deformation information, identifies the type of tactile stimulation (such as pressure, sliding, pinching, etc.), and decouples the acting force distribution of the three-dimensional force based on the deformation of the image. Finally, force decoupling and feedback: Through the optimization of the deep learning algorithm, this application can achieve three-dimensional force decoupling, that is, separating the force signals in multiple directions for processing to obtain more accurate force perception data. In some embodiments, the marking layer is used to enhance the visualization of the deformation caused by tactile stimulation, enabling the imaging module to clearly capture the changes. The marking layer includes a coating film layer and an acrylic plate layer, and an elastomer material is filled between the coating film layer and the acrylic plate layer. The coating film layer includes multiple layers of silicone coatings, and a spiral pattern is provided on the top silicone coating.

[0033] The core of the deformation module is to capture the surface deformation caused by external stimuli through a specially designed marking layer. Specifically, it includes a coating film layer (Coating Layer), an acrylic plate layer (Acrylic Plate Layer), and an elastomer material (Elastomer) filled between the two. Among them, the coating film layer is made by spin-coating three layers of silicone (SORTA-Clear 12), and the thickness of each layer is about 1 mm. The acrylic plate layer is designed in an I-shaped structure to support the deformation module through the acrylic plate layer and prevent the elastomer from falling off. Among different acrylic plate layers, the internal cavity size gradually decreases to ensure stable data acquisition.

[0034] In some embodiments, the marking layer includes three layers of silicone coatings. The material of the top silicone coating is white silicone, and a spiral pattern is created on the surface of the white silicone through a UV marking machine to realize the visualization of the response to external stimuli. The outer diameter of the spiral pattern is 10 mm, the line width is 20 μm, and the spacing between adjacent lines is 200 μm. When the deformation module is subjected to external stimuli, the changes in the spiral pattern reflect the surface deformation caused by tactile stimulation.

[0035] In some embodiments, the silicone coating is evenly covered on the acrylic plate layer by spin-coating to form a hierarchical structure of a marking layer, a light shielding layer, and an analog skin appearance. The light shielding layer is the middle layer of the three layers of silicone coatings.

[0036] This application uses a white silicone marking layer and fabricates patterns using ultraviolet lithography technology. Alternative solutions can employ different types of marking materials (such as transparent films, polymers, or metal films) and different pattern fabrication techniques (such as laser etching, hot embossing, etc.). These variations can accommodate different application requirements, such as use in extreme environments or more miniaturized devices.

[0037] In some embodiments, the elastomeric material is made by mixing silicone and silicone oil in proportion, and the elastomeric material is fixed between the coating film layer and the acrylic plate layer through a casting process. This elastomeric material has high flexibility and can accurately reflect surface deformations caused by tactile stimuli. During the casting process of the elastomeric material, a defoaming machine is used to remove air bubbles to ensure the stability of the structure.

[0038] Figure 1 The implementation process of the flexible multifunctional tactile perception electronic skin provided by this application is shown. Among them, Figure 1 Figure (a) in shows the schematic structure of the coating film layer of the deformation module. Figure 1 Figure (b) in shows the schematic I-type design of the acrylic plate layer. Figure 1 Figure (c) in shows the layered structure of each layer of the deformation module. Figure 1 Figure (d) in shows the installation schematic of the LED strip and the camera of the imaging module. Figure 1 Figure (e) in shows the schematic of the complete electronic skin prototype. As Figure 1 shown, the flexible electronic skin of this application is based on the principle of visual touch. When the outside world stimulates the contact surface, it will cause the image of the marking layer of the deformation module to deform. The embedded camera at the bottom of the substrate of the imaging module captures the deformed image and inputs it into the deep learning algorithm processing module (deep learning model), thereby analyzing the external stimuli, such as three-dimensional forces and corresponding interaction actions.

[0039] The main function of the imaging module is to obtain real-time images of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through image data. In some embodiments, the imaging module further includes an LED strip (LED Strip) for providing an illumination source; the LED strip is a self-adhesive COB strip, installed inside the substrate, providing stable illumination. The power supply is 12V and adopts a flicker-free design. The LED strip serves as an illumination source, providing uniform light for the camera, avoiding interference from external light, and ensuring clear images. The camera of the imaging module uses an endoscope camera, which has a resolution of 1280×720 and a viewing angle of 80°, and can obtain clear images within a focal length range of 30 to 80 mm. The camera is installed in a dedicated slot opened on the substrate to ensure the stability of the camera and obtain clear image data. The camera is connected via USB to ensure stable data acquisition.

[0040] In some embodiments, the deep learning algorithm processing module includes a deep learning model; the deep learning model includes an action interaction recognition model and a three-dimensional force perception model; the action interaction recognition model uses a ResNeXt model and an R(2+1)D convolutional network to classify and recognize image data; image features are extracted through deep learning to recognize the type of tactile interaction; the three-dimensional force perception model uses a regression model to predict the distribution of three-dimensional forces, adopts an Adam optimizer and mean squared error as the loss function, optimizes the model through training data, and finally realizes accurate three-dimensional force perception.

[0041] The core working principle of this application is based on visual-tactile perception. Image data is processed in real time through an embedded camera and a deep learning algorithm to accurately identify the type of tactile stimulus and decouple the distribution of three-dimensional forces.

[0042] It should be noted that an embedded camera is used in this application to capture the deformation data of the surface marking layer. As an alternative, other types of optical sensors (such as CMOS sensors or micro imaging modules) can be used to achieve the same image capture function. These sensors can be replaced in terms of different sizes, performances, and costs without affecting the overall function and accuracy of the system.

[0043] For the capture and processing of tactile stimuli, when an external tactile stimulus (such as pressing, sliding, etc.) acts on the flexible skin, the marking pattern of the coating film layer deforms, the camera captures the change of the pattern, and corresponding features are extracted through image processing techniques. Subsequently, these image data are transmitted to the deep learning model for analysis to identify the nature and spatial distribution of the stimulus.

[0044] For three-dimensional force perception and decoupling, by combining sensor and visual information, this application can decouple complex three-dimensional forces and provide accurate force perception data. This process is achieved through a multi-dimensional force data acquisition system and a deep learning algorithm; this application designs a set of high-precision force sensing systems, and uses a rigid six-axis force sensor to obtain force data in different directions and intensities. At the same time, this application uses a deep learning regression model, based on image data and force sensor data, to predict the specific values of three-dimensional forces, thereby realizing the precise decoupling of forces.

[0045] In terms of data acquisition and processing, for tactile action recognition, this application constructs a data set containing four types of tactile actions (quiet, pressing, touching, pinching), and enhances each type of data to ensure data diversity. Moreover, in order to improve the lighting robustness, a dynamic brightness adjustment technology is adopted to optimize the recognition performance under different lighting conditions.

[0046] Figure 2 It is a flowchart of image data enhancement processing and a schematic diagram of a tactile action data set constructed for a robotic arm provided in the embodiments of this application. Among them, Figure 2Among them, (a) and (b) are the image data enhancement processing flowcharts; Figure 2 Among them, (c) is a schematic diagram of the tactile action dataset constructed by the robotic arm. Figure 2 In (a), in order to make the feature information of the image more obvious, the image is binarized, Figure 2 In (b), in order to increase the number of the dataset, rotation and brightness processing are performed on it; Figure 2 In (c), the actual situation of the robotic arm collecting data is shown. When the indenter of the robotic arm touches the contact surface, a deformed marker layer pattern and corresponding three-dimensional force data will be obtained. These data will form a force-pattern data pair, which will be input into the deep learning model for the next step of training.

[0047] Regarding the training and optimization of the deep learning model, the present application is implemented by using an action interaction recognition model and a three-dimensional force perception model. Among them, the action interaction recognition model uses the ResNeXt model and the R(2+1)D convolutional network to classify image data and recognize actions. Image features are extracted through deep learning to identify the type of tactile interaction. The three-dimensional force perception model uses a regression model to predict the distribution of three-dimensional forces, and adopts the Adam optimizer and the mean square error (MSE) as the loss function. The model is optimized through training data to finally achieve accurate three-dimensional force perception.

[0048] The electronic skin of the present application can obtain the image data of tactile stimuli in real time through the deformation module and the imaging module. The image data is processed by the deep learning model to identify the type of tactile action and decouple the three-dimensional force data. According to the recognition result, real-time feedback is provided, which is applicable to scenarios such as human-computer interaction and robot emotional interaction.

[0049] In addition, the embodiment of the present application also provides an application of the flexible multifunctional tactile perception electronic skin as described in the above embodiment in the fields of robot emotional interaction, intelligent wearable devices, intelligent medical devices, and virtual reality / augmented reality.

[0050] Specifically, in terms of robot emotional interaction, the single skin of the present application provides more natural and flexible tactile perception for the robot, enhancing the emotional experience of human-computer interaction, enabling the robot to make more humanized responses according to different tactile stimuli. In wearable devices such as smart gloves and health monitoring devices, the single skin of the present application can provide real-time tactile feedback and monitoring functions, enhancing the wearing experience. In terms of virtual reality / augmented reality, the single skin of the present application is used for tactile feedback in virtual reality (VR) or augmented reality (AR) systems to enhance the immersive experience, especially in application scenarios that require tactile interaction, such as virtual operations, games, and training. In terms of intelligent medical devices, the single skin of the present application can be used in the medical field, such as devices like prosthetics and rehabilitation robots, to provide high-precision tactile perception and feedback to help patients restore limb perception functions. The flexible multi-functional tactile sensing e-skin provided by this application has the following functional characteristics: (1) Complex tactile motion recognition: This application can effectively recognize complex tactile motions, such as stroking, pinching, sliding, etc. Compared with traditional tactile sensors, it can provide richer and more accurate tactile feedback. (2) Three-dimensional force perception and decoupling: By combining visual perception and deep learning algorithms, this application can accurately perceive and decouple external forces in three-dimensional space, realizing precise perception and analysis of complex three-dimensional force fields. (3) High response speed and high precision: The system performs excellently in terms of response speed and recognition accuracy, can process tactile stimuli in real time and provide quick feedback, meeting the requirements of fast interaction and efficient recognition in complex scenarios. (4) Flexibility and stability: Through the design of flexible e-skin, this application can adapt to dynamically changing tactile stimuli and has high stability, being suitable for long-term and complex interaction scenarios. Compared with the prior art, the advantages of this application are as follows: (1) It has higher tactile perception accuracy and complexity. The prior art usually relies on traditional resistive or capacitive sensors to sense tactile signals. These sensors are insufficient in capturing complex tactile motions (such as pinching, sliding), especially having great limitations in the perception of force distribution and direction. In contrast, this application can accurately capture the deformation of the surface marking layer and analyze the three-dimensional force distribution by combining the principle of visual tactile perception and deep learning algorithms, and can more finely identify complex tactile interactions. For example, it can identify motions such as sliding and pinching. This vision-based tactile sensing technology can greatly improve the recognition accuracy of the system for complex motions.

[0051] (2) Its three-dimensional force decoupling ability is superior to the prior art. Traditional force sensors (such as resistive and piezoelectric sensors) usually cannot effectively decouple forces in different directions and of different types, resulting in only limited tactile perception information being provided, especially being very difficult for the precise measurement of multi-dimensional forces. However, this application can effectively achieve the decoupling and discrimination of three-dimensional forces by combining the image data captured by a camera and deep learning algorithms, greatly improving the accuracy and fineness of force perception. Compared with the prior art, this application can not only recognize simple pressures, but also accurately analyze complex force distributions and interaction methods such as sliding and pinching, solving the limitations of traditional force sensors.

[0052] (3) Higher environmental adaptability and robustness. Existing technologies are often limited by the illumination or noise of the sensors themselves. Especially in environments with large variations in illumination conditions, the recognition accuracy may be greatly reduced. In contrast, this application adopts dynamic brightness adjustment and image enhancement technologies, which can effectively resist the interference of environmental illumination changes and improve the stability and robustness of the system in complex environments. This is significantly better than the resistive or capacitive sensors in the existing technologies, which are usually sensitive to illumination changes and cannot be adjusted or optimized by software means.

[0053] (4) Multi-task learning ability and data utilization efficiency. Force perception and tactile recognition in existing technologies often require separately designed independent systems or models. In contrast, this application adopts deep learning architectures (such as ResNeXt and R(2+1)D convolutional networks), which can achieve multi-task learning of tactile action recognition and three-dimensional force perception, effectively improving the data utilization efficiency and the adaptability of the model. Through a unified deep learning framework, this application can complete multiple tasks simultaneously in a single system, avoiding the waste of computing resources and performance bottlenecks caused by task separation in traditional technologies.

[0054] (5) Improvement in cost-effectiveness. Traditional flexible electronic skin sensors require multiple separate sensor components (such as pressure sensors, temperature sensors, etc.). These components not only increase the complexity of the system but also bring higher costs. In contrast, this application combines visual perception and deep learning, concentrating the tasks that originally required multiple sensors in an integrated system. While reducing the hardware cost, it simplifies the system design and improves the integration and reliability of the system.

[0055] (6) Better user interaction experience. Although existing electronic skin technologies can provide certain tactile perception capabilities, they often cannot achieve rich emotional interactions or high-quality human-machine interactions. In contrast, this application, by realizing fine tactile recognition and three-dimensional force decoupling, can not only recognize simple tactile actions but also distinguish different emotional interactions (such as caressing, pinching, etc.), making the emotional expressions of robots or interactive devices more natural and flexible, greatly enhancing the user interaction experience.

[0056] The flexible multi-functional tactile perception electronic skin provided by this application has significant advantages in terms of high efficiency and accuracy, flexibility and comfort, low cost and high integration. Compared with traditional tactile sensors, the electronic skin of this application can more accurately recognize complex tactile actions and provide precise decoupling of three-dimensional forces. Moreover, the flexible electronic skin of this application can adapt to objects and environments of different shapes, ensuring high-efficiency performance in various interaction scenarios. In addition, this application simplifies the hardware design and adopts visual perception technology, reducing the system cost and improving the integration, making it suitable for large-scale production and application.

[0057] This application has been experimentally verified and proven to be feasible. The experimental results show that this technical solution has high accuracy and operability. The following are the specific situations and results of the experiments and simulation verification: Experimental verification: The electronic skin of this application has demonstrated its excellent performance in multiple experiments, especially in tactile motion recognition and three-dimensional force perception. The experimental results show that the electronic skin can accurately identify different tactile motions, such as touch, pressure, pinch, etc., and it performs particularly well in detecting complex motions such as sliding force (e.g., stroking). By integrating this electronic skin into a toy dog for an emotional interaction experiment, its effectiveness and potential in various tactile interaction scenarios have been verified.

[0058] Three-dimensional force perception verification: In the three-dimensional force perception experiment, by comparing this electronic skin with a rigid six-axis force sensor, the results show that this electronic skin is superior to traditional resistive and capacitive sensors in three-dimensional force decoupling, can effectively identify and process complex motions containing slip trends (such as pinching and stroking), and can also accurately identify conventional pressure stimuli.

[0059] Data acquisition and model training: To verify the accuracy of three-dimensional force recognition, we used the cooperation of a robotic arm and a rigid three-dimensional force sensor to collect 10,586 pairs of image-force data pairs. On this basis, through training with a deep learning model, the predicted values were compared with the actual force measurement values. The experimental results show that this electronic skin has reasonable accuracy in the resolution of three-axis forces, exhibits good prediction ability, and can especially decouple and identify multi-dimensional forces in real time during dynamic interactions.

[0060] Results and prospects: The experimental results of this electronic skin show that it is superior to traditional thin-film sensors in real-time three-dimensional force decoupling and complex motion recognition, and has strong environmental adaptability. In addition, although there are already good experimental results, the technology of this application still has room for further optimization, especially in aspects such as deep learning model optimization, improvement of skin flexibility and durability, and multi-sensor array configuration.

[0061] With the above technical solution, an embodiment of this application provides a flexible multifunctional tactile perception electronic skin and its application, including: a deformation module, an imaging module, and a deep learning algorithm processing module; the deformation module includes a marking layer with a marking pattern on its surface; the deformation module is used to capture in real time the deformation information of the marking layer caused by external stimuli; the imaging module includes an embedded camera; the imaging module is used to obtain the real-time image of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through the image data; the deep learning algorithm processing module: processes the image data captured by the imaging module in real time through a deep learning algorithm, extracts the characteristic information in the tactile signal, and then analyzes the nature and distribution of external stimuli.

[0062] The embodiment of the present application provides a flexible multi-functional tactile sensing e-skin. By combining deep learning algorithms and visual tactile sensing, it can perform tactile sensing in a more efficient and accurate manner, especially showing great advantages when dealing with complex tactile interactions (such as slipping, pinching, etc.). Compared with traditional flexible e-skins, the present application not only improves the accuracy of tactile sensing but also achieves breakthroughs in three-dimensional force decoupling and real-time recognition of complex actions, having stronger application potential and broad prospects.

[0063] Those of ordinary skill in the art can understand that the above-described embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made to them in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make their respective changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A flexible multi-functional tactile sensing electronic skin, characterized in that, Including: A deformation module, an imaging module, and a deep learning algorithm processing module; The deformation module includes a marking layer with a marking pattern on its surface; The deformation module is used to capture in real time the deformation information of the marking layer caused by external stimuli; The imaging module includes an embedded camera; the imaging module is used to obtain a real-time image of the marking layer of the deformation module and reflect the characteristics of tactile stimuli through image data; The deep learning algorithm processing module: performs real-time processing on the image data captured by the imaging module through deep learning algorithms, extracts the characteristic information in the tactile signals, and then analyzes the nature and distribution of external stimuli.

2. The flexible multi-functional tactile sensing electronic skin according to claim 1, characterized in that, When an external tactile stimulus acts on the e-skin, the marking pattern of the marking layer deforms, the endoscopic camera captures the change of the marking pattern, and extracts the pattern features; then the image features are transmitted to the deep learning algorithm processing module for analysis to identify the type and spatial distribution of external stimuli.

3. The flexible multi-functional tactile sensing electronic skin according to claim 1, characterized in that, The marking layer is used to enhance the visualization of the deformation caused by tactile stimuli, so that the imaging module can clearly capture the changes; The marking layer includes a coating film layer and an acrylic plate layer, and an elastomeric material is filled between the coating film layer and the acrylic plate layer; the coating film layer includes multiple silicone coating layers, and a spiral pattern is provided on the top silicone coating layer.

4. The flexible multi-functional tactile sensing electronic skin according to claim 3, wherein The marking layer includes three silicone coating layers, and the material of the top silicone coating layer is white silicone, and a spiral pattern is created on the surface of the white silicone by a UV marking machine; The outer diameter of the spiral pattern is 10 mm, the line width is 20 μm, and the spacing between adjacent lines is 200 μm; When the deformation module is subjected to external stimuli, the change of the spiral pattern reflects the surface deformation caused by tactile stimuli.

5. The flexible multi-functional tactile sensing electronic skin according to claim 4, characterized in that, The silicone coating layer is uniformly covered on the acrylic plate layer by spin coating to form a hierarchical structure of a marking layer, a light shielding layer, and an analog skin appearance; the light shielding layer is the middle layer of the three silicone coating layers.

6. The flexible multi-functional tactile sensing electronic skin according to claim 3, characterized in that, The acrylic plate layer is of type I structure and is used to support the deformation module.

7. The flexible multifunctional tactile sensing electronic skin according to claim 3, characterized in that The elastomeric material is made by mixing silicone and silicone oil in proportion, and the elastomeric material is fixed between the coating film layer and the acrylic plate layer by a casting process.

8. The flexible multi-functional tactile sensing electronic skin according to claim 1, wherein, The imaging module further includes an LED strip for providing a light source; the LED strip is a self-adhesive COB strip, powered by 12V, and adopts a flicker-free design.

9. The flexible multi-functional tactile sensing electronic skin according to claim 1, characterized in that, The deep learning algorithm processing module includes a deep learning model; The deep learning model includes an action interaction recognition model and a three-dimensional force perception model; The action interaction recognition model uses a ResNeXt model and an R(2+1)D convolutional network to perform classification and action recognition of image data; extracts image features through deep learning to identify the type of tactile interaction; The three-dimensional force perception model uses a regression model to predict the distribution of three-dimensional forces, adopts an Adam optimizer and mean squared error as the loss function, optimizes the model through training data, and finally realizes accurate three-dimensional force perception.

10. An application of the flexible multifunctional tactile sensing electronic skin according to any one of claims 1 to 9, characterized in that, The e-skin is applied to the fields of robot emotional interaction, smart wearable devices, smart medical devices, virtual reality / augmented reality.

Images