Tactile sensor, electronic skin and agent

Abstract

The invention discloses a tactile sensor, an electronic skin and an intelligent agent, and belongs to the technical field of sensors. The tactile sensor comprises a tactile sensing structure, the tactile sensing structure comprises a temperature sensing layer and a pressure sensing layer which are arranged in a stacked mode, the color of the temperature sensing layer changes within a first color range along with the temperature change of the tactile sensing structure, and the color of the pressure sensing layer changes within a second color range along with the pressure change of the tactile sensing structure; the image collector is located on one side of the tactile sensing structure, the image collector is used for collecting a color image of the tactile sensing structure, and image information of the color image in the first color range and the second color range is used for representing temperature information and pressure information of a contact object of the tactile sensor; the touch sensing structure and the image collector are arranged on the support structure, a first containing space is defined by the support structure, and the image collector and the touch sensing structure are located at the two opposite ends of the first containing space.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sensors, and particularly relates to a tactile sensor, an electronic skin and an intelligent body. BACKGROUND

[0002] Tactile sensation is a general term of mechanical stimulation such as contact, sliding and pressure sensation. The tactile sensor can imitate human skin to express temperature, humidity and pressure sensation in a quantitative manner, and can be applied in the fields of robot technology, wearable devices, industrial automation and the like. At present, the tactile sensor can only realize single-mode sensing, for example, can only sense the pressure mode or can only sense the temperature mode, and cannot realize multi-mode sensing, which may limit the application scenarios of the tactile sensor. SUMMARY

[0003] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a tactile sensor, an electronic skin and an intelligent body, which can simultaneously sense the temperature and the applied pressure of a contact object, realize multi-mode sensing, and has a simple sensor structure and a wide application scenario.

[0004] In a first aspect, the present application provides a tactile sensor, comprising:

[0005] a tactile sensing structure, the tactile sensing structure comprising a temperature sensing layer and a pressure sensing layer stacked, the color of the temperature sensing layer changing in a first color range with the temperature change of the tactile sensing structure, and the color of the pressure sensing layer changing in a second color range with the pressure change of the tactile sensing structure;

[0006] an image collector, the image collector being located on one side of the tactile sensing structure, and the image collector being configured to collect a color image of the tactile sensing structure, and the image information of the first color range and the second color range of the color image being configured to represent the temperature information and the pressure information of a contact object of the tactile sensor;

[0007] a support structure, the tactile sensing structure and the image collector being arranged in the support structure, and the support structure defining a first accommodating space, and the image collector and the tactile sensing structure being located at opposite ends of the first accommodating space.

[0008] According to the tactile sensor of the present application, the temperature sensing layer and the pressure sensing layer are stacked, the color of the temperature sensing layer changes according to the temperature change of the contact object, the color of the pressure sensing layer changes according to the pressure applied by the contact object, the image collector collects the color image of the color superposition of the temperature sensing layer and the pressure sensing layer, the temperature information and the pressure information of the contact object can be identified according to the color image, the multi-mode sensing of the tactile sensor is realized, and the sensor structure is simple and has a wide application scenario.

[0009] According to an embodiment of the present application, the first color range is a color range corresponding to changes in hue dimension, saturation dimension and lightness dimension in a hexagonal cone model color space; and the second color range is a color range corresponding to changes in hue dimension and saturation dimension in the hexagonal cone model color space.

[0010] According to an embodiment of the present application, the color of the temperature sensing layer is positively correlated with temperature in the lightness dimension within the first color range, and the color of the pressure sensing layer is positively correlated with pressure in the hue dimension within the second color range.

[0011] According to an embodiment of the present application, the first color range is a color range from white to black, and the second color range is a color range from red to blue.

[0012] According to an embodiment of the present application, the image collector is located on a side corresponding to a side of the pressure sensing layer away from the temperature sensing layer, the pressure sensing layer is light-transmitting, and the temperature sensing layer is light-blocking.

[0013] According to an embodiment of the present application, the material of the temperature sensing layer is an organic reversible temperature-sensitive color-changing material.

[0014] According to an embodiment of the present application, the material of the pressure sensing layer is a structural color elastic photopolymer.

[0015] According to an embodiment of the present application, the tactile sensor further comprises:

[0016] A light source module is arranged on the support structure and located at an end where the image collection device is located, and the light source module is used to illuminate the first containing space to provide light for the image collector to collect images.

[0017] According to an embodiment of the present application, the light source module comprises a point light source and a light diffusion layer, and the point light source is arranged on a side of the light diffusion layer away from the tactile sensing structure.

[0018] According to an embodiment of the present application, the tactile sensor further comprises:

[0019] A transparent layer is arranged on the support structure and located between the image collector and the tactile sensing structure, and the transparent layer deforms with the deformation of the tactile sensing structure.

[0020] According to an embodiment of the present application, the tactile sensor further comprises:

[0021] A wear-resistant layer is arranged on a side of the tactile sensing structure away from the image collector.

[0022] In a second aspect, the present application provides an electronic skin, comprising:

[0023] The tactile sensor according to the first aspect described above;

[0024] A processor connected with the tactile sensor, the processor being configured to determine temperature information and pressure information of the object contacted by the tactile sensor according to image information of the first color range and the second color range in the color image collected by the tactile sensor.

[0025] According to the electronic skin of the present application, the temperature sensing layer and the pressure sensing layer are stacked, the temperature sensing layer changes color according to the temperature of the contacted object, the pressure sensing layer changes color according to the pressure applied by the contacted object, the color image of the color superposition of the temperature sensing layer and the pressure sensing layer is collected by the image collector, the temperature information and the pressure information of the contacted object can be identified according to the color image, the multi-modal perception of the tactile sensor is realized, the sensor structure is simple, and the application scenarios are wide.

[0026] In a third aspect, the present application provides an intelligent agent, comprising:

[0027] The electronic skin according to the second aspect described above.

[0028] According to the intelligent agent of the present application, the temperature sensing layer and the pressure sensing layer are stacked, the temperature sensing layer changes color according to the temperature of the contacted object, the pressure sensing layer changes color according to the pressure applied by the contacted object, the color image of the color superposition of the temperature sensing layer and the pressure sensing layer is collected by the image collector, the temperature information and the pressure information of the contacted object can be identified according to the color image, the multi-modal perception of the tactile sensor is realized, the sensor structure is simple, and the application scenarios are wide.

[0029] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0031] Figure 1 is one of the structure schematic diagrams of the tactile sensor provided by the embodiments of the present application;

[0032] Figure 2 is the second structure schematic diagram of the tactile sensor provided by the embodiments of the present application;

[0033] Figure 3 is the third structure schematic diagram of the tactile sensor provided by the embodiments of the present application;

[0034] Figure 4 Fig. 4 is a structural schematic diagram of a haptic sensor provided by an embodiment of the present application;

[0035] Figure 5 Fig. 1 is a flow schematic diagram of a haptic sensing method provided by an embodiment of the present application;

[0036] Figure 6 Fig. 2 is a flow schematic diagram of a haptic sensing method provided by an embodiment of the present application;

[0037] Figure 7 Fig. 3 is a flow schematic diagram of a haptic sensing method provided by an embodiment of the present application;

[0038] Figure 8 Fig. 5 is a structural schematic diagram of a haptic sensing device provided by an embodiment of the present application;

[0039] Figure 9 Fig. 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0040] Reference signs:

[0041] temperature sensing layer 110, pressure sensing layer 120, image collector 130, support structure 140,

[0042] first accommodating space 150, wear-resistant layer 160, point light source 170, light diffusion layer 180, transparent layer 190. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0044] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0045] The haptic sensor, electronic skin, haptic perception method, haptic perception device, intelligent body, electronic device, and readable storage medium provided by the embodiments of the present application will be described in detail below with reference to specific embodiments and application scenarios in combination with the drawings.

[0046] As shown in Figure 1 The haptic sensor includes a haptic sensing structure, an image collector 130, and a support structure 140.

[0047] The haptic sensing structure includes a temperature sensing layer 110 and a pressure sensing layer 120 arranged in layers.

[0048] The temperature sensing layer 110 is a layer structure that can sense the temperature of a contact object of the haptic sensing structure, and the pressure sensing layer 120 is a layer structure that can sense the pressure applied by the contact object to the haptic sensing structure.

[0049] In this embodiment, the temperature sensing layer 110 and the pressure sensing layer 120 can have the same area, and the temperature sensing layer 110 and the pressure sensing layer 120 can be arranged in close contact.

[0050] In this embodiment, the temperature sensing layer 110 and the pressure sensing layer 120 are both made of an elastic material. When the haptic sensing structure is pressed by a contact object, the temperature sensing layer 110 and the pressure sensing layer 120 can produce corresponding deformation, and when the contact object no longer presses the haptic sensing structure, the temperature sensing layer 110 and the pressure sensing layer 120 can return to the original shape, which can be a flat shape.

[0051] In this embodiment, the color of the temperature sensing layer 110 changes within a first color range with the change of the temperature of the haptic sensing structure, and the color of the pressure sensing layer 120 changes within a second color range with the change of the pressure of the haptic sensing structure.

[0052] It should be noted that the color can include a hue (Hue) dimension, a saturation (Saturation) dimension, and a lightness (Value) dimension.

[0053] The first color range and the second color range can be a color range corresponding to one dimension, two dimensions, or three dimensions of the hue dimension, the saturation dimension, and the lightness dimension.

[0054] In this embodiment, the change dimensions corresponding to the first color range and the second color range can be different, and in addition, the range boundary values of the same change dimension corresponding to the first color range and the second color range can be different.

[0055] In this embodiment, when pressed by a contact object, the temperature sensing layer 110 and the pressure sensing layer 120 respectively produce color changes in different dimensions corresponding to the temperature of the contact object and the pressure applied by the contact object to the haptic sensing structure.

[0056] In this embodiment, the temperature sensing layer 110 has different colors when the temperature of the contact object is different, and the pressure sensing layer 120 has different colors when the pressure applied by the contact object to the tactile sensing structure is different.

[0057] In this embodiment, the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 can be superimposed in the direction intersecting the plane where the tactile sensing structure is located, corresponding to the color of the tactile sensing structure.

[0058] In this embodiment, the image collector 130 is located on one side of the tactile sensing structure, and the image collector 130 is used to collect a color image of the tactile sensing structure.

[0059] The image collector 130 is a device capable of image collection, such as a monocular camera.

[0060] The image collector 130 can be arranged in the direction intersecting the plane where the tactile sensing structure is located, and the color image of the tactile sensing structure is collected.

[0061] The color image is a corresponding image obtained by superimposing the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120, and the image information in the first color range and the second color range of the color image is used to represent the temperature information and the pressure information of the contact object of the tactile sensor.

[0062] The image information in the first color range of the color image is information corresponding to the color of the temperature sensing layer 110, and the image information in the second color range of the color image is information corresponding to the color of the pressure sensing layer 120.

[0063] The temperature information can represent the temperature of the contact object, and the pressure information can represent the pressure applied by the contact object to the tactile sensor.

[0064] In this embodiment, the color of the temperature sensing layer 110 can represent the temperature information of the contact object of the tactile sensor, and the color of the pressure sensing layer 120 can represent the pressure information of the contact object of the tactile sensor.

[0065] The color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 are superimposed on the color image, and the image information of the color image can represent the temperature information and the pressure information of the contact object of the tactile sensor at the same time.

[0066] The image information of the color image can be separated in the first color range and the second color range respectively, so as to separate the temperature information of the contact object of the tactile sensor and the pressure information of the contact object of the tactile sensor.

[0067] In this embodiment, the tactile sensing structure and the image acquisition device 130 are disposed on the support structure 140, the support structure 140 defines a first receiving space 150, and the image acquisition device 130 and the tactile sensing structure are located at opposite ends of the first receiving space 150.

[0068] Among them, the support structure 140 is a component used to support and fix the tactile sensing structure, and can be made of materials such as steel or acrylic.

[0069] In this embodiment, the support structure 140 can support and fix the edge of the tactile sensing structure. Corresponding to the middle part of the tactile sensing structure, the support structure 140 can be hollow, and the hollow part can provide space for deformation of the tactile sensing structure.

[0070] The support structure 140 may also include a flat plate structure. The tactile sensing structure can be set on the flat plate structure of the support structure 140. The flat plate structure is a transparent structure. A deformable layer structure can be set between the tactile sensing structure and the support structure 140 to provide space for the tactile sensing structure to deform.

[0071] In this embodiment, the support structure 140 may include an enclosing sidewall structure, the two ends of which may define a hollow portion corresponding to the enclosing shape, and the first accommodating space 150 may be defined in the following ways.

[0072] Firstly, such as Figure 1 As shown, flat plate structures can be provided at both ends of the surrounding sidewall structure to cover the hollow part, and the surrounding sidewall structure and the flat plate structures at both ends define the first accommodating space 150.

[0073] The tactile sensing structure and the image acquisition unit 130 can be respectively set on the flat plate structures at both ends of the side wall structure. The tactile sensing structure can be stacked on the flat plate structure at one end of the side wall structure, and the image acquisition unit can be embedded in the flat plate structure at the other end of the side wall structure.

[0074] The camera of the image acquisition device 130 faces the first accommodating space 150 and is not obstructed by the corresponding flat structure. The flat structure corresponding to the tactile sensing structure can be a transparent layer 190, and the color of the tactile sensing structure can be acquired by the image acquisition device 130.

[0075] Secondly, such as Figure 2 As shown, the hollow portion at one end of the surrounding sidewall structure can be directly covered by the tactile sensing structure, and the hollow portion at the other end can be directly covered by the image acquisition device 130. The surrounding sidewall structure, the tactile sensing structure, and the image acquisition device 130 define the first accommodating space 150.

[0076] The support structure 140 supports the edges of the tactile sensing structure, the edges of the image collector 130 closely fit the surrounding side wall structure of the support structure 140, and the camera of the image collector 130 faces the first containing space 150.

[0077] Thirdly, as shown in Figure 3 The hollow part at one end of the surrounding side wall structure can be directly covered by the tactile sensing structure, and the other end is provided with a flat plate structure, and the image collector 130 is embedded into the flat plate structure. The surrounding side wall structure, the tactile sensing structure, and the corresponding flat plate structure of the image collector 130 define the first containing space 150.

[0078] The camera of the image collector 130 faces the first containing space 150, and the camera is not blocked by the corresponding flat plate structure.

[0079] Fourthly, as shown in Figure 4 The hollow part at one end of the surrounding side wall structure can be provided with a flat plate structure, and the tactile sensing structure can be stacked on the flat plate structure at one end of the side wall structure. The hollow part at the other end can be directly covered by the image collector 130. The surrounding side wall structure, the corresponding flat plate structure of the tactile sensing structure, and the image collector 130 define the first containing space 150.

[0080] The corresponding flat plate structure of the tactile sensing structure can be a transparent layer 190.

[0081] It should be noted that the structure defining the first containing space 150 is not limited to the above four cases.

[0082] In this embodiment, the tactile sensing structure and the image collector 130 are arranged in the support structure 140, which can make the tactile sensor more integrated.

[0083] The image collector 130 and the tactile sensing structure are located at opposite ends of the first containing space 150, which can ensure that the image collector 130 has sufficient working distance when collecting the color image of the tactile sensing structure.

[0084] In this embodiment, the tactile sensor can further include a housing structure, and the side of the tactile sensing structure away from the image collector 130 is exposed from the housing structure.

[0085] In the related art, the tactile sensor can usually only realize single-mode sensing, for example, only pressure mode or only temperature mode, and cannot realize multi-mode sensing, which may limit the application scenarios of the tactile sensor.

[0086] In the embodiment of the present application, the tactile sensing structure of the tactile sensor includes a temperature sensing layer 110 and a pressure sensing layer 120, the temperature sensing layer 110 and the pressure sensing layer 120 are arranged in layers, and the tactile sensing structure can simultaneously identify the temperature of the contact object and the pressure applied by the contact object to the tactile sensing structure.

[0087] When the contact object contacts the tactile sensor, the temperature sensing layer 110 can sense the temperature of the contact object to change color in the first color range, and the pressure sensing layer 120 can sense the pressure of the contact object to change color in the second color range.

[0088] The color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 can be superimposed in a direction intersecting the plane where the tactile sensing structure is located, and the superimposed color can be collected by the image collector 130 to obtain a color image of the tactile sensor.

[0089] The color image simultaneously includes the temperature information and the pressure information of the contact object, and the color image can be used to separate the image information in the first color range and in the second color range to separate the temperature information and the pressure information, so that the tactile sensor can perceive the temperature of the contact object while perceiving the pressure of the contact object, that is, multi-modal perception is realized.

[0090] The tactile sensor simultaneously perceives the temperature and the pressure of the contact object, which can improve the operation accuracy of the device using the tactile sensor, and the application scenario of the tactile sensor is more extensive.

[0091] According to the tactile sensor provided in the embodiment of the present application, by arranging the temperature sensing layer 110 and the pressure sensing layer 120 in layers, the temperature sensing layer 110 changes color according to the temperature of the contact object, the pressure sensing layer 120 changes color according to the pressure applied by the contact object, the image collector 130 collects the color image of the superimposed temperature sensing layer 110 and pressure sensing layer 120, and the temperature information and the pressure information of the contact object can be identified according to the color image, multi-modal perception of the tactile sensor is realized, the sensor structure is simple, and the application scenario is extensive.

[0092] In some embodiments, the first color range is a color range corresponding to changes in hue dimension, saturation dimension and lightness dimension in a hexagonal cone model color space; and the second color range is a color range corresponding to changes in hue dimension and saturation dimension in the hexagonal cone model color space.

[0093] The hexagonal cone model color space is a color space that can decompose color into hue dimension, saturation dimension and lightness dimension.

[0094] The hue dimension represents the type of color, such as red, yellow, green, etc., and the range of the hue dimension starts from red, passes through orange, yellow, green, blue, indigo, and back to red, forming a closed loop.

[0095] The saturation dimension represents the purity or intensity of the color, i.e., how much gray component is in the color, and the range of the saturation dimension varies from gray to full saturation.

[0096] The lightness dimension represents the light-dark degree of the color, and the range of the lightness dimension varies from black to white.

[0097] It should be noted that the color image of the tactile sensing structure collected by the image collector 130 corresponds to a Red Green Blue (RGB) color space, and the color image can be converted from the RGB color space to the hexagonal pyramid model color space.

[0098] In this embodiment, the first color range includes the range of the hue dimension, the range of the saturation dimension, and the range of the lightness dimension.

[0099] The first color range can correspond to the entire range or a partial range of the hue dimension, can correspond to the entire range or a partial range of the saturation dimension, and can correspond to the entire range or a partial range of the lightness dimension.

[0100] For example, the hue dimension range corresponding to the first color range can be red to blue, the saturation dimension range corresponding to the first color range can be gray to full saturation, and the lightness dimension range corresponding to the first color range can be black to white.

[0101] In this embodiment, the second color range includes the range of the hue dimension and the range of the saturation dimension.

[0102] The second color range can correspond to the entire range or a partial range of the hue dimension and can correspond to the entire range or a partial range of the saturation dimension.

[0103] For example, the hue dimension range corresponding to the second color range can be red to blue, and the saturation dimension range corresponding to the second color range can be gray to full saturation.

[0104] In this embodiment, the first color range and the second color range can overlap in the hue dimension and the saturation dimension, and the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 overlap in the hue dimension and the saturation dimension.

[0105] The image information of the color image in the hue dimension includes the overlap of the information corresponding to the temperature sensing layer 110 and the pressure sensing layer 120, the image information of the color image in the saturation dimension also includes the overlap of the information corresponding to the temperature sensing layer 110 and the pressure sensing layer 120, and the image information of the color image in the lightness dimension includes the information corresponding to the temperature sensing layer 110.

[0106] In actual implementation, the corresponding temperature information and pressure information can be separated according to the image information corresponding to the hue dimension, the saturation dimension and the lightness dimension of the color image, using a trained neural network model or by calculating a corresponding decoupling equation.

[0107] In some embodiments, the color of the temperature sensing layer 110 in the lightness dimension within the first color range is positively correlated with the temperature, and the color of the pressure sensing layer 120 in the hue dimension within the second color range is positively correlated with the pressure.

[0108] In this embodiment, the higher the temperature of the contact object of the tactile sensing structure, the brighter the temperature sensing layer 110, and the lightness dimension changes from no brightness to maximum brightness, corresponding to a value from 1 to 0. The higher the temperature of the contact object of the tactile sensing structure, the greater the value of the color of the temperature sensing layer 110 in the lightness dimension.

[0109] For example, when the temperature of the contact object of the tactile sensing structure is 25°C, the value of the color of the temperature sensing layer 110 in the lightness dimension is 0.5, and when the temperature of the contact object of the tactile sensing structure is 30°C, the value of the color of the temperature sensing layer 110 in the lightness dimension is 0.8.

[0110] In this embodiment, the angle 0° or 360° of the hue dimension can represent red, 120° represents green, and 240° represents blue. The greater the pressure applied by the contact object of the tactile sensing structure, the greater the angle value of the hue dimension of the color of the pressure sensing layer 120.

[0111] For example, when the contact object first presses the tactile sensing structure, the color of the pressure sensing layer 120 in the hue dimension can be red, and as the pressure applied by the contact object to the tactile sensing structure increases, the color of the pressure sensing layer 120 in the hue dimension can gradually approach green.

[0112] In some embodiments, the first color range is a color range from white to black, and the second color range is a color range from red to blue.

[0113] In this embodiment, the first color range corresponds to a temperature range.

[0114] For example, the first color range corresponds to a temperature range of 18-45℃, when the temperature is greater than 45℃, the white boundary value of the first color range is displayed, when the temperature is less than 18℃, the black boundary value of the first color range is displayed, and when the temperature is between 18℃ and 45℃, the color between black and white of the first color range is displayed.

[0115] In this embodiment, the second color range corresponds to a pressure range.

[0116] For example, the pressure range can be a range between 0 and a pressure corresponding to the maximum pressing degree of the tactile sensing structure.

[0117] When the tactile sensing structure is not pressed, the red boundary value of the second color range is displayed, when the tactile sensing structure is pressed to the maximum degree, the blue boundary value of the second color range is displayed, and when the pressure is between 0 and the pressure corresponding to the maximum pressing degree, the color between red and blue of the second color range is displayed.

[0118] In some embodiments, the image collector 130 is located on the side corresponding to the side of the pressure sensing layer 120 away from the temperature sensing layer 110, the pressure sensing layer 120 is light-transmitting, and the temperature sensing layer 110 is light-blocking.

[0119] In this embodiment, the pressure sensing layer 120 is made of a light-transmitting material, which is a colored semi-transparent film, and the color of the temperature sensing layer 110 can be transmitted through the pressure sensing layer 120.

[0120] The color image collected by the image collector 130 located on the side corresponding to the side of the pressure sensing layer 120 away from the temperature sensing layer 110 can include the superposition of the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120.

[0121] In this embodiment, the temperature sensing layer 110 is arranged on the side of the pressure sensing layer 120 away from the image collector 130, and the temperature sensing layer 110 can be closer to the contact object, so that the temperature sensing is more accurate.

[0122] In some embodiments, the material of the temperature sensing layer 110 is an organic reversible temperature-sensitive color-changing material.

[0123] The organic reversible temperature-sensitive color-changing material is a material that can change its own color according to the change of the contact temperature.

[0124] The organic reversible temperature-sensitive color-changing material can include an electron donor and an electron acceptor, and the electron transfer can occur between the electron donor and the electron acceptor when the temperature changes. During the electron transfer process, the organic reversible temperature-sensitive color-changing material can absorb or radiate light of a certain wavelength, which appears as a change in color.

[0125] The organic reversible thermochromic material can further include a regulator, a sensitizer and other solvents, wherein the regulator can be used to regulate the thermochromic temperature range and the thermochromic speed, and the sensitizer can be used to enhance the thermochromic sensitivity and stability of the organic reversible thermochromic material.

[0126] In some embodiments, the material of the pressure sensing layer 120 is a structural color elastic photopolymer.

[0127] The structural color elastic photopolymer is a high molecular material that can change optical properties through elastic deformation. When the structural color elastic photopolymer deforms, the optical properties such as the refractive index and scattering of light will change, thereby causing the color of the material to change.

[0128] In some embodiments, the tactile sensor further includes a light source module.

[0129] The light source module is arranged on the support structure 140 and located at one end where the image acquisition device is located. The light source module is used to illuminate the first containing space 150 to provide light for the image acquisition device 130 to acquire images.

[0130] In this embodiment, the light source module can include a light source, which can be a point light source 170, a surface light source, etc. The surface light source can uniformly illuminate the first containing space 150, and the light source is a white light source.

[0131] In this embodiment, the light source module can include a plurality of light sources arranged around the image acquisition device 130 to provide light for the image acquisition device 130 at various angles.

[0132] In some embodiments, as shown in Figure 1 the light source module includes a point light source 170 and a light diffusion layer 180, and the point light source 170 is arranged on the side of the light diffusion layer 180 away from the tactile sensing structure.

[0133] The light diffusion layer 180 is used to diffuse the light emitted by the point light source 170 to the first containing space 150 more uniformly, so that the first containing space 150 is uniformly illuminated.

[0134] In some embodiments, as shown in Figure 1 the tactile sensor further includes a transparent layer 190.

[0135] The transparent layer 190 is arranged on the support structure 140 and located between the image acquisition device 130 and the tactile sensing structure. The transparent layer 190 deforms with the deformation of the tactile sensing structure.

[0136] In this embodiment, the support structure 140 can include a flat plate structure, the tactile sensing structure can be arranged on the flat plate structure of the support structure 140, and a transparent layer 190 is arranged between the tactile sensing structure and the support structure 140 to provide a deformation space for the tactile sensing structure.

[0137] In this embodiment, the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 can be transmitted through the transparent layer 190.

[0138] In some embodiments, as shown in FIG. 1, the tactile sensor further includes a wear-resistant layer 160. Figure 1

[0139] In this embodiment, the wear-resistant layer 160 is arranged on the side of the tactile sensing structure away from the image collector 130.

[0140] When the contact object presses the tactile sensing structure, the wear-resistant layer 160 is arranged between the contact object and the tactile sensing structure, which can protect the tactile sensing structure from being worn.

[0141] The embodiments of the present application also provide an electronic skin.

[0142] The electronic skin includes the tactile sensor described above and a processor connected with the tactile sensor.

[0143] The processor is configured to determine the temperature information and the pressure information of the contact object of the tactile sensor according to the image information of the first color range and the second color range of the color image collected by the tactile sensor.

[0144] In this embodiment, the processor can determine the temperature information and the pressure information of the contact object of the tactile sensor according to the image information of the color image through a trained neural network model or a corresponding decoupling equation.

[0145] According to the electronic skin provided by the embodiments of the present application, the temperature sensing layer 110 and the pressure sensing layer 120 are arranged in a stacked manner, the temperature sensing layer 110 changes color according to the temperature of the contact object, the pressure sensing layer 120 changes color according to the pressure applied by the contact object, the image collector 130 collects a color image in which the colors of the temperature sensing layer 110 and the pressure sensing layer 120 are superimposed, and the temperature information and the pressure information of the contact object can be identified according to the color image, so that the tactile sensor realizes multi-modal perception, and the sensor structure is simple and the application scenarios are wide.

[0146] The embodiments of the present application also provide an intelligent agent, which includes the electronic skin described above, and the electronic skin can be arranged on the surface of the intelligent agent.

[0147] ​According to the intelligent agent provided in the embodiments of this application, by setting up a layered temperature sensing layer 110 and a pressure sensing layer 120, the temperature sensing layer 110 changes color according to the temperature of the contact object, and the pressure sensing layer 120 changes color according to the pressure applied to the contact object. The image acquisition unit 130 acquires a color image of the superimposed colors of the temperature sensing layer 110 and the pressure sensing layer 120. The temperature information and pressure information of the contact object can be identified based on the color image, realizing multimodal perception of the tactile sensor. The sensor has a simple structure and a wide range of application scenarios.

[0148] This application also provides a tactile sensing method that can identify the temperature and pressure of a target object in contact with a tactile sensing structure.

[0149] The tactile sensing method of this application embodiment can be applied to the tactile sensing structure of the tactile sensor described above.

[0150] Among them, the tactile sensing method can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0151] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0152] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0153] The tactile sensing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the tactile sensing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The tactile sensing method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0154] like Figure 5 As shown, the tactile sensing method includes steps 510, 520 and 530.

[0155] Step 510: Obtain a color detection image of the tactile sensing structure in contact with the target object.

[0156] The tactile sensing structure includes a temperature sensing layer 110 and a pressure sensing layer 120 stacked together.

[0157] The temperature sensing layer 110 is a layer structure that can sense the temperature of the target object contacting the tactile sensing structure, and the pressure sensing layer 120 is a layer structure that can sense the pressure of the target object on the tactile sensing structure.

[0158] In this embodiment, the temperature sensing layer 110 and the pressure sensing layer 120 can have the same area, and the temperature sensing layer 110 and the pressure sensing layer 120 can be closely attached.

[0159] In this embodiment, the temperature sensing layer 110 and the pressure sensing layer 120 are both made of elastic material, and the temperature sensing layer 110 and the pressure sensing layer 120 can deform correspondingly when the tactile sensing structure is pressed by the target object, and return to the original shape when the target object no longer presses the tactile sensing structure. The original shape can be a flat shape.

[0160] The target object is an object contacting the tactile sensing structure, which can be a person, an article, or any object with a physical structure.

[0161] In this embodiment, the color of the temperature sensing layer 110 changes within a first color range with the change of the temperature of the tactile sensing structure, and the color of the pressure sensing layer 120 changes within a second color range with the change of the pressure of the tactile sensing structure.

[0162] In this embodiment, the first color range is a color range corresponding to the change of the hue dimension, the saturation dimension and the lightness dimension in the hexagonal cone model color space, and the second color range is a color range corresponding to the change of the hue dimension and the saturation dimension in the hexagonal cone model color space.

[0163] The hexagonal cone model color space is a color space that can decompose color into hue dimension, saturation dimension and lightness dimension.

[0164] The hue dimension represents the type of color, such as red, yellow, green, etc. The hue dimension ranges from red, through orange, yellow, green, blue, indigo, to magenta, and back to red, forming a closed loop.

[0165] The saturation dimension represents the purity or intensity of the color, i.e. the amount of gray component in the color. The saturation dimension ranges from gray to full saturation.

[0166] The lightness dimension represents the lightness or darkness of the color. The lightness dimension ranges from black to white.

[0167] In this embodiment, the first color range includes the range of the hue dimension, the range of the saturation dimension and the range of the lightness dimension.

[0168] The first color range can correspond to the entire range or a partial range of the hue dimension, the entire range or a partial range of the saturation dimension, and the entire range or a partial range of the lightness dimension.

[0169] For example, the hue dimension range corresponding to the first color range can be red to blue, the saturation dimension range corresponding to the first color range can be gray to full saturation, and the lightness dimension range corresponding to the first color range can be black to white.

[0170] In this embodiment, the second color range includes a range of the hue dimension and a range of the saturation dimension.

[0171] The second color range can correspond to the entire range or a partial range of the hue dimension and the entire range or a partial range of the saturation dimension.

[0172] For example, the hue dimension range corresponding to the second color range can be red to blue, and the saturation dimension range corresponding to the second color range can be gray to full saturation.

[0173] In this embodiment, when the contact object is pressed, the temperature sensing layer 110 and the pressure sensing layer 120 respectively correspond to the temperature of the contact object and the pressure applied by the contact object to the tactile sensing structure to generate color changes in different dimensions.

[0174] In this embodiment, the temperature of the contact object is different, and the color of the temperature sensing layer 110 is different; the pressure applied by the contact object to the tactile sensing structure is different, and the color of the pressure sensing layer 120 is different.

[0175] In this embodiment, the material of the temperature sensing layer 110 can be an organic reversible temperature-sensitive color-changing material.

[0176] The organic reversible temperature-sensitive color-changing material is a material that can change its own color according to the change of the contact temperature.

[0177] The organic reversible temperature-sensitive color-changing material can include an electron donor and an electron acceptor, and the electron donor and the electron acceptor can generate electron transfer when the temperature changes. During the electron transfer process, the organic reversible temperature-sensitive color-changing material can absorb or radiate light of a certain wavelength, which appears as a change in color.

[0178] The organic reversible temperature-sensitive color-changing material can also include a regulator, a sensitizer and other solvents, wherein the regulator can be used to control the color-changing temperature range and the color-changing speed, and the sensitizer can be used to enhance the color-changing sensitivity and stability of the organic reversible temperature-sensitive color-changing material.

[0179] In this embodiment, the material of the pressure sensing layer 120 is a structural color elastic photopolymer.

[0180] The structural color elastic photopolymer is a polymer material that can change optical properties through elastic deformation. When the structural color elastic photopolymer deforms, the optical properties such as the refractive index and scattering of light change, thereby causing the color of the material to change.

[0181] In this embodiment, when the target object contacts the tactile sensing structure, the temperature sensing layer 110 can change to a corresponding color according to the temperature of the target object, and the pressure sensing layer 120 can change to a corresponding color according to the pressure applied by the target object to the tactile sensing structure.

[0182] The color of the temperature sensing layer 110 and the color of the pressure sensing layer 120 can be superimposed in a direction intersecting the plane where the tactile sensing structure is located, corresponding to the color of the tactile sensing structure when the target object contacts the tactile sensing structure.

[0183] In this embodiment, the color detection image is the image of the tactile sensing structure when the tactile sensing structure contacts the target object.

[0184] The color of the color detection image is the superposition of the color of the temperature sensing layer 110 and the color of the pressure sensing layer 120, and the color detection image can simultaneously represent the temperature information and the pressure information of the tactile sensing structure.

[0185] The image information of the color detection image in the first color range is the information corresponding to the color of the temperature sensing layer 110, and the image information of the color detection image in the second color range is the information corresponding to the color of the pressure sensing layer 120.

[0186] In this embodiment, the color detection image of the tactile sensing structure can be collected by an image collector 130 such as a monocular camera.

[0187] In this embodiment, the pressure sensing layer 120 can be light-transmitting, and the temperature sensing layer 110 can be light-blocking. The color of the temperature sensing layer 110 can be transmitted through the pressure sensing layer 120, and the image collector 130 is located on the side of the pressure sensing layer 120 away from the temperature sensing layer 110.

[0188] In this embodiment, the temperature sensing layer 110 is arranged on the side of the pressure sensing layer 120 away from the image collector 130, and the temperature sensing layer 110 can be closer to the target object, and the sensed temperature is more accurate.

[0189] It should be noted that the color image collected by the image collector 130 corresponds to a red-green-blue (RGB) color space, and the color image can be converted from the RGB color space to a hexagonal pyramid model color space.

[0190] Step 520, difference between the color detection image and the color reference image corresponding to the tactile sensing structure in the hexagonal pyramid model color space to obtain a color difference value image corresponding to the target object.

[0191] The color reference image is an image corresponding to the tactile sensing structure without contacting the target object and at a reference temperature. The reference temperature can be a set temperature, for example, 36°C.

[0192] In this embodiment, the image acquisition device for collecting the color detection image and the color reference image is the same, and the relative position of the image acquisition device and the tactile sensing structure is the same.

[0193] In this embodiment, the color detection image and the color reference image are respectively differentiated in hue dimension, saturation dimension and lightness dimension to obtain corresponding color difference value images.

[0194] The color difference value image can represent the temperature change and pressure change of the tactile sensing structure after the target object contacts the tactile sensing structure.

[0195] Step 530, based on the hue feature, saturation feature and lightness feature of the color difference value image, the temperature information and pressure information of the tactile sensing structure contacting the target object are obtained according to the color tactile mapping relationship.

[0196] The color tactile mapping relationship is a mapping relationship between the hue feature, saturation feature and lightness feature of the image of the tactile sensing structure and the temperature information and pressure information of the contact object corresponding to the image.

[0197] In this embodiment, the temperature information and pressure information can be obtained through a trained model according to the existing color tactile mapping relationship, or the decoupling equation can be established according to the existing color tactile mapping relationship, and the temperature information and pressure information are obtained by solving the decoupling equation.

[0198] In this embodiment, the temperature information can represent the temperature of the target object, and the pressure information can represent the pressure applied by the target object to the tactile sensing structure.

[0199] It should be noted that the pressure and temperature corresponding to each contact position can be different when the target object contacts the tactile sensing structure. The temperature information corresponding to the target object can include the temperature distribution of each contact position, and the pressure information corresponding to the target object can include the pressure distribution of each contact position.

[0200] In this embodiment, after obtaining the hue feature, saturation feature and lightness feature of a certain pixel point of the color difference value image, the temperature information and pressure information of the target object corresponding to the color difference value image can be obtained through the existing color tactile mapping relationship.

[0201] In the related art, the tactile sensor can only perform single-mode tactile sensing, and different tactile sensors are used to separately identify different modes of tactile, for example, one tactile sensor only identifies temperature, and another tactile sensor only identifies pressure. When performing multi-mode identification, multiple different tactile sensors are usually used, and the electrical signals output by the multiple tactile sensors are superimposed to achieve multi-mode identification. However, this method uses more tactile sensors, and the signal processing is complex.

[0202] According to the tactile sensing method provided in the embodiments of the present application, when the target object contacts the tactile sensing structure, the temperature sensing layer 110 changes to the corresponding color according to the temperature of the target object, and the pressure sensing layer 120 changes to the corresponding color according to the pressure applied by the target object. The color changes of the two layers are superimposed in the color detection image of the tactile sensing structure, and the temperature information and the pressure information corresponding to the color detection image are obtained according to the color tactile mapping relationship. The image signal of the color detection image corresponding to the target object is processed to obtain the temperature of the target object and the pressure applied to the tactile sensing structure. One tactile sensor can be used to achieve multi-mode identification, and the identification process is simple without the electrical signals output by the tactile sensing structure.

[0203] In some embodiments, based on the color feature, saturation feature and brightness feature of the color difference image, the temperature information and the pressure information of the target object contacting the tactile sensing structure are obtained according to the color tactile mapping relationship, including:

[0204] As shown in Figure 6 The color feature, saturation feature and brightness feature of the color difference image are input into the tactile sensing model of the tactile sensing structure, and the temperature information and the pressure information of the target object contacting the tactile sensing structure output by the tactile sensing model are obtained.

[0205] The tactile sensing model is trained based on the calibration image sample set with the color tactile mapping relationship.

[0206] In this embodiment, when the target object contacts the tactile sensing structure, the pressure borne by each position of the tactile sensing structure and the temperature received by each position can be different, and correspondingly, the colors of each pixel point in the color difference image can be different.

[0207] In this embodiment, the color feature, saturation feature and brightness feature corresponding to a pixel in the color difference image are input into the tactile sensing model, and the trained tactile sensing model can output the corresponding temperature information and pressure information.

[0208] In this embodiment, the color difference image can be converted into a two-dimensional data structure, where one dimension represents each pixel in the color difference image, and the other dimension includes the two-dimensional coordinates corresponding to each pixel, as well as the corresponding color feature, saturation feature and brightness feature.

[0209] The two-dimensional data structure corresponding to the color difference image is input into the trained tactile sensing model, and the temperature information and pressure information corresponding to each pixel can be output, thereby obtaining the temperature and applied pressure of the corresponding target object.

[0210] In this embodiment, the calibration image sample set can include multiple calibration sample images, and the calibration sample images are images of the tactile sensing structure with temperature labels and pressure labels corresponding to the tactile sensing structure.

[0211] In some embodiments, the calibration image sample set is obtained based on the following steps:

[0212] The calibration object with a sample temperature is controlled to contact the tactile sensing structure with a sample pressure, and a calibration sample image of the tactile sensing structure contacting the calibration object is obtained.

[0213] The multiple calibration sample images of the calibration image sample set are obtained by traversing the sample temperatures in the sample temperature set and the sample pressures in the sample pressure set.

[0214] The calibration object can be a spherical object, a square object, etc., the sample temperature is a set temperature transmitted to the calibration object, and the sample pressure is a set pressure applied to the calibration object.

[0215] In this embodiment, the sample pressure can be represented by the depth of the tactile sensing structure pressed by the calibration object.

[0216] In this embodiment, the calibration object can be placed on a temperature-controlled heating plate at a sample temperature, and then the calibration object is pressed to a depth corresponding to a sample pressure on the tactile sensing structure, and then the tactile sensing structure is photographed to obtain a calibration sample image.

[0217] In this embodiment, the sample temperature set can include multiple sample temperatures, and the sample pressure set can include multiple sample pressures.

[0218] In this embodiment, the calibration object can be pressed to different depths corresponding to different sample pressures while the calibration object is at a certain sample temperature.

[0219] The process of repeatedly traversing the sample pressures in the sample pressure set while the calibration object is at different sample temperatures and pressing the calibration object to depths corresponding to different sample pressures can obtain multiple calibration sample images.

[0220] In this embodiment, the image acquisition device for acquiring the color detection image and the image acquisition device for acquiring the calibration sample image are the same, and the relative positions of the image acquisition device and the tactile sensing structure are the same.

[0221] In some embodiments, the calibration sample image of the tactile sensing structure contacting the calibration object is acquired, including:

[0222] The calibration object with a sample temperature contacts the tactile sensing structure at a sample pressure, and an initial sample image of the tactile sensing structure contacting the calibration object is acquired;

[0223] Based on the size information of the calibration object, the initial sample image and a color reference image corresponding to the tactile sensing structure are subtracted in the hexagonal pyramid model color space to obtain the calibration sample image.

[0224] The initial sample image is an image obtained by the image acquisition device photographing the tactile sensing structure when the calibration object contacts the tactile sensing structure at the sample pressure.

[0225] The color reference image is an image corresponding to the tactile sensing structure without contacting the target object and at a reference temperature, and the reference temperature can be a set temperature, for example, 36℃.

[0226] In this embodiment, the size information of the calibration object can be the area of the maximum cross section of the calibration object, or the length of the longest side in the maximum cross section, etc.

[0227] For example, the calibration object is a sphere, and the size information of the calibration object can be the diameter corresponding to the circular shape of the maximum cross section.

[0228] In this embodiment, the calibration object can press the tactile sensing structure at a fixed position on the surface of the tactile sensing structure each time, and the color change corresponding to the calibration object in each obtained initial sample image is in the same area range.

[0229] According to the size information of the calibration object, an area range can be determined, and the color change corresponding to the calibration object each time is in the area range, and the area range is as small as possible.

[0230] For example, the calibration object is a sphere a, the diameter corresponding to the circular shape of the maximum cross section of the sphere a is 3cm, and the sphere a starts from the center point of the surface of the tactile sensing structure each time, and presses the tactile sensing structure in the direction perpendicular to the plane on which the surface of the tactile sensing structure lies, and a circular area range b with the center point of the surface of the tactile sensing structure as the center and the diameter of 3cm can be determined, and the color change corresponding to the sphere a each time is in the area range b.

[0231] It should be noted that when the ball a presses the tactile sensing structure in a direction that is not perpendicular to the plane on which the tactile sensing structure is located, the determined area range can be greater than the area range b.

[0232] In this embodiment, the color change of the calibration object corresponds to a determined area range each time. The part in the determined area range of the initial sample image can be retained, and the remaining part can be deleted. At the same time, the part in the corresponding area range of the color reference image can also be retained, and the remaining part can be deleted.

[0233] The retained part of the initial sample image and the retained part of the color reference image are respectively subtracted in the hue dimension, the saturation dimension and the brightness dimension to obtain a calibration sample image.

[0234] In some embodiments, the calibration object with a controlled sample temperature contacts the tactile sensing structure with a sample pressure, and a calibration sample image of the tactile sensing structure contacting the calibration object is obtained, comprising:

[0235] Based on the size information of the calibration object, an object model is constructed, and a pressure label corresponding to the sample pressure is established.

[0236] Based on the size information of the calibration object and the thermal imaging image of the calibration object, a temperature label corresponding to the sample temperature is established.

[0237] In this embodiment, when the calibration object is at the sample temperature and the calibration object presses the tactile sensing structure with the sample pressure, the pressure and the temperature sensed by each position point of the tactile sensing structure can be different.

[0238] In this embodiment, according to the size information of the calibration object and the depth of the calibration object pressing the tactile sensing structure, an object model defined by the deformation of the tactile sensing structure generated when the calibration object presses the tactile sensing structure can be formed.

[0239] For example, the calibration object is a ball, the ball presses the tactile sensing structure to the plane corresponding to the deformation of the tactile sensing structure without alignment with the maximum cross section of the ball, and the deformation of the tactile sensing structure can define a hemispherical structure, so that an object model of the hemispherical structure can be obtained.

[0240] It can be understood that the pixels in the object model correspond one-to-one to the pixels in the corresponding calibration sample image.

[0241] In this embodiment, according to the coordinates of a pixel in the object model and the coordinates of the pixel corresponding to the position when the tactile sensing structure does not generate deformation, the pressing depth of the corresponding position of the pixel can be determined, and the pressure of the corresponding position, i.e. a pressure label corresponding to the sample pressure, can be determined.

[0242] In this embodiment, the pressure label is used to mark the pressure of the corresponding pixel in the calibration sample image, and each pressure label corresponding to the sample pressure can mark the pressure of each pixel in the corresponding calibration sample image.

[0243] In this embodiment, the infrared temperature measurement imager can be used to obtain the temperature corresponding to each deformation position of the tactile sensing structure after the calibration object contacts the tactile sensing structure, and a thermal imaging image is obtained.

[0244] It should be noted that the resolution of the thermal imaging image obtained by the infrared temperature measurement imager can be different from the resolution of the calibration sample image. For example, the resolution of the thermal imaging image is 256x192, and the resolution of the calibration sample image is 640x480.

[0245] According to the size information of the calibration object, the part of the thermal imaging image corresponding to the calibration sample image can be intercepted, and an interpolation up-sampling method such as nearest neighbor interpolation or bilinear interpolation can be used to increase the resolution of the thermal imaging image to the resolution of the calibration sample image.

[0246] In this embodiment, the pixels in the thermal imaging image of the calibration object correspond one-to-one to the pixels in the corresponding calibration sample image.

[0247] In this embodiment, the temperature corresponding to a pixel in the resolution-increased thermal imaging image is a temperature label corresponding to the sample temperature.

[0248] In this embodiment, the temperature label is used to mark the temperature of the corresponding pixel in the calibration sample image, and each temperature label corresponding to the sample temperature can mark the temperature of each pixel in the corresponding calibration sample image.

[0249] In some embodiments, the tactile sensing model is trained based on the following steps:

[0250] The plurality of calibration sample images in the calibration image sample set are input to the tactile sensing model to be trained, and based on the hue feature, saturation feature and lightness feature of the calibration sample image, the predicted temperature information and predicted pressure information output by the tactile sensing model are obtained.

[0251] The predicted temperature information is compared with the sample temperature, and the predicted pressure information is compared with the sample pressure, and based on the comparison result, the parameters of the tactile sensing model are updated to obtain the trained tactile sensing model.

[0252] In this embodiment, the hue feature, saturation feature and lightness feature of the calibration sample image are input to the tactile sensing model to obtain the prediction of the corresponding temperature and corresponding pressure by the tactile sensing model, and the predicted temperature information and predicted pressure information are obtained respectively.

[0253] The predicted temperature information output by the tactile sensing model is compared with the sample temperature, i.e., the temperature label corresponding to the calibration sample image, and the predicted pressure information output by the tactile sensing model is compared with the sample pressure, i.e., the pressure label corresponding to the calibration sample image.

[0254] The parameters of the tactile sensing model can be updated with the difference between the predicted temperature information and the sample temperature being minimized and the difference between the predicted pressure information and the sample pressure being minimized as the target.

[0255] In this embodiment, the plurality of calibration sample images in the calibration image sample set are sequentially used to train the tactile sensing model to obtain the trained tactile sensing model.

[0256] In some embodiments, the tactile sensing model includes a nonlinear transformation layer, a random deactivation layer, a fully connected layer, and a regression layer.

[0257] In this embodiment, the tactile sensing model can be a neural network structure, such as a multilayer perceptron feedforward artificial neural network structure.

[0258] In this embodiment, the nonlinear transformation layer, the random deactivation layer, and the fully connected layer are used to obtain the temperature information and the pressure information of the target object, and the regression layer is used to obtain the regression labels corresponding to the temperature information and the pressure information of the target object, respectively.

[0259] In actual execution, the color difference image is converted into a two-dimensional data structure, where one dimension represents each pixel in the color difference image, the other dimension includes the two-dimensional coordinates corresponding to each pixel, and the corresponding color feature, saturation feature, and brightness feature.

[0260] Then, the two-dimensional data structure corresponding to the color difference image is input to the nonlinear transformation layer, the random deactivation layer, and the fully connected layer for temperature feature and pressure feature extraction to obtain the temperature information and the pressure information of the corresponding target object, and then the temperature information and the pressure information are input to the regression layer to obtain the regression labels corresponding to the temperature information and the pressure information.

[0261] In some embodiments, the brightness dimension of the color of the temperature sensing layer 110 in the first color range is positively correlated with the temperature, and the hue dimension of the color of the pressure sensing layer 120 in the second color range is positively correlated with the pressure.

[0262] In this embodiment, the higher the temperature of the target object contacted by the tactile sensing structure, the brighter the temperature sensing layer 110, and the brightness dimension changes from no brightness to maximum brightness corresponding to the value from 1 to 0, and the higher the temperature of the target object contacted by the tactile sensing structure, the greater the value of the color of the temperature sensing layer 110 in the brightness dimension.

[0263] For example, when the temperature of the target object contacted by the tactile sensing structure is 25°C, the color of the temperature sensing layer 110 has a value of 0.5 in the lightness dimension; when the temperature of the target object contacted by the tactile sensing structure is 30°C, the color of the temperature sensing layer 110 has a value of 0.8 in the lightness dimension.

[0264] In this embodiment, the hue dimension angle of 0° or 360° can represent red, 120° can represent green, and 240° can represent blue. The greater the pressure applied to the target object by the tactile sensing structure, the greater the hue dimension angle value of the pressure sensing layer 120.

[0265] For example, when the target object first presses the tactile sensing structure, the hue dimension of the pressure sensing layer 120 can be red. As the pressure applied by the target object to the tactile sensing structure increases, the hue dimension of the pressure sensing layer 120 can gradually approach green.

[0266] The following describes a specific implementation of a tactile sensing model for training and a tactile perception method.

[0267] like Figure 7 As shown, firstly, a calibration image sample set is obtained. Based on the thermal image obtained by the thermal imager, the thermal image is upsampled so that the resolution of the thermal image is equal to the resolution of the calibration image in the calibration image sample set, i.e., the resolution of the calibration sample image. The temperature is labeled on the calibration sample image to obtain a calibration sample image with temperature calibration.

[0268] The pressure is labeled on the calibration sample image according to the pressure model, that is, the object model, to obtain the calibration sample image with pressure calibration.

[0269] The calibration sample images with temperature and pressure calibrations are input into the calibration model, i.e., the tactile sensing model, to train the tactile sensing model.

[0270] Then, the input image (i.e., the color detection image) and the reference frame (i.e., the color reference image) can be subtracted in the hexagonal pyramid model color space (i.e., the Hue Saturation Value, HSV) space to obtain the difference image, i.e., the color difference image.

[0271] The color difference image is input into the trained tactile sensing model to obtain the temperature and pressure outputs of the tactile sensing model, which correspond to the temperature and pressure information of the target object.

[0272] The tactile sensing method provided in this application can be executed by a tactile sensing device. This application uses the execution of the tactile sensing method by a tactile sensing device as an example to illustrate the tactile sensing device provided in this application.

[0273] This application also provides a tactile sensing device.

[0274] like Figure 8 As shown, the tactile sensing device includes:

[0275] The acquisition module 810 is used to acquire a color detection image of the tactile sensing structure in contact with the target object. The tactile sensing structure includes a temperature sensing layer 110 and a pressure sensing layer 120 stacked together. The color of the temperature sensing layer 110 changes within a first color range as the temperature of the tactile sensing structure changes. The color of the pressure sensing layer 120 changes within a second color range as the pressure of the tactile sensing structure changes. The first color range is the color range corresponding to the changes in the hue dimension, saturation dimension, and lightness dimension in the hexagonal pyramid model color space. The second color range is the color range corresponding to the changes in the hue dimension and saturation dimension in the hexagonal pyramid model color space.

[0276] The first processing module 820 is used to subtract the color detection image from the color reference image corresponding to the tactile sensing structure in the color space of the hexagonal pyramid model to obtain the color difference image corresponding to the target object.

[0277] The second processing module 830 is used to obtain temperature and pressure information of the target object in contact with the tactile sensing structure based on the hue, saturation and brightness features of the color difference image and the color tactile mapping relationship.

[0278] According to the tactile sensing method provided in the embodiments of this application, when a target object comes into contact with the tactile sensing structure, the temperature sensing layer 110 changes to the corresponding color according to the temperature of the target object, and the pressure sensing layer 120 changes to the corresponding color according to the pressure applied by the target object. The color changes of the two layers are superimposed on the color detection image of the tactile sensing structure. According to the color tactile mapping relationship, the temperature information and pressure information corresponding to the color detection image are obtained. The image signal of the color detection image corresponding to the target object is processed to obtain the temperature of the target object and the pressure applied to the tactile sensing structure. Multimodal recognition can be achieved using a single tactile sensor, and recognition is achieved through image signals, without the need for electrical signals output by the tactile sensing structure, making the recognition process simple.

[0279] In some embodiments, the second processing module 830 is used to input the color features, saturation features and brightness features of the color difference image into the tactile sensing model of the tactile sensing structure to obtain the temperature information and pressure information of the contact target object output by the tactile sensing model.

[0280] The tactile sensing model is trained based on a set of calibrated image samples with color-tactile mapping relationships.

[0281] In some embodiments, the second processing module 830 is configured to control the calibration object to contact the haptic sensing structure at the sample pressure at the sample temperature, and acquire a calibration sample image of the haptic sensing structure contacting the calibration object.

[0282] The second processing module 830 is configured to acquire a plurality of calibration sample images in the calibration image sample set by traversing the sample temperature in the sample temperature set and the sample pressure in the sample pressure set.

[0283] In some embodiments, the second processing module 830 is configured to control the calibration object to contact the haptic sensing structure at the sample pressure at the sample temperature, and acquire an initial sample image of the haptic sensing structure contacting the calibration object.

[0284] The second processing module 830 is configured to subtract the initial sample image and a color reference image corresponding to the haptic sensing structure in the hexagonal pyramid model color space based on the size information of the calibration object, to obtain the calibration sample image.

[0285] In some embodiments, the second processing module 830 is configured to construct an object model based on the size information of the calibration object, and establish a pressure label corresponding to the sample pressure.

[0286] The second processing module 830 is configured to establish a temperature label corresponding to the sample temperature based on the size information of the calibration object and the thermal imaging image of the calibration object.

[0287] In some embodiments, the second processing module 830 is configured to input the plurality of calibration sample images in the calibration image sample set to the haptic sensing model to be trained, and acquire predicted temperature information and predicted pressure information output by the haptic sensing model based on hue features, saturation features, and brightness features of the calibration sample images.

[0288] The second processing module 830 is configured to compare the predicted temperature information with the sample temperature, and compare the predicted pressure information with the sample pressure, update parameters of the haptic sensing model based on a comparison result, and obtain the trained haptic sensing model.

[0289] In some embodiments, the haptic sensing model includes a nonlinear change layer, a random inactivation layer, a full connection layer, and a regression layer. The nonlinear change layer, the random inactivation layer, and the full connection layer are configured to obtain temperature information and pressure information of a target object, and the regression layer is configured to obtain regression labels corresponding to the temperature information and the pressure information of the target object, respectively.

[0290] In some embodiments, the color of the temperature sensing layer 110 is positively correlated with the temperature in the brightness dimension in the first color range, and the color of the pressure sensing layer 120 is positively correlated with the pressure in the hue dimension in the second color range.

[0291] The haptic perception apparatus in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and can also be a server, a network attached storage (NAS), a personal computer (PC), and the like. The embodiments of the present application are not limited in this regard.

[0292] The haptic perception apparatus in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an IOS operating system, or other possible operating system, and the embodiments of the present application are not limited in this regard.

[0293] The haptic perception apparatus provided in the embodiments of the present application can implement Figures 5 to 7 The method embodiments implement various processes, and to avoid repetition, the processes are not described herein.

[0294] The embodiments of the present application further provide an agent.

[0295] The agent includes a haptic sensing structure and a processor, the processor is connected with the haptic sensing structure, and the processor is configured to implement the haptic perception method.

[0296] According to the agent provided in the embodiments of the present application, when the target object contacts the haptic sensing structure, the temperature sensing layer 110 changes to a corresponding color according to the temperature of the target object, the pressure sensing layer 120 changes to a corresponding color according to the pressure applied by the target object, the color changes of the two layers are superimposed on the color detection image of the haptic sensing structure, the temperature information and the pressure information corresponding to the color detection image are obtained according to the color haptic mapping relationship, the image signal of the color detection image corresponding to the target object is processed, the temperature of the target object and the pressure applied to the haptic sensing structure are obtained, one haptic sensor can be used to implement multi-modal recognition, and the image signal is used for recognition, and no electric signal output by the haptic sensing structure is needed, and the recognition process is simple.

[0297] In some embodiments, asFigure 9 As shown in the above-mentioned embodiments, the electronic device 900 further includes a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. The computer program is executed by the processor 901 to implement the processes of the above-mentioned embodiments of the method for perceiving touch and achieve the same technical effects. To avoid repetition, details are not described herein.

[0298] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0299] The embodiments of the present application further provide a non-transitory computer readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement the processes of the above-mentioned embodiments of the method for perceiving touch and achieve the same technical effects. To avoid repetition, details are not described herein.

[0300] The processor is the processor in the electronic device described in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0301] The embodiments of the present application further provide a computer program product including a computer program. The computer program is executed by a processor to implement the above-mentioned method for perceiving touch.

[0302] The processor is the processor in the electronic device described in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0303] The embodiments of the present application further provide a chip including a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run a program or an instruction to implement the processes of the above-mentioned embodiments of the method for perceiving touch and achieve the same technical effects. To avoid repetition, details are not described herein.

[0304] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip, etc.

[0305] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a", "comprising", or "includes a", does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Additionally, it should be noted that the scope of the methods and apparatus of the present embodiments are not limited by the order of the steps or the sequence for performing the steps, as some steps can occur simultaneously, in other steps can occur sequentially, or in between other steps can occur, as can be readily appreciated by those skilled in the art. Further, the features of certain examples described can be combined with features of other examples.

[0306] From the above description of the embodiments, it is apparent that the above-mentioned method of the embodiments can be realized by means of software and a general purpose hardware platform. Of course, it can also be realized by hardware, but in most cases, the former is a better embodiment. Based on such an understanding, the technical solution of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, computer, server, or network device) to execute the method described in the embodiments of the present application.

[0307] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-described specific embodiments, which are merely illustrative rather than restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

[0308] In the description of the present application, it should be understood that the terms "center", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are merely for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0309] In the description of the application, "a first feature", "a second feature" can include one or more of the features.

[0310] In the description of the application, "a plurality" means two or more.

[0311] In the description of the application, "on" or "under" of a first feature to a second feature can include that the first and second features are in direct contact, or can include that the first and second features are not in direct contact but are in contact through another feature between them.

[0312] In the description of the application, "on", "above" and "over" of a first feature to a second feature includes that the first feature is directly above and obliquely above the second feature, or only means that the first feature is higher than the second feature in horizontal height.

[0313] The embodiments of the application are described above in conjunction with the drawings, but the application is not limited to the specific embodiments described above, which are only illustrative, not restrictive, and those of ordinary skill in the art can make many forms under the inspiration of the application without departing from the purpose of the application and the scope protected by the claims, all of which belong to the protection of the application.

[0314] In the description of the application, the description of the terms "one embodiment", "some embodiments", "illustrative embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the application. In the description of the application, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0315] Although the embodiments of the application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the application, and the scope of the application is defined by the claims and their equivalents.

Claims

1. A tactile sensor characterized by, The haptic sensor comprises: a haptic sensing structure, the haptic sensing structure comprising a temperature sensing layer and a pressure sensing layer arranged in a stack, a color of the temperature sensing layer varying in a first color range with a temperature change of the haptic sensing structure, a color of the pressure sensing layer varying in a second color range with a pressure change of the haptic sensing structure; an image collector, the image collector being located at one side of the haptic sensing structure, the image collector being configured to collect a color image of the haptic sensing structure, the color image having image information in the first color range and the second color range for representing temperature information and pressure information of an object contacted by the haptic sensor; a support structure, the haptic sensing structure and the image collector being arranged in the support structure, the support structure defining a first accommodating space, the image collector and the haptic sensing structure being located at opposite ends of the first accommodating space.

2. The tactile sensor according to claim 1, characterized by, The first color range is a color range corresponding to variations of hue, saturation and lightness dimensions in a hexagonal cone model color space, and the second color range is a color range corresponding to variations of hue and saturation dimensions in the hexagonal cone model color space.

3. The tactile sensor according to claim 2, wherein The lightness dimension of the color of the temperature sensing layer in the first color range is positively correlated with the temperature, and the hue dimension of the color of the pressure sensing layer in the second color range is positively correlated with the pressure.

4. The tactile sensor according to claim 3, characterized by The first color range is a color range from white to black, and the second color range is a color range from red to blue.

5. The tactile sensor according to claim 1, characterized by The image collector is located at a side corresponding to a side of the pressure sensing layer away from the temperature sensing layer, and the pressure sensing layer is light-transmissive.

6. The tactile sensor according to any one of claims 1 to 5, wherein The temperature sensing layer is made of an organic reversible temperature-sensitive color-changing material.

7. The tactile sensor according to any one of claims 1 to 5, wherein The pressure sensing layer is made of a structural color elastic photopolymer.

8. The tactile sensor according to any one of claims 1 to 5, wherein The haptic sensor further comprises: a light source module, the light source module being arranged in the support structure and located at an end where the image collector is located, the light source module being configured to illuminate the first accommodating space to provide light compensation for the image collector to collect the image.

9. The tactile sensor according to claim 8, characterized by The light source module comprises a point light source and a light diffusion layer, and the point light source is arranged at a side of the light diffusion layer away from the haptic sensing structure.

10. The tactile sensor according to any one of claims 1 to 5, wherein The haptic sensor further comprises: a transparent layer, the transparent layer being arranged in the support structure and located between the image collector and the haptic sensing structure, the transparent layer being deformed with the deformation of the haptic sensing structure.

11. The tactile sensor according to any one of claims 1 to 5, wherein The haptic sensor further comprises: a wear-resistant layer, the wear-resistant layer being arranged at a side of the haptic sensing structure away from the image collector.

12. An e-skin, characterized by, The haptic sensor comprises: the haptic sensor according to any one of claims 1-11; a processor, the processor being connected to the haptic sensor, the processor being configured to determine temperature information and pressure information of an object contacted by the haptic sensor according to image information in the first color range and the second color range of the color image collected by the haptic sensor.

13. An agent, characterized in that The electronic skin comprises: the electronic skin according to claim 12.

Images