Multi-mode touch sensor, multi-mode sensing composite robot system and method

Abstract

The invention belongs to the technical field of sensors, and provides a multi-mode tactile sensor, a multi-mode sensing composite robot system and a multi-mode sensing method wherein the multi-mode tactile sensor comprises a first sensing layer, a thermal insulation layer, an autoclave material layer and a second sensing layer which are arranged in sequence; the first sensing layer comprises a first hot film resistor, a first cold film resistor and a first substrate, the first hot film resistor is used for acquiring a first tactile sensing signal, and the first tactile sensing signal comprises at least one of a steady-state signal, a dynamic signal and a sliding signal; the determination module is used for determining at least one of heat conductivity coefficient, thermal diffusivity, roughness, texture and sliding speed; the first cold film resistor is used for acquiring a temperature signal of a to-be-measured substance; the second sensing layer is used for acquiring a pressure signal and an environment temperature signal. According to the multimode touch sensor, the two sensing layers are integrated, detection of pressure and sliding speed can be achieved, and detection of the heat conductivity coefficient, the heat diffusivity, the roughness, the texture and the temperature of a contact substance can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of sensor technology, and in particular to a multi-mode tactile sensor, a multi-mode perception composite robot system and a method. Background Art

[0002] With the rapid development of artificial intelligence technology, robots can complete more complex tasks in daily life and bring more convenience to humans. How to give robots highly anthropomorphic multi-sensory and operational capabilities so that they can work in complex environments remains a technical difficulty.

[0003] In the prior art, tactile sensors can be divided into resistive, piezoelectric, capacitive, triboelectric, thermal induction, etc. according to different sensing principles. Tactile sensing attributes mainly include pressure, temperature, thermal conductivity, thermal diffusivity, roughness, texture, sliding, etc. However, existing sensors only have the ability to sense one or a few combined tactile sensing attributes, and can only detect a single attribute or a few attributes of an object, which is difficult to meet the multi-sensory needs of robots.

[0004] Most existing robot perceptions rely solely on visual perception, which can only achieve the acquisition of object image information. When grasping or identifying objects, due to the lack of multimodal tactile perception signals, it is difficult to grasp objects stably and difficult to identify objects of similar shapes. Summary of the invention

[0005] The present invention provides a multi-mode tactile sensor, a multi-mode perception composite robot system and a method, which are used to solve the defects in the prior art that the detection perception attributes are single or insufficient and it is difficult to meet the multi-perception requirements of the robot.

[0006] The present invention provides a multi-mode tactile sensor, comprising: a first sensing layer, a heat insulation layer, a pressure heat material layer and a second sensing layer arranged in sequence;

[0007] The first sensing layer includes a first hot film resistor, a first cold film resistor and a first substrate, the first hot film resistor is used to obtain a first tactile perception signal, the first tactile perception signal includes at least one of a steady-state signal, a dynamic signal and a sliding signal generated by the first hot film resistor contacting and / or sliding with the material to be measured; based on the first tactile perception signal, a tactile perception parameter of the material to be measured is determined, the tactile perception parameter includes at least one of the thermal conductivity, thermal diffusivity, roughness, texture and sliding speed of the material to be measured; the first cold film resistor is used to obtain a temperature signal of the material to be measured;

[0008] The second sensing layer includes a second hot film resistor, a second cold film resistor and a second substrate. The second hot film resistor is used to obtain a pressure signal; and the second cold film resistor is used to obtain an ambient temperature signal.

[0009] According to a multi-mode tactile sensor provided by an embodiment of the present invention, the pressure-thermal material layer includes multiple component phases, among which there are at least two component phases with different thermal conductivities and Young's moduli, the pressure-thermal material is used to convert pressure into a change in the thermal conductivity of the pressure-thermal material; the hot film resistor is an electrically heated thermistor, and the cold film resistor is an unheated thermistor.

[0010] According to a multi-mode tactile sensor provided by an embodiment of the present invention, the step of determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal comprises at least one of the following steps:

[0011] Determining the thermal conductivity of the substance to be measured based on a steady-state signal generated by the contact between the first thermal film resistor and the substance to be measured;

[0012] Determine the thermal diffusivity and / or roughness of the material to be measured based on a dynamic signal generated by the contact between the first thermal film resistor and the material to be measured;

[0013] Based on a sliding signal generated by the first hot film resistor and the substance to be measured contacting and sliding, a sliding speed and / or a texture of the substance to be measured is determined.

[0014] According to a multi-mode tactile sensor provided by an embodiment of the present invention, the method of determining the thermal diffusivity and / or roughness of the material to be tested based on the dynamic signal generated by the contact between the first hot film resistor and the material to be tested includes:

[0015] Based on the dynamic signal generated by the contact between the first thermal film resistor and the substance to be measured, the maximum slope and / or adjustment time of the dynamic signal is extracted to determine the roughness and / or thermal diffusivity of the substance.

[0016] According to a multi-mode tactile sensor provided by an embodiment of the present invention, the sliding signal generated by the first hot film resistor and the material to be tested contacting and sliding, determining the sliding speed and / or the texture of the material to be tested, includes:

[0017] Based on the macro characteristic principle, the sliding signal is filtered to obtain a macro characteristic signal with a lower frequency, and the sliding speed is determined based on the macro characteristic signal, and / or,

[0018] Based on the microscopic characteristic principle, the sliding signal is filtered to obtain a microscopic characteristic signal with a higher frequency, and the texture of the substance to be tested is determined based on the microscopic characteristic signal.

[0019] According to a multi-mode tactile sensor provided by an embodiment of the present invention, determining the texture of the substance to be tested based on the microscopic characteristic signal includes:

[0020] Performing Fourier transform on the microscopic characteristic signal to obtain a characteristic frequency;

[0021] Based on the characteristic frequency and the sliding speed, the texture of the substance to be tested is determined.

[0022] The present invention also provides a multi-mode perception composite robot system, comprising:

[0023] A composite robot, the composite robot having a manipulator, a manipulator arm and a controller, and / or a moving device, the moving device being used to move the composite robot;

[0024] A multi-mode tactile sensor, which is disposed at the working end of the manipulator and is used to obtain a multi-mode tactile perception signal of the manipulator grasping an object;

[0025] and / or a visual device, wherein the visual device is used to obtain image information of an object, and the controller identifies, locates, and determines the posture of the object based on the image information;

[0026] And / or a voice interaction device, wherein the voice interaction device is used for voice interaction of the composite robot.

[0027] The present invention also provides a method for object positioning, posture determination and grasping by integrating vision and touch of a multi-mode perception composite robot system, the method comprising:

[0028] The visual device acquires image information and determines the position of the object based on the image information;

[0029] The controller extracts the outline of the object based on the image information to determine the grasping posture of the object;

[0030] The multi-mode tactile sensor acquires tactile information, and the manipulator grasps the object based on the tactile information, the position of the object, and the grasping posture.

[0031] The present invention also provides a method for object recognition by integrating vision and touch of a multi-mode perception composite robot system, the method comprising:

[0032] The visual device acquires image information, and determines the type of the object to be detected based on the image information;

[0033] The multi-mode tactile sensor acquires a multi-mode tactile perception signal generated by contact with the object to be detected, and identifies the object to be detected based on the multi-mode tactile perception signal and the type of the object to be detected.

[0034] The present invention also provides an object grasping method of a sliding detection feedback control of a multi-mode perception composite robot system, the method comprising:

[0035] The multi-mode tactile sensor acquires a sliding signal generated during the process of the manipulator grasping an object;

[0036] The controller performs closed-loop feedback control of the gripping force of the manipulator based on the sliding signal, thereby achieving non-slip gripping of objects with minimum gripping force.

[0037] The multi-mode tactile sensor provided in the embodiment of the present invention integrates a first sensing layer, a heat insulation layer, a pressure-heat material layer and a second sensing layer. The first hot film resistor of the first sensing layer obtains a first tactile perception signal. The first tactile perception signal includes at least one of a steady-state signal, a dynamic signal and a sliding signal. Based on the steady-state signal, the dynamic signal and the sliding signal, the thermal conductivity, thermal diffusivity, roughness, texture and sliding speed of the material to be measured can be detected. The first cold film resistor of the first sensing layer can detect the temperature of the material to be measured. The second sensing layer can detect pressure and ambient temperature. It can be seen that the multi-mode tactile sensor of this embodiment integrates two sensing layers, which can detect pressure and sliding speed, as well as the thermal conductivity, thermal diffusivity, roughness, texture and temperature of the contact material.

[0038] Furthermore, the multimodal perception composite robot system provided by the embodiment of the present invention integrates a composite robot, a multimodal tactile sensor, a visual device and a voice interaction device, so that the system has multimodal perception capabilities of vision, touch and hearing as well as fine operation capabilities. The multimodal tactile sensor can be used to detect the multimodal tactile perception properties of materials, and the fusion of vision and touch can be used to achieve accurate identification and stable grasping of objects based on multimodal perception. Visual perception can be used to locate and determine the posture of objects, and sliding detection feedback can be used to control the grasping force to achieve stable grasping and fine operation of objects, which can meet the multi-perception requirements of the multimodal perception composite robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a schematic structural diagram of a multi-mode tactile sensor provided by an embodiment of the present invention;

[0041] Figure 2 This is one of the flow charts of determining the tactile perception parameters of the substance to be tested based on the first tactile perception signal provided by an embodiment of the present invention;

[0042] Figure 3This is a second flow chart of determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal provided by an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of a process for determining the thermal diffusivity and / or roughness of a substance provided by an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of a flow chart of determining a sliding speed and / or a texture of a substance to be tested provided by an embodiment of the present invention;

[0045] Figure 6 is a flow chart of a multi-mode tactile perception detection method provided by an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of the structure of a multi-mode perception composite robot system provided by an embodiment of the present invention;

[0047] Figure 8 It is a flow chart of a method for positioning, determining posture and grasping an object by integrating vision and touch provided in an embodiment of the invention;

[0048] Fig. 9 It is a schematic diagram of the process of recognizing an object by a multi-mode perception composite robot system provided by an embodiment of the present invention;

[0049] Fig.10 It is a schematic diagram of the process of grasping an object by a multi-mode perception composite robot system provided in an embodiment of the present invention.

[0050] Reference numerals:

[0051] 1. Multi-mode tactile sensor; 11. First sensing layer; 12. Heat insulation layer; 13. Pressure heat material layer; 14. Second sensing layer;

[0052] 2. Composite robot; 21. Manipulator; 22. Multi-degree-of-freedom arm;

[0053] 3. Mobile device;

[0054] 4. Visual device;

[0055] 5. Controller;

[0056] 6. Hearing device;

[0057] 7. Voice device. DETAILED DESCRIPTION

[0058] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0059] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0060] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0061] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0062] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0063] An embodiment of a first aspect of the present invention provides a multi-mode tactile sensor, Figure 1 The schematic diagram of the structure of the multi-mode tactile sensor provided by the embodiment of the present invention is illustrated as follows: Figure 1 As shown, the multi-mode tactile sensor includes a first sensing layer 11, a heat insulating layer 12, a pressure-heating material layer 13 and a second sensing layer 14 which are arranged in sequence.

[0064] It can be understood that the second sensing layer 14 is located at the bottom layer of the heat-insulating material layer 13, the thermal insulation layer 12 is located at the top layer of the heat-insulating material layer 13, and the first sensing layer 11 is located at the top layer of the thermal insulation layer 12. Then the first sensing layer 11, the thermal insulation layer 12, the heat-insulating material layer 13 and the second sensing layer 14 are arranged in sequence from the top layer to the bottom layer.

[0065] Among them, the first sensing layer 11 includes a first hot film resistor, a first cold film resistor and a first substrate, and the first hot film resistor and the first cold film resistor are integrated in the first substrate; the first hot film resistor is used to obtain a first tactile perception signal, and the first tactile perception signal includes at least one of a steady-state signal, a dynamic signal and a sliding signal generated by the contact and / or sliding of the first hot film resistor and the material to be measured; the first cold film resistor is used to obtain a temperature signal of the material to be measured.

[0066] It can be understood that the first hot film resistor on the first sensing layer 11 generates Joule heat (i.e., electric heating) under the action of the circuit, and its own temperature is higher than the ambient temperature. When the multi-mode tactile sensor contacts the substance to be measured, the first hot film resistor contacts the substance to be measured to exchange heat. The heat exchange process is affected by the thermal conductivity, thermal diffusivity, roughness, texture properties, and sliding speed of the substance to be measured, thereby affecting the temperature of the first hot film resistor. By utilizing the thermistor characteristics of the first hot film resistor and conditioning the circuit output, it is possible to detect multiple tactile perception properties of the substance to be measured.

[0067] It is understandable that based on the first tactile perception signal and in combination with the process signal perception method, the thermal conductivity, thermal diffusivity, roughness, texture of the material to be tested, and the sliding speed between the material to be tested and in contact can be detected.

[0068] It can be understood that the steady-state signal can be a steady-state signal generated by the contact between the first thermal film resistor and the substance to be measured, and the thermal conductivity of the substance to be measured can be detected based on the steady-state signal.

[0069] It can be understood that the dynamic signal can be a dynamic signal generated by the contact between the first thermal film resistor and the material to be measured, and the thermal diffusivity and roughness of the material to be measured can be detected based on the dynamic signal.

[0070] It can be understood that the sliding signal may be a sliding signal generated by the first thermal film resistor and the material to be measured contacting and sliding, and the sliding speed and the texture of the material to be measured may be detected based on the sliding signal.

[0071] It can be understood that the first cold film resistor is used to obtain the temperature signal of the material to be measured. Based on the temperature signal, the temperature of the material to be measured can be obtained, thereby realizing the detection of the temperature of the material to be measured. The first cold film resistor can also be used to perform temperature compensation on the first hot film resistor.

[0072] The second sensing layer 14 is used to obtain a second tactile perception signal, and the second tactile perception signal includes a pressure signal and an ambient temperature signal. The second sensing layer 14 includes a second hot film resistor, a second cold film resistor, and a second substrate. The second hot film resistor is used to obtain a pressure signal, based on which pressure can be detected; the second cold film resistor is used to obtain an ambient temperature signal, based on which ambient temperature can be detected, and the second cold film resistor can also be used to perform temperature compensation on the second hot film resistor.

[0073] It can be understood that the second sensing layer 14 uses the second thermal film resistor that is electrically heated above the ambient temperature to measure the change in thermal conductivity of the thermostatic material caused by pressure. The heat exchange process between the second thermal film resistor and the thermostatic material will be affected by the thermal conductivity of the thermostatic material. When pressure is applied, the thermostatic material deforms, causing the equivalent thermal conductivity of the thermostatic material to change, changing the heat exchange between the second thermal film resistor and the thermostatic material, and then changing the output signal of the second thermal film resistor. Based on this, pressure detection can be achieved.

[0074] It is understandable that the pressure can be obtained based on the pressure signal, thereby realizing the pressure detection during the contact process of the substance to be tested; and the ambient temperature can be detected based on the ambient temperature signal.

[0075] It can be understood that the autoclave material is composed of a plurality of component phases, among which there are at least two component phases with different thermal conductivity and Young's modulus, such as: an elastic polymer matrix, doped metal particles and pores; the autoclave material has an autoclave effect, in which the equivalent thermal conductivity of the material is regulated by the pressure and changes with the pressure. This is because when the autoclave material is subjected to pressure, the volume fractions of its various component phases change, thereby causing the equivalent thermal conductivity of the autoclave material to change.

[0076] The multi-mode tactile sensor provided in the embodiment of the present invention integrates a first sensing layer 11, a heat insulating layer 12, a pressure-heating material layer 13 and a second sensing layer 14. The first hot film resistor of the first sensing layer 11 is used to obtain a first tactile perception signal. The first tactile perception signal includes at least one of a steady-state signal, a dynamic signal and a sliding signal. Based on the steady-state signal, the dynamic signal and the sliding signal, the thermal conductivity, thermal diffusivity, roughness, texture and sliding speed of the material to be measured can be detected, and the first cold film resistor of the first sensing layer 11 can detect the temperature of the material to be measured; the second hot film resistor of the second sensing layer 14 is used to obtain a pressure signal, and the second cold film resistor is used to obtain a temperature signal, which can detect pressure and ambient temperature. It can be seen that the multi-mode tactile sensor of this embodiment integrates two sensing layers, which can detect pressure and sliding speed, as well as the thermal conductivity, thermal diffusivity, roughness, texture and temperature of the contact material.

[0077] It should be noted that the existing tactile sensors only have the ability to sense one or a few combined tactile sensing attributes, and can only detect a single attribute or a few attributes of an object. In addition, the structure is relatively complex, the preparation process is cumbersome, and the integration is not high, which cannot meet the multi-sensory needs of robots. The multi-mode tactile sensor of this embodiment can be used to detect pressure, ambient temperature, temperature of contact material, thermal conductivity, thermal diffusivity, roughness, texture, sliding speed, and has the advantages of high integration and simple structure.

[0078] It should be noted that the conditioning circuit of the multi-mode tactile sensor can be one of a constant temperature difference conditioning circuit, a constant voltage conditioning circuit, and a constant current conditioning circuit.

[0079] It can be understood that the heat insulation layer 12 is a material with a low thermal conductivity, which is used to reduce or eliminate the thermal crosstalk between the first sensing layer 11 and the second sensing layer 14 .

[0080] In one embodiment of the present invention, the first hot film resistor and the second hot film resistor are electrically heated thermistors, and the first cold film resistor and the second cold film resistor are unheated thermistors.

[0081] In one embodiment of the present invention, Figure 2 One of the flow charts of determining the tactile perception parameters of the substance to be tested based on the first tactile perception signal provided by an embodiment of the present invention is illustrated, such as Figure 2 As shown, determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal is achieved by the following steps:

[0082] Step 100: Acquire a first tactile perception signal.

[0083] Step 200: Determine a tactile perception parameter of the material to be tested based on the first tactile perception signal, where the tactile perception parameter includes at least one of thermal conductivity, thermal diffusivity, roughness, texture, and sliding speed.

[0084] It is understandable that the first tactile perception signal output by the first hot film resistor of the multi-mode tactile sensor in contact with the substance to be tested can be obtained by processing the first tactile perception signal output by the first hot film resistor using the process signal sensing method to obtain the tactile perception parameters of the substance to be tested.

[0085] In a specific embodiment of the present invention, Figure 3 The second flow chart of determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal provided by the embodiment of the present invention is illustrated. Figure 3 As shown, determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal includes at least one of the following steps:

[0086] Step 300: Determine the thermal conductivity of the substance to be measured based on a steady-state signal generated by the contact between the first thermal film resistor and the substance to be measured.

[0087] It can be understood that when the first sensing layer 11 contacts the substance to be measured, since the temperature of the first thermal film resistor on the first sensing layer 11 is higher than the temperature of the substance to be measured, heat exchange occurs between the two, and the heat exchange process is divided into two stages: a dynamic process and a steady-state process; in the dynamic process, the heat exchange between the first thermal film resistor and the substance to be measured is in dynamic fluctuation; in the steady-state process, the heat exchange between the first thermal film resistor and the substance to be measured tends to thermal equilibrium, and the output signal of the first thermal film resistor reaches a stable state.

[0088] It can be understood that the signal output by the first hot film resistor in a dynamic process is a dynamic signal, and the signal output by the first hot film resistor in a stable state is a steady-state signal.

[0089] It can be understood that a steady-state signal generated by the contact between the first thermal film resistor and the substance to be measured is obtained, and the thermal conductivity of the substance to be measured is detected based on the steady-state signal.

[0090] The heat exchange between the first thermal film resistor and the substance to be measured reaches thermal equilibrium, the first thermal film resistor reaches a stable state, and the output signal of the first thermal film resistor is recorded as a steady-state signal.

[0091] It should be noted that the steady-state signal is mainly affected by the thermal conductivity of the material to be measured, and the thermal conductivity of the material to be measured can be detected based on the steady-state signal.

[0092] Step 400: Determine the thermal diffusivity and / or roughness of the material to be tested based on a dynamic signal generated by the contact between the first thermal film resistor and the material to be tested.

[0093] Step 500: Determine the sliding speed and / or the texture of the material to be tested based on a sliding signal generated by the first hot film resistor and the material to be tested contacting and sliding.

[0094] It should be noted that step 300, step 400 and step 500 do not represent the order in which the steps are executed, and the three steps can be executed independently.

[0095] In one embodiment of the present invention, Figure 4 As shown, step 400, based on the dynamic signal generated by the contact between the first thermal film resistor and the material to be measured, determines the thermal diffusivity and / or roughness of the material, specifically includes the following steps:

[0096] Step 401: Acquire a dynamic signal generated by the contact between the first thermal film resistor and the substance to be measured.

[0097] Step 402: Detect the thermal diffusivity and / or roughness of the material to be tested based on the dynamic signal.

[0098] It can be understood that in the dynamic process, when the first thermal film resistor and the substance to be measured just come into contact, the temperature difference between the two is the largest, and the heat exchange process is the strongest. At this time, the output signal of the first thermal film resistor on the first sensing layer 11 changes most dramatically, and the signal change slope reaches the maximum. The maximum slope is mainly affected by the surface roughness of the substance to be measured. As the heat exchange between the first thermal film resistor and the substance to be measured tends to balance, the time when the dynamic process ends is recorded as the adjustment time, which is mainly affected by the thermal diffusivity of the substance to be measured. Finally, the first thermal film resistor and the substance to be measured reach thermal equilibrium, the first thermal film resistor reaches a stable state, and the output signal of the first thermal film resistor is recorded as a steady-state signal.

[0099] For example, the maximum slope and / or adjustment time of the dynamic signal is extracted, and the roughness and / or thermal diffusivity of the material to be tested is determined based on the maximum slope and / or adjustment time.

[0100] It can be understood that the maximum slope of the dynamic signal is extracted, and the surface roughness of the substance to be tested is detected based on the maximum slope.

[0101] It can be understood that the adjustment time of the dynamic signal is extracted, and the thermal diffusivity of the substance to be measured is detected based on the adjustment time.

[0102] In one embodiment of the present invention, Figure 5 As shown, step 500, based on the sliding signal generated by the first hot film resistor and the material to be tested contacting and sliding, determines the sliding speed and / or the texture of the material to be tested, specifically including the following steps:

[0103] Step 501: Obtain a sliding signal generated during the sliding process between the first thermal film resistor and the substance to be tested.

[0104] It can be understood that the sliding process is a relative sliding between the first hot film resistor and the substance to be measured; the signal output by the first hot film resistor during the sliding process is recorded as a sliding signal.

[0105] Step 502: extract characteristic signals based on filtering to determine the sliding speed and / or the texture of the material to be tested.

[0106] It can be understood that when the multi-mode tactile sensor and the material to be measured slide, the contact area between the first thermal film resistor on the first sensing layer 11 and the material to be measured will change from a higher temperature area that has reached thermal equilibrium with the first thermal film resistor to a lower temperature area on the material to be measured, thereby changing the heat exchange between the first thermal film resistor and the material to be measured; the greater the sliding speed, the larger the contact area of ​​the first thermal film resistor, and the stronger the heat exchange. By utilizing the thermal sensitive characteristics of the first thermal film resistor and outputting it through a conditioning circuit, the sliding speed can be detected by changing the output signal (sliding signal).

[0107] It should be noted that the texture structure on the surface of the material to be tested will form periodic micro air cavities corresponding to the texture. Due to the different thermal conductivities of air and the material to be tested, when the multi-mode tactile sensor and the material to be tested slide, the sliding signal will show a higher frequency signal change along with the texture structure; the lower frequency macroscopic characteristic signal and the higher frequency microscopic characteristic signal are extracted by filtering, the lower frequency macroscopic characteristic signal is used to detect the sliding speed, and the higher frequency microscopic characteristic signal is used to detect the texture of the material to be tested.

[0108] It can be understood that the sliding speed between the first thermal film resistor and the sliding signal generated during the sliding process of the material to be tested is combined with the method of extracting microscopic features and macroscopic features to detect the sliding speed between the material to be tested and the texture of the material to be tested.

[0109] For example, to detect the sliding speed, step 502 includes the following contents:

[0110] Based on the macro-characteristic principle, the sliding signal is filtered to obtain a macro-characteristic signal with a lower frequency; based on the macro-characteristic signal with a lower frequency, the sliding speed is determined.

[0111] It can be understood that the output signal of the first hot film resistor during the sliding process of the multi-mode tactile sensor and the material to be tested is obtained, and the lower frequency macro characteristic signal is extracted by filtering. The lower frequency macro characteristic signal is used to detect the sliding speed of the object, thereby realizing the detection of the sliding speed.

[0112] For example, to detect the texture of the substance to be tested, step 502 includes the following contents:

[0113] Based on the principle of microscopic characteristics, the sliding signal is filtered to obtain a microscopic characteristic signal with a higher frequency, and the texture of the material to be tested is determined based on the microscopic characteristic signal with a higher frequency.

[0114] It can be understood that the output signal of the multi-mode tactile sensor and the first hot film resistor during the sliding process of the material to be tested is obtained, and the higher frequency microscopic characteristic signal is extracted by filtering. The higher frequency microscopic characteristic signal is used to detect the texture of the material to be tested, thereby realizing the detection of the texture of the material to be tested.

[0115] For example, when the sliding speed and the texture of the material to be tested are detected simultaneously, step 502 includes the following contents:

[0116] Based on the principle of macroscopic and microscopic characteristics, the sliding signal is filtered to obtain a lower-frequency macroscopic characteristic signal and a higher-frequency microscopic characteristic signal; the texture of the material to be tested is detected based on the higher-frequency microscopic characteristic signal, and the sliding speed is detected based on the lower-frequency macroscopic characteristic signal.

[0117] In one embodiment of the present invention, in step 502, determining the texture of the substance to be tested based on the microscopic characteristic signal of a higher frequency includes the following:

[0118] The higher frequency microscopic characteristic signal is subjected to Fourier transform to obtain the characteristic frequency; based on the characteristic frequency and the sliding speed, the texture of the material to be tested is determined.

[0119] It can be understood that by performing Fourier transform on the microscopic characteristic signal to obtain the characteristic frequency, the texture of the material to be tested can be detected in combination with the sliding speed.

[0120] It should be noted that the spectrum diagram is obtained by Fourier transforming the microscopic characteristic signal. Characteristic frequency spectral lines and their multiple frequency spectral lines will appear in the spectrum diagram. The frequency of the characteristic spectral lines is affected by the texture of the material to be tested. The denser the texture on the surface of the material to be tested, the more peaks and troughs the multi-mode tactile sensor and the material to be tested pass through within a unit length during the sliding process, and the higher the frequency corresponding to the characteristic frequency spectral lines. Therefore, the texture of the material to be tested can be detected by utilizing the microscopic characteristic signal.

[0121] A specific multi-modal tactile perception detection method is provided below: Figure 6 The flowchart of the multi-mode tactile perception detection method provided by the embodiment of the present invention is illustrated as follows: Figure 6 As shown, the detection method includes:

[0122] Step 101 : obtaining an output signal of a first hot film resistor during the entire contact process between the multi-mode tactile sensor and the substance to be tested, wherein the output signal includes a dynamic signal corresponding to a dynamic process and a steady-state signal corresponding to a steady-state process.

[0123] Step 102: extracting the maximum slope and adjustment time of the dynamic signal by using the dynamic process, detecting the surface roughness of the material to be tested based on the maximum slope, and detecting the thermal diffusivity of the material to be tested based on the adjustment time;

[0124] Step 103: extract a steady-state signal using a steady-state process, and detect the thermal conductivity of the substance to be tested based on the steady-state signal.

[0125] Step 104 : obtaining an output signal of the first hot film resistor during the sliding process between the multi-mode tactile sensor and the material to be tested, where the output signal is a sliding signal.

[0126] Step 105: Filter the sliding signal to obtain a macroscopic characteristic signal and a microscopic characteristic signal; and detect the sliding speed based on the macroscopic characteristic signal.

[0127] Step 106: Perform Fourier transform on the microscopic characteristic signal to obtain a characteristic frequency; based on the characteristic frequency and the sliding speed, detect the texture of the substance to be tested.

[0128] It should be noted that step 101 and step 104 do not represent the order in which the steps are executed. Step 101 may be executed first and then step 104, or step 104 may be executed first and then step 101.

[0129] The embodiment of the second aspect of the present invention provides a multi-mode perception composite robot system, Figure 7 The schematic diagram of the structure of the multi-mode perception composite robot system provided by the embodiment of the present invention is illustrated as follows: Figure 7 As shown, the multi-mode perception composite robot system includes a composite robot 2 and a multi-mode tactile sensor 1.

[0130] The composite robot 2 has a manipulator 21 , the multi-mode tactile sensor 1 is disposed at a working end of the manipulator 21 , and the controller 5 is electrically connected to the composite robot 2 and the multi-mode tactile sensor 1 .

[0131] Among them, the composite robot 2 has a manipulator 21, a multi-degree-of-freedom arm 22 and a controller 5. The manipulator 21 is arranged at the end of the multi-degree-of-freedom arm 22. The manipulator 21 is used to grasp objects, and the multi-degree-of-freedom arm 22 is used to perform multi-dimensional movements; the multi-mode tactile sensor 1 is arranged at the working end of the manipulator 21. In this embodiment, the multi-mode tactile sensor 1 is located at the fingertips of the manipulator 21 and is used to detect the multi-mode tactile perception properties of the material to be tested; the controller 5 is electrically connected to the multi-degree-of-freedom arm 22, and the controller 5 is used to control the multi-dimensional movement of the multi-degree-of-freedom arm 22; the controller 5 is electrically connected to the manipulator 21, and is used to control the grasping action of the manipulator 21; the controller 5 is electrically connected to the multi-mode tactile sensor 1, and is used to obtain the output signal of the multi-mode tactile sensor 1, thereby detecting the multi-mode tactile perception properties of the material (object) to be tested.

[0132] It should be noted that the multi-mode tactile sensor may adopt the multi-mode tactile sensor structure provided by any of the above embodiments, and the multi-mode tactile sensor 1 may also adopt other tactile sensors.

[0133] The multi-mode perception composite robot system provided by the embodiment of the present invention integrates a multi-mode tactile sensor 1 formed by a first sensing layer 11, an insulating layer 12, a pressure-heating material layer 13 and a second sensing layer 14. It utilizes the method of process signal perception and extraction of macroscopic and microscopic features to detect the pressure, sliding speed, thermal conductivity, thermal diffusivity, roughness, texture and temperature between the contacting material and the material to be tested, thereby realizing the detection of multi-mode perception physical quantities, greatly enriching the tactile perception capability of the composite robot 2 and meeting the multi-perception requirements of the multi-mode perception composite robot system.

[0134] In one embodiment of the present invention, the multi-mode perception composite robot system further includes a mobile device 3, which is used to move the composite robot 2. For example, the mobile device 3 can be an automatic guided vehicle (AGV) with a mobile function, a wheeled mobile chassis, a foot-type mobile chassis, etc.

[0135] In one embodiment of the present invention, the multi-modal perception compound robot system also includes a voice interaction device, which is used for voice interaction of the compound robot 2.

[0136] It is understandable that the voice interaction device can be set on the composite robot 2.

[0137] For example, the voice interaction device includes an auditory device 6 and a voice device 7 , and both the auditory device 6 and the voice device 7 are arranged on the composite robot 2 .

[0138] According to an embodiment of the present invention, the multi-mode perception composite robot system further includes a visual device 4, which is disposed on the composite robot 2. The visual device 4 is used for visual perception of the surrounding environment to achieve object recognition, positioning and posture determination in the environment. For example, the visual device 4 can be a monocular camera, a binocular camera, etc.

[0139] It can be understood that the visual device 4 is used to obtain image information of the object, and the controller 5 locates and positions the object based on the image information. The controller 5 uses the visual device 4 to obtain image information, estimates the position of the object based on the image information, and locates the object; the controller 5 uses the image information to extract the outline of the object, and determines the grasping posture of the object, that is, the posture of the object.

[0140] Based on the above multi-mode perception composite robot system, this embodiment proposes a method for object positioning, posture determination and grasping by integrating vision and touch. Figure 8 The flowchart of the method for object positioning, posture determination and grasping by integrating vision and touch provided by the embodiment of the invention is illustrated as follows: Figure 8 As shown, the method comprises the following steps:

[0141] Step 801: The visual device acquires image information and determines the position of the object based on the image information.

[0142] It can be understood that the compound robot 2 uses the visual device 4 to capture images of the surrounding environment, and uses the images captured by the visual device 4 to determine the distance between the object and the compound robot 2 for locating the object to determine the position of the object.

[0143] Step 802: The controller extracts the outline of the object based on the image information and determines the grasping posture of the object.

[0144] Step 803: The multi-mode tactile sensor acquires tactile information, and the manipulator stably grasps the object based on the tactile information, the position of the object, and the grasping posture.

[0145] It can be understood that the outer contour of the object is extracted based on the image, and the direction perpendicular to the long axis is selected to grasp the object, so as to determine the grasping posture of the object and ensure the stability of the object grasping.

[0146] According to the multi-mode perception composite robot system of the embodiment of the present invention, the object is recognized and grasped based on the visual device 4 and the multi-mode tactile sensor 1.

[0147] It can be understood that the visual device 4 is used to obtain image information to identify the object, and then the multi-modal tactile sensor is used to obtain multi-modal tactile information to segment the object. The multi-modal perception composite robot system of this embodiment adopts a visual and tactile fusion method to identify and segment the object.

[0148] This embodiment provides a method for object recognition by integrating vision and touch, which includes the following contents:

[0149] The visual device acquires image information and determines the type of the object to be measured based on the image information; the multi-mode tactile sensor acquires the multi-mode tactile perception signal generated by contact with the object to be measured, and identifies the object to be measured based on the multi-mode tactile perception signal and the type of the object to be measured.

[0150] For example, Fig. 9 As shown, the object recognition method specifically includes the following steps:

[0151] Step 901: Use a visual device to obtain an image of an object.

[0152] It can be understood that the composite robot 2 captures the image of the object through the visual device 4 and performs preliminary recognition of the object;

[0153] Step 902: Perform preliminary recognition of the object in combination with the visual recognition model.

[0154] It is understandable that the image is input into a visual recognition model, such as YOLOV3, SSD model, etc., for object detection to achieve preliminary recognition of the object.

[0155] Step 903: Use the multi-mode tactile sensor 1 to obtain tactile perception parameters to segment the object.

[0156] It can be understood that the tactile information collected by the multi-mode tactile sensor 1 is further subdivided. Generally speaking, different objects have different shapes, sizes, hardness, thermal properties, electrical properties and surface textures. One or more sensing capabilities of the multi-mode tactile sensor are used to detect objects and identify them.

[0157] It should be noted that the multi-mode tactile sensor 1 is used for tactile perception detection of the material to be tested, but for actual scenarios, the tactile information detected by the multi-mode tactile sensor 1 alone cannot enable the robot 2 to have the ability to complete complex tasks. Therefore, the composite robot system of the embodiment of the present invention integrates the multi-mode tactile sensor 1, the visual device 4 and the voice interaction device, and has multiple perception capabilities (sight, touch, and hearing), and can complete various complex tasks such as environmental perception, object positioning and posture determination, object recognition and grasping, object movement and transportation, etc.

[0158] In one embodiment of the present invention, the object is identified and grasped based on the visual device 4 and the multi-mode tactile sensor 1 .

[0159] It can be understood that the visual device 4 is used to obtain image information to identify the object, and then the multi-modal tactile sensor is used to obtain multi-modal tactile information to segment the object. The composite robot system of this embodiment adopts a vision and touch fusion method to identify and segment the object.

[0160] It can be understood that the sliding detection feedback control method is used to achieve non-slip grasping of objects. Specifically, the multi-mode tactile sensor 1 is used to detect the sliding signal of the object, and the sliding signal is fed back to the controller 5. The controller 5 controls the gripping force of the manipulator 21 on the object based on the sliding signal to achieve non-slip grasping of the object with minimum gripping force.

[0161] This embodiment provides a method for grasping an object using sliding detection feedback control, the method comprising the following contents:

[0162] The multi-mode tactile sensor acquires the sliding signal generated by the robot when grasping an object. The controller performs closed-loop feedback control of the robot's grasping force based on the sliding signal of the object, achieving non-slip grasping of the object with the minimum grasping force.

[0163] For example, Fig.10 As shown, the method for grasping an object specifically includes the following steps:

[0164] Step 1001: Grasp the object using a manipulator, and detect sliding signals in real time using a multi-mode tactile sensor.

[0165] During the initial grasping process, the manipulator 21 applies a relatively small grasping force.

[0166] Step 1002: Determine whether sliding occurs by using the sliding signal detected in real time by the multi-mode tactile sensor.

[0167] It is understandable that the sliding detection function of the multi-mode tactile sensor 1 is used to determine whether sliding occurs during the process of the manipulator 21 grasping the object.

[0168] Step 1003: Sliding occurs, and the gripping force of the manipulator 21 is increased.

[0169] It is understandable that if slippage occurs, the gripping force of the manipulator 21 is gradually increased.

[0170] Step 1004: No sliding occurs and the gripping force remains unchanged.

[0171] It can be understood that if no sliding occurs, the existing gripping force is maintained unchanged to achieve stable gripping of the object.

[0172] The multi-mode perception composite robot system provided in the embodiment of the present invention integrates a composite robot 2, a multi-mode tactile sensor 1, a moving device 3, a visual device 4, an auditory device 6, and a voice device 7. It adopts a multi-sensory fusion method, so that the multi-mode perception composite robot system has multi-modal perception capabilities of vision, touch, and hearing, as well as fine operation capabilities. It can complete environmental perception, object positioning and posture determination, recognition and grasping, movement and transportation, and can be used in fields such as industrial production, medical rehabilitation, and life services, and has broad application value.

[0173] The following is a specific working process of the multi-mode perception composite robot system, which includes:

[0174] Taking the automatic picking scene of indoor items as an example, under the control of the controller 5, the mobile device 3 moves the composite robot 2 into the room, and then takes pictures of the indoor environment through the visual device 4 to obtain image information.

[0175] The controller 5 processes the image captured by the visual device 4 and inputs the pre-trained model to perform preliminary recognition of the object; the controller 5 uses the image captured by the visual device 4 to calculate the distance between the composite robot 2 and the object, and plans the corresponding driving path of the working end of the manipulator 21 and approaches the object; and the controller 5 uses the image to extract the outer contour of the object and determine the grasping posture of the object.

[0176] The controller 5 controls the multi-degree-of-freedom arm 22 to perform multi-dimensional motion based on the planned driving path, so that the fingertip of the manipulator 21 approaches the object and grasps the object.

[0177] When the fingertips of the manipulator 21 grasp an object, the multi-mode tactile sensor 1 on the manipulator 21 is used to detect in real time whether sliding occurs between the working end of the manipulator 21 and the object. If sliding occurs, the gripping force of the manipulator 21 is increased; if no sliding occurs, the gripping force is kept unchanged.

[0178] After the grasping is completed, the object is identified using the recognition algorithm that integrates the visual device 4 and the multi-modal tactile sensor 1 (vision and touch). First, the image captured by the visual device 4 is used for preliminary recognition, and then the multi-modal tactile sensor 1 is used to obtain multi-modal tactile information to subdivide the object, thereby achieving accurate identification of the object.

[0179] Furthermore, the visual device 4 is used to identify the picking position of the object, and the multi-mode perception composite robot system uses the multi-degree-of-freedom arm 22 and the manipulator 21 to pick the object to the corresponding position.

[0180] The multi-mode perception composite robot system provided by the embodiment of the present invention has multiple functions of environmental perception, object positioning and posture determination, object recognition and grasping, movement and transportation, thereby being able to accomplish a variety of complex tasks.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-mode tactile sensor, characterized in that: include: A first sensing layer, a heat insulating layer, a pressure heat material layer and a second sensing layer are sequentially arranged; The first sensing layer includes a first hot film resistor, a first cold film resistor and a first substrate, the first hot film resistor is used to obtain a first tactile perception signal, the first tactile perception signal includes at least one of a steady-state signal, a dynamic signal and a sliding signal generated by the first hot film resistor contacting and / or sliding with the material to be measured; based on the first tactile perception signal, a tactile perception parameter of the material to be measured is determined, the tactile perception parameter includes at least one of a thermal conductivity, a thermal diffusivity, a roughness, a texture and a sliding speed of the material to be measured; The first cold film resistor is used to obtain the temperature signal of the substance to be measured; The second sensing layer comprises a second hot film resistor, a second cold film resistor and a second substrate, wherein the second hot film resistor is used to obtain a pressure signal; The second cold film resistor is used to obtain an ambient temperature signal.

2. The multi-mode tactile sensor according to claim 1, characterized in that: The pressure-heat material layer includes multiple component phases, among which there are at least two component phases with different thermal conductivity and Young's modulus. The pressure-heat material is used to convert pressure into a change in the thermal conductivity of the pressure-heat material; the hot film resistor is an electrically heated thermistor, and the cold film resistor is an unheated thermistor.

3. The multi-mode tactile sensor according to claim 1, characterized in that: The step of determining the tactile perception parameter of the substance to be tested based on the first tactile perception signal comprises at least one of the following steps: Determining the thermal conductivity of the substance to be measured based on a steady-state signal generated by the contact between the first thermal film resistor and the substance to be measured; Determine the thermal diffusivity and / or roughness of the material to be measured based on a dynamic signal generated by the contact between the first thermal film resistor and the material to be measured; Based on a sliding signal generated by the first hot film resistor and the substance to be measured contacting and sliding, a sliding speed and / or a texture of the substance to be measured is determined.

4. The multi-mode tactile sensor according to claim 3, characterized in that: The determining of the thermal diffusivity and / or roughness of the material based on the dynamic signal generated by the contact between the first thermal film resistor and the material to be measured comprises: Based on the dynamic signal generated by the contact between the first thermal film resistor and the substance to be measured, the maximum slope and / or adjustment time of the dynamic signal is extracted to determine the roughness and / or thermal diffusivity of the substance.

5. The multi-mode tactile sensor according to claim 3, characterized in that: The determining of the sliding speed and / or the texture of the substance to be tested based on the sliding signal generated by the first hot film resistor and the substance to be tested contacting and sliding comprises: Based on the macro characteristic principle, the sliding signal is filtered to obtain a macro characteristic signal with a lower frequency, and the sliding speed is determined based on the macro characteristic signal, and / or, Based on the microscopic characteristic principle, the sliding signal is filtered to obtain a microscopic characteristic signal with a higher frequency, and the texture of the substance to be tested is determined based on the microscopic characteristic signal.

6. The multi-mode tactile sensor according to claim 5, characterized in that: The determining the texture of the substance to be tested based on the microscopic characteristic signal comprises: Performing Fourier transform on the microscopic characteristic signal to obtain a characteristic frequency; Based on the characteristic frequency and the sliding speed, the texture of the substance to be tested is determined.

7. A multi-mode perception composite robot system, characterized in that: include: A composite robot, the composite robot having a manipulator, a manipulator arm and a controller, and / or a moving device, the moving device being used to move the composite robot; A multi-mode tactile sensor, which is disposed at the working end of the manipulator and is used to obtain a multi-mode tactile perception signal of the manipulator grasping an object; and / or a visual device, wherein the visual device is used to obtain image information of an object, and the controller identifies, locates, and determines the posture of the object based on the image information; And / or a voice interaction device, wherein the voice interaction device is used for voice interaction of the composite robot.

8. A method for object positioning, posture determination and grasping by integrating vision and touch, characterized in that: The multi-modal perception compound robot system of claim 7, wherein the method comprises: The visual device acquires image information and determines the position of the object based on the image information; The controller extracts the outline of the object based on the image information to determine the grasping posture of the object; The multi-mode tactile sensor acquires tactile information, and the manipulator grasps the object based on the tactile information, the position of the object, and the grasping posture.

9. A method for object recognition by integrating vision and touch, characterized in that: The multi-modal perception compound robot system of claim 7, wherein the method comprises: The visual device acquires image information, and determines the type of the object to be detected based on the image information; The multi-mode tactile sensor acquires a multi-mode tactile perception signal generated by contact with the object to be detected, and identifies the object to be detected based on the multi-mode tactile perception signal and the type of the object to be detected.

10. A method for grasping an object by sliding detection feedback control, characterized in that: The multi-modal perception compound robot system of claim 7, wherein the method comprises: The multi-mode tactile sensor acquires a sliding signal generated during the process of the manipulator grasping an object; The controller performs closed-loop feedback control of the gripping force of the manipulator based on the sliding signal, thereby achieving non-slip gripping of objects with minimum gripping force.

Images