Visual tactile sensor, clamping force control method and device and related equipment

Abstract

The embodiment of the invention discloses a visual tactile sensor, a clamping force control method and device and related equipment, a clamping device is controlled to clamp an object to move, and the visual tactile sensor is arranged on the clamping device; in the process of controlling the clamping device to clamp the object to move, the clamping force of the clamping device is adjusted in response to event information output by an event camera on the visual touch sensor. Therefore, through the visual tactile sensor arranged on the clamping device, in the process of controlling the clamping device to clamp the object to move, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so that the object is prevented from sliding or even falling off, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to a visual-tactile sensor, a clamping force control method, a device and related equipment. Background Art

[0002] With the development of machine technology, the application scenarios of clamping devices or gripping devices are becoming more and more extensive. However, at present, in the process of controlling the clamping device to clamp an object, it is easy for the object to fall off the clamping device due to the difficulty in timely adjusting the force of the clamping device to clamp the object, resulting in a poor user experience. Summary of the invention

[0003] The embodiments of the present application provide a visual-tactile sensor, a clamping force control method, device, electronic device and storage medium. The visual-tactile sensor provided on the clamping device can, during the process of controlling the movement of the clamping device to clamp an object, timely adjust the clamping force of the clamping device in response to event information output by an event camera on the visual-tactile sensor to prevent the object from slipping or even falling off, thereby improving the user experience.

[0004] In a first aspect, an embodiment of the present application provides a visual-tactile sensor, wherein the visual-tactile sensor is disposed on a clamping device, and when the clamping device clamps an object, the visual-tactile sensor contacts the object, and the visual-tactile sensor includes a contact module, an image acquisition module, and a processing module;

[0005] The contact module is used to deform according to the force conditions;

[0006] The image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera, and the event camera is used to output event information according to the deformation of the contact module;

[0007] The processing module is used to adjust the clamping force of the clamping device based on the received event information.

[0008] In a second aspect, an embodiment of the present application provides a clamping force control method, comprising:

[0009] Controlling a clamping device to clamp an object and move it, wherein the clamping device is provided with the above-mentioned visual-tactile sensor;

[0010] In the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to the event information output by the event camera on the visual tactile sensor. In a third aspect, the embodiment of the present application further provides a clamping force control device, comprising:

[0011] A control unit, used for controlling a clamping device to clamp an object and move it, wherein the clamping device is provided with the visual-tactile sensor;

[0012] The force adjustment unit is used to adjust the clamping force of the clamping device in response to event information output by the event camera on the visual tactile sensor during the process of controlling the movement of the clamping device to clamp the object.

[0013] In a fourth aspect, an embodiment of the present application further provides a clamping device, characterized in that it comprises the above-mentioned visual-tactile sensor, wherein the visual-tactile sensor is arranged on the clamping device and at the contact point between the clamping device and the object clamped by the clamping device.

[0014] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the clamping force control methods provided in the embodiments of the present application.

[0015] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute any step of the clamping force control method provided in the embodiment of the present application.

[0016] In the seventh aspect, an embodiment of the present application also provides a computer program product, including a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs any step of the clamping force control method provided in the embodiment of the present application.

[0017] By adopting the scheme of the embodiment of the present application, the clamping device can be controlled to clamp the object and move it, and a visual tactile sensor is provided on the clamping device; in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to the event information output by the event camera on the visual tactile sensor. Therefore, by using the visual tactile sensor provided on the clamping device, in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so as to prevent the object from sliding or even falling off, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of an application scenario of the visual-tactile sensor provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a system architecture of the clamping force control method provided in the embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of an embodiment of the flow chart of the clamping force control method provided in the embodiments of the present application;

[0022] Figure 4 is a schematic structural diagram of a clamping force control device provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The following disclosure provides many different embodiments or examples to implement different features of the subject matter provided. Specific examples of components and arrangements are described below to make the disclosure concise. Of course, these are only examples and are not intended to be limiting. For example, in the following description, a first feature formed on or formed on a second feature may include an embodiment in which the first feature and the second feature are formed in direct contact, and may also include an embodiment in which an additional feature may be formed between the first feature and the second feature so that the first feature and the second feature may not be in direct contact. In addition, the disclosure may reuse reference numbers and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not inherently dictate the relationship between the various embodiments and / or configurations discussed.

[0026] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0027] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0030] It should be noted that the current common tactile sensor solutions are divided into piezoresistive, capacitive, photoelectric, electromagnetic and visual tactile. Among them, visual tactile tactile sensors are visual tactile sensors, which are the forefront and focus of tactile sensor development in recent years. The essence of visual tactile sensors is to use visual imaging technology and deep learning data processing methods to increase the measurement dimension of tactile information, that is, to perform perception measurements on the position, distribution and direction of contact force. Compared with the piezoresistive, capacitive, photoelectric, and electromagnetic tactile perception principles mentioned above, visual tactile sensors will have a more intuitive, clear, and independently decoupled image input, and then establish a mapping relationship between tactile information and image information through calibration training based on reinforcement learning to achieve higher-dimensional tactile measurement. In theory, visual tactile sensors can measure force information in various directions and different distribution positions, and the upper limit of measurement for characteristics such as tangential friction and slip is also higher.

[0031] However, the inventors have found that existing visual-tactile sensors have certain limitations, which are mainly reflected in the low frame rate of the image output by the image acquisition module. When the object contacted by the visual-tactile sensor moves rapidly, motion blur will occur, and a clear image cannot be captured, resulting in the response speed and accuracy of the visual-tactile sensor being limited by the frame rate. In addition, there is a large amount of redundant data in the image it outputs, which reduces the efficiency of back-end image processing, and ultimately makes it difficult for existing visual-tactile sensors to respond in a timely manner. For example, in the scenario of a clamping device applied to a robot, when the object grasped by the clamping device slides, the existing visual-tactile sensor cannot promptly feedback the force information and displacement information, resulting in the clamping device being unable to quickly respond to the sliding of the object, and even causing the object to fall off, resulting in a poor user experience.

[0032] Based on this, see Figure 1 The embodiment of the present application provides a visual tactile sensor. The visual tactile sensor is arranged on a clamping device. When the clamping device clamps an object, the visual tactile sensor contacts the object. The visual tactile sensor includes a contact module 11, an image acquisition module 13 and a processing module 14.

[0033] The contact module 11 is used to deform according to the force conditions.

[0034] The image acquisition module 13 is used to photograph the contact module 11 . The image acquisition module 13 includes an event camera. The event camera is used to output event information according to the deformation of the contact module 11 .

[0035] The processing module 14 is used to adjust the clamping force of the clamping device based on the received event information.

[0036] The contact module 11 is a module that is in direct contact with the object clamped by the clamping device. The specific shape of the contact module 11 can be adjusted according to actual conditions. For example, the specific shape of the contact module 11 is circular, square, or rectangular.

[0037] The field of view of the image acquisition module 13 covers the surface of the contact module 11 inside the visual-tactile sensor. When the surface of the contact module 11 inside the visual-tactile sensor is deformed, the deformation of the contact module 11 is detected and corresponding event information is output.

[0038] Among them, the end condition of adjusting the clamping force of the clamping device can be adjusted according to the actual situation, and the embodiment of the present application is not limited. For example, the processing module can adjust the clamping force of the clamping device based on the received event information until there is no event camera and no new event is generated. For another example, the processing module can also adjust the clamping force of the clamping device based on the received event information until the new event generated by the event camera does not meet the corresponding requirements for adjusting the clamping force of the clamping device (such as the number of events generated within a preset time exceeds the preset number of events).

[0039] It is understandable that the image acquisition module 13 in the traditional visual tactile sensor mostly adopts CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor, CIS). When shooting a motion scene through a CIS camera, due to the low output frame rate of the CIS camera, it is often impossible to capture the complete and clear motion information of the object, resulting in the response speed and accuracy of the visual tactile sensor being limited. At the same time, when shooting through a CIS camera, each frame of the image collected by the CIS camera will capture the dynamic area and the static area of ​​the object at the same time, and in the application of the visual tactile sensor, the static area in the field of view of the CIS camera belongs to a useless area, so the image captured by the CIS camera has more redundant information, which will cause the amount of data of the back-end image processing to increase and the efficiency to decrease, thereby further slowing down the response speed of the visual tactile sensor.

[0040] The image acquisition module in the visual tactile sensor provided in the embodiment of the present application adopts an event camera (EVS). The event camera is a new type of image sensor based on visual perception. Unlike the CIS camera, the event camera does not perceive the environment by capturing images, but perceives the environment by recording events of changes in light intensity. The working principle of the event camera is to detect changes in light intensity at the pixel level. When the light intensity increases or decreases (the change in light intensity exceeds the set threshold), the pixel triggers the event camera to shoot an event. Among them, the event includes a timestamp, position information and polarity, wherein the timestamp indicates the time when the event occurs, the position information indicates the pixel position where the event occurs, and the polarity indicates whether the light intensity becomes stronger or weaker, which can reflect the direction of movement. More importantly, the event camera only outputs event data in the dynamic area of ​​its field of view, and there is no event in the static area of ​​its field of view, and no event data will be output. That is, the event information in the embodiment of the present application only contains event data in the area where the contact module is deformed, and does not contain event data in other areas.

[0041] At the same time, since event cameras generally define a 2-bit data to represent the state of a pixel based on frame mode acquisition, the pixel states include 00, 01 and 10, where 00 indicates that the corresponding pixel state is no event, 01 indicates that the corresponding pixel state is a positive event (UP event), and 10 indicates that the corresponding pixel state is a negative event (DOWN event), while in traditional CIS cameras, each pixel provides an absolute value of light intensity. Therefore, compared with event cameras, traditional CIS cameras need to use more bits of data to represent each pixel state, such as 10 bits of data to represent a pixel state.

[0042] Therefore, the embodiment of the present application uses the event information output by the event camera for image processing, which greatly reduces redundant data and improves image processing efficiency, thereby making the visual tactile sensor have a faster response speed. In addition, since the event camera detects changes in light intensity, it can detect and output at a higher frame rate than the CIS camera, and can capture clearer and more complete motion images, that is, the visual tactile sensor can respond to changes in the contact module 11 faster and more accurately and output corresponding event information.

[0043] In some embodiments, the processing module 14 is further configured to increase the clamping force of the clamping device based on the received event information during the movement of the clamping device clamping the object.

[0044] There are many ways to increase the clamping force of the clamping device.

[0045] For example, during the movement of the clamping device clamping the object, when the processing module 1 receives the event information output by the event camera in the image acquisition module, the clamping force of the clamping device is directly increased according to the preset unit clamping force. For example, the preset unit clamping force is 1, and the current clamping force of the clamping device is 2. When the processing module 1 receives the event information output by the event camera in the image acquisition module, the clamping force of the clamping device is directly increased to 3.

[0046] It should be noted that the processing module does not immediately increase the clamping force of the clamping device upon receiving the event information. Instead, the clamping force of the clamping device will be directly increased only when the clamping device clamps the object and is in a moving state and event information is detected. To this end, in this example, it is necessary to judge the state of the clamping device. When it is determined that the clamping device clamps the object and is in a moving state, the clamping force of the clamping device will be directly increased only when event information is detected.

[0047] For another example, during the movement of an object clamped by the clamping device, when the processing module 1 receives event information output by the event camera in the image acquisition module, it can analyze the deformation information of the contact module 11 based on the received event information; determine whether the object clamped by the clamping device slides based on the deformation information; if the object slides, increase the clamping force of the clamping device.

[0048] Exemplarily, the specific process of the movement of the clamping device clamping the object may include the following steps:

[0049] Step A: The clamping device grabs and holds the object, and the contact module 11 in the visual-tactile sensor arranged on the clamping device contacts the object.

[0050] It should be noted that the visual-tactile sensor is arranged at the contact point between the clamping device and the object clamped by the clamping device.

[0051] Step B: During the movement of the clamping device clamping the object, if the contact module 11 is deformed, the event camera in the image acquisition module 13 outputs corresponding event information to the processing module 14 .

[0052] The moving process of the clamping device clamping the object refers to the process when the clamping device fixes and grasps the object, that is, the process after the clamping device clamps the object and starts to move.

[0053] Step C: When receiving the event information, the processing module 14 determines that the object is sliding or experiencing other unstable phenomena, and increases the clamping force (ie, the grasping force) of the clamping device to improve the stability when grasping the object.

[0054] Specifically, the clamping device grabs and clamps the object, and the contact module 11 contacts the object. In the process of the clamping device fixing and grasping the object, that is, after the clamping device clamps the object and starts to move (or the process of the clamping device clamping the object), if the object slides, the contact module 11 will be deformed, and the light intensity at the position where the contact module is deformed will also change. At this time, the event camera in the image acquisition module 13 outputs the event information of the deformation area of ​​the contact module 11 to the processing module 14, and the event information includes the pixel position, timestamp and corresponding polarity of the light intensity change, so that the processing module 14 can completely restore the whole process of the deformation of the contact module 11. It should be noted that the event information does not contain data (redundant data) in the static area. Finally, when the processing module 14 receives the event data, it is considered that the object slides, and then the clamping force of the clamping device is increased until no event information is generated, that is, until the object does not slide. Thus, the visual tactile sensor of the embodiment of the present application can respond to the deformation of the contact module 11 quickly and promptly, so as to make feedback and countermeasures more quickly.

[0055] In some embodiments, an initial force may be applied to the gripping device to grasp the object. If the processing module 14 receives event information, the force is continuously adjusted through the above steps until no event information is generated.

[0056] It can be understood that the above steps can also quantify the size of the initial force for stably grasping the object, as well as the smaller initial force that can stably grasp the object.

[0057] Exemplarily, an initial force is given to the clamping device to grasp the object. If the processing module 14 receives event information, the force is continuously adjusted through the above steps until no event information is generated, and the force without event information is selected as a smaller initial force that can stably grasp the object; if no event information is generated, the initial force is reduced until the processing module 14 receives event information and selects the last force without event information as a smaller initial force that can stably grasp the object.

[0058] It should be noted that the amplitude of adjusting the gripping force of the clamping device in the embodiment of the present application is a preset unit value. Wherein, adjusting the gripping force of the clamping device refers to increasing or decreasing the gripping force of the clamping device.

[0059] In some embodiments, the image acquisition module 13 includes multiple event cameras, and the field of view formed by the multiple event cameras covers the surface of the contact module 11 inside the visual-tactile sensor; or, the image acquisition module 13 includes at least one pair of image acquisition units, the image acquisition unit includes an event camera and a CIS camera, and the common field of view between the event camera and the CIS camera in the image acquisition unit covers the surface of the contact module 11 inside the visual-tactile sensor.

[0060] Specifically, the image acquisition module 13 may include multiple event cameras, that is, the visual tactile sensor is a binocular visual tactile sensor, or the visual tactile sensor is a multi-visual visual tactile sensor, wherein the fields of view of the multiple event cameras cover the surface of the contact module 11 inside the visual tactile sensor.

[0061] The image acquisition module 13 may also include a CIS camera, and the CIS camera and the event camera are arranged in binocular form. The common field of view of the CIS camera and the event camera covers the surface of the contact module 11 inside the visual tactile sensor.

[0062] It is understandable that the change in the light intensity on the inner surface of the contact module 11 will cause the change in the stress condition, so that the event camera outputs the corresponding event information, but the event camera cannot establish a one-to-one mapping relationship with the actual stress magnitude and can only obtain relative depth information. The embodiment of the present application can combine the CIS camera and the event camera to identify absolute stress and make up for the above shortcomings.

[0063] In actual applications, when the object is not moving, a CIS camera is used to count the deformation of the contact module 11 and record the corresponding stress and depth information. Then, when the object is moving, an event camera is used to count the deformation of the contact module 11 and record the corresponding depth information. The stress at each depth information recorded by the event camera is calculated based on the depth information and corresponding stress recorded by the CIS camera, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude.

[0064] The image acquisition module 13 may also include at least one event camera and at least one CIS camera. The event camera and the CIS camera may be arranged in pairs, each event camera corresponds to a CIS camera, and the common field of view between each event camera and its corresponding CIS camera can be summarized to completely cover the surface of the contact module 11 inside the visual tactile sensor.

[0065] Specifically, when the image acquisition module 13 is an image sensor that is a mixture of a CIS camera and an event camera, the mixed image sensor may be one or more, and the setting principle is to be able to completely cover the surface of the contact module 11 inside the visual-tactile sensor.

[0066] It should be noted that the image sensor that is a mixture of a CIS camera and an event camera is called a hybrid vision sensor (Hybrid Vision Sensor, HVS), which integrates a CIS camera and an event camera and supports the simultaneous output of CIS data and event data. It is understandable that, similar to the previous embodiment, the visual tactile sensor using HVS in this embodiment can also identify absolute stress, and calculate the stress under the depth information of the event data according to the depth information and the corresponding stress in the CIS data, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude. In addition, in this embodiment, by using HVS as the image acquisition module 13, unlike binocular or multi-visual visual tactile sensors, there is no need to additionally model the binocular multi-medium light refraction propagation error, which reduces the complexity of the design.

[0067] In some embodiments, the contact module 11 further includes a contact body, which is used to contact the object clamped by the clamping device so that the contact body is deformed according to the force applied to the object clamped by the clamping device, and the material of the contact body includes at least one of silicone or latex.

[0068] In this embodiment, the contact module 11 also includes at least one marking layer, each marking layer is provided with a marking point array, and the marking layer is provided on the surface of the contact body inside the visual tactile sensor; the image acquisition module 13 is used to photograph the marking layer and output event information when the marking layer is deformed.

[0069] Among them, the role of the marker layer is to calibrate the deformation information of the contact body through the displacement or deformation degree of the marker. The deformation information contains the attribute characteristics of the contact force. The displacement direction and degree of the marker point represent the distribution and magnitude of the contact force. The area of ​​the marker point is positively correlated with the normal force.

[0070] It can be understood that, by providing a marking layer including marking points, the image acquisition module 13 can capture the deformation of the contact module 11 more clearly and accurately.

[0071] It should be noted that the single-layer marking point array is dense and the single area is small, and the error of image processing is easily confused with the change in its area. Therefore, the contact module 11 can include multiple marking layers, each of which is provided with a marking point array, thereby improving the force contrast.

[0072] It should be noted that the shape, distribution density and size of the markers are also closely related to the recognition accuracy. Each marker corresponds to a local area of ​​the contact surface. In theory, the more markers there are, the larger the effective recognition area is, and the accuracy and sensitivity are also improved accordingly. However, high-density markers also bring difficulties to image processing and the characterization of other contact features.

[0073] In this embodiment, the contact module 11 further includes a transparent film layer, which is arranged on the surface of the contact body inside the visual tactile sensor; the image acquisition module 13 is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

[0074] The thin film layer is transparent, and the thinner the thin film layer is, the stronger the light transmittance is, and vice versa. The thickness of the thin film is not limited in the present embodiment.

[0075] By using a thin film layer instead of a marking layer, the image acquisition module 13 can more clearly and accurately capture the light intensity change caused by the deformation of the contact module 11 without making marking points, thereby outputting corresponding event information more accurately and timely.

[0076] In some embodiments, the visual-tactile sensor further includes an illumination module 12, which is used to illuminate the contact module 11 in the visual-tactile sensor so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11.

[0077] It should be noted that the lighting module provides illumination to ensure that the image acquisition module outputs clearer event information. The timing of the image acquisition module photographing the contact module is related to whether the contact module is deformed, and is not affected by whether the lighting module is activated.

[0078] The lighting module 12 is used to provide uniform lighting in the visual-tactile sensor, so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11 .

[0079] Furthermore, it is understandable that if the clamping device is flexible enough to adjust the magnitude of the force at different parts at the same time, then the embodiment of the present application can also determine which part of the force of the clamping device needs to be increased according to the location where the event occurs. For example, the moving direction of the object can be determined according to the change of the event, and then the force at the corresponding position can be adjusted to achieve stable grasping.

[0080] In summary, the visual-tactile sensor applied to the clamping device proposed in the embodiment of the present application can achieve a higher output frame rate by setting an image acquisition module 13 with an event camera, and the output event information does not contain data in the static area, which greatly reduces redundant data and has a faster response speed. At the same time, the processing module 14 increases the gripping force of the clamping device upon receiving the above event information, without the need to perform other processing on the event information, further improving the response and feedback speed of the visual-tactile sensor when applied to the clamping device, so that the clamping device can quickly respond to the sliding of the object, improve the gripping stability of the clamping device, and prevent the object from falling off.

[0081] In order to solve the problem that in the process of controlling the clamping device to clamp the object, it is easy to cause the object to fall off the clamping device due to the difficulty in timely adjusting the force of the clamping device to clamp the object, resulting in poor user experience, please refer to Figure 2 , embodiments of the present application provide a clamping force control method, device, electronic device, and computer-readable storage medium.

[0082] The clamping force control method provided in the embodiment of the present application can be applied to the system architecture 200. Figure 2 ,like Figure 2 As shown, the system architecture 200 may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0083] Users can use terminal devices 201, 202, 203 to interact with server 205 through network 204 to receive or send messages, etc.

[0084] Terminal devices 201, 202, 203 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Eperts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. The terminal may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. The terminal and the server may perform two-way communication via a network.

[0085] The server 205 may be a server that provides various services, such as a backend server that supports the pages displayed on the terminal devices 201, 202, and 203. The server may be an independent server, or a server network or server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0086] It should be noted that the clamping force control method provided in the embodiment of the present application is executed by a server, and accordingly, the clamping force control device is arranged in the server.

[0087] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 201, 202, 203 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0088] The following is a detailed description with reference to the accompanying drawings. In this embodiment, the execution subject is an electronic device that can obtain the target video. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0089] Please refer to Figure 3 The specific process of the clamping force control method can be as follows: Step 301 to Step 302, wherein:

[0090] Step 301: Control a clamping device to clamp an object and move it, wherein a visual tactile sensor is provided on the clamping device.

[0091] Among them, visual tactile sensors are Figure 1 The specific content of the visual-tactile sensor shown can be referred to the relevant description of the visual-tactile sensor, and will not be elaborated in the embodiment of the present application.

[0092] Step 302: In the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to event information output by the event camera on the visual tactile sensor.

[0093] There are many ways to adjust the clamping force, which can be adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0094] For example, based on the unit clamping force, the clamping force of the clamping device is directly increased. For example, in the process of controlling the movement of the clamping device to clamp the object, in response to the event information output by the event camera on the visual tactile sensor, the clamping force of the clamping device is adjusted, including: obtaining the unit clamping force for adjusting the clamping force of the clamping device; in the process of controlling the movement of the clamping device to clamp the object, in response to the event information output by the event camera on the visual tactile sensor, the current clamping force of the clamping device is increased based on the unit clamping force.

[0095] For another example, if within a preset time period, the number of times in response to event information output by the event camera on the visual-tactile sensor is greater than a preset number, the current clamping force of the clamping device is directly increased according to the maximum clamping force.

[0096] By way of example, the specific process of the clamping force control method provided in the embodiment of the present application may be: obtaining a movement instruction for an object; based on the movement instruction, controlling the clamping device to clamp the object according to a target clamping force, and controlling the clamping device to clamp the object to move; in the process of controlling the movement of the clamping device to clamp the object, in response to event information output by an event camera on a visual-tactile sensor, increasing the target clamping force to obtain a new target clamping force; controlling the clamping device to clamp the target object according to the new target clamping force until the movement instruction is completed.

[0097] Among them, the initial clamping force of the controlled clamping device can be determined; the clamping device is controlled to clamp the object according to the initial clamping force, and the clamping device is controlled to move; in the process of controlling the clamping device to move, the initial clamping force is adjusted according to the generation information of the event information of the visual-tactile sensor until it is adjusted to the target clamping force, and the target clamping force is used to indicate the minimum clamping force for controlling the clamping device to move and the visual-tactile sensor does not generate event information.

[0098] By adopting the clamping force control method of the embodiment of the present application, the clamping device can be controlled to clamp the object and move it, and a visual tactile sensor is provided on the clamping device; in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to the event information output by the event camera on the visual tactile sensor. Therefore, through the visual tactile sensor provided on the clamping device, in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so as to prevent the object from sliding or even falling off, thereby improving the user experience.

[0099] This embodiment further provides a clamping force control device, which can be integrated into an electronic device, such as a computer device. The computer device can be a terminal, a server, or other device, and this embodiment has no limitation on this.

[0100] For example, Figure 4 As shown, the clamping force control device may include:

[0101] The control unit 401 is used to control the clamping device to clamp the object and move it. The clamping device is provided with the above-mentioned visual tactile sensor.

[0102] The force adjustment unit 402 is used to adjust the clamping force of the clamping device in response to event information output by the event camera on the visual tactile sensor during the process of controlling the movement of the clamping device to clamp the object.

[0103] Optionally, in the device of the embodiment of the present application, in the force adjustment unit 402, in the process of controlling the movement of the clamping device to clamp the object, in response to the event information output by the event camera on the visual tactile sensor, the clamping force of the clamping device is adjusted, specifically including:

[0104] Obtaining a unit clamping force for adjusting the clamping force of the clamping device;

[0105] In the process of controlling the movement of the clamping device to clamp the object, in response to event information output by the event camera on the visual tactile sensor, the current clamping force of the clamping device is increased based on the unit clamping force.

[0106] By using the clamping force control device provided in the embodiment of the present application, the clamping device can be controlled to move the clamped object through the control unit 401, and the clamping device is provided with a visual tactile sensor; the force adjustment unit 402 adjusts the clamping force of the clamping device in response to the event information output by the event camera on the visual tactile sensor during the process of controlling the movement of the clamping device to clamp the object. Therefore, through the visual tactile sensor provided on the clamping device, in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so as to prevent the object from sliding or even falling off, thereby improving the user experience.

[0107] Correspondingly, such as Figure 1 As shown, an embodiment of the present application further provides a clamping device, which includes the visual-tactile sensor as described above in claim 1, and the visual-tactile sensor is arranged at the contact point between the clamping device and the object clamped by the clamping device.

[0108] The specific content can be referred to the relevant description of the visual tactile sensor, and will not be elaborated in the embodiments of the present application.

[0109] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal devices. Alternatively, the electronic device may be a server.

[0110] like Figure 5 As shown, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0111] The processor 501 is the control center of the electronic device 500, and uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or loading software programs and / or units stored in the memory 502, and calling data stored in the memory 502. The processor 501 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0112] In the embodiment of the present application, the processor 501 in the electronic device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions, such as:

[0113] Controlling the clamping device to clamp the object and move it, the clamping device being provided with the above-mentioned visual-tactile sensor;

[0114] In the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to event information output by the event camera on the visual tactile sensor.

[0115] The electronic device provided by the embodiment of the present application can be controlled to move the clamping device by clamping the object, and the clamping device is provided with a visual tactile sensor; in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to the event information output by the event camera on the visual tactile sensor. Therefore, through the visual tactile sensor provided on the clamping device, in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so as to prevent the object from sliding or even falling off, thereby improving the user experience.

[0116] Furthermore, various functions implemented by running the application stored in the memory 502 can also be found in the description of the aforementioned embodiments, which will not be repeated here.

[0117] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0118] Optional, such as Figure 5 As shown, the electronic device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 5 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0120] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0121] The audio circuit 505 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another electronic device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0122] The input unit 506 may be configured to receive input event signal sequences and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0123] The power supply 507 is used to supply power to various components of the electronic device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0124] although Figure 5 Not shown, the electronic device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0127] To this end, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform any of the clamping force control methods provided in the embodiments of the present application. For example, the computer program can perform the following steps of the clamping force control method:

[0128] Controlling the clamping device to clamp the object and move it, the clamping device being provided with the above-mentioned visual-tactile sensor;

[0129] In the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to event information output by the event camera on the visual tactile sensor.

[0130] By using the computer-readable storage medium provided in the embodiment of the present application, the clamping device can be controlled to clamp the object and move it, and a visual tactile sensor is provided on the clamping device; in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to the event information output by the event camera on the visual tactile sensor. Therefore, through the visual tactile sensor provided on the clamping device, in the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device can be adjusted in time according to the event information output by the event camera on the visual tactile sensor, so as to prevent the object from sliding or even falling off, thereby improving the user experience.

[0131] Furthermore, for the detailed steps of the above method steps, reference can be made to the description in the above embodiments, which will not be repeated here.

[0132] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0133] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0134] Since the computer program stored in the computer-readable storage medium can execute any one of the clamping force control methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the clamping force control methods provided in the embodiments of the present application can be achieved. Please see the previous embodiments for details and will not be repeated here.

[0135] According to one aspect of the present application, a computer program product is also provided, including a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the methods provided in various optional implementations of the above embodiments.

[0136] In the above-mentioned clamping force control device, computer-readable storage medium, electronic device, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the clamping force control device, computer-readable storage medium, computer program product, electronic device and its corresponding units described above can refer to the description of the clamping force control method in the above embodiment, and will not be repeated here.

[0137] The above is a detailed introduction to a visual-tactile sensor, a clamping force control method, a device, an electronic device, a computer-readable storage medium and a computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A visual tactile sensor, characterized in that: The visual tactile sensor is arranged on the clamping device. When the clamping device clamps an object, the visual tactile sensor contacts the object. The visual tactile sensor includes a contact module, an image acquisition module and a processing module. The contact module is used to deform according to the force conditions; The image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera, and the event camera is used to output event information according to the deformation of the contact module; The processing module is used to adjust the clamping force of the clamping device based on the received event information.

2. The visual-tactile sensor according to claim 1, characterized in that: The processing module is further configured to increase the clamping force of the clamping device based on the received event information during the movement of the clamping device clamping the object.

3. The visual-tactile sensor according to claim 2, characterized in that: The processing module is further used to increase the clamping force of the clamping device based on the received event information during the movement of the clamping device clamping the object, including: The processing module is also used to analyze the deformation information of the contact module according to the received event information during the movement of the object clamped by the clamping device; determine whether the object clamped by the clamping device slides according to the deformation information; if the object slides, increase the clamping force of the clamping device.

4. The visual-tactile sensor according to claim 1, characterized in that: The image acquisition module includes a plurality of event cameras, and the field of view formed by the plurality of event cameras covers the surface of the contact module inside the visual-tactile sensor; Alternatively, the image acquisition module includes at least one pair of image acquisition units, the image acquisition unit includes an event camera and a CIS camera, and the common field of view between the event camera and the CIS camera in the image acquisition unit covers the surface of the contact module inside the visual-tactile sensor.

5. The visual-tactile sensor according to claim 1, characterized in that: The contact module also includes a contact body, which is used to contact the object clamped by the clamping device so that the contact body is deformed according to the force applied to the object clamped by the clamping device. The material of the contact body includes at least one of silicone or latex.

6. The visual-tactile sensor according to claim 5, characterized in that: The contact module further comprises at least one marking layer, each of which is provided with a marking point array, and the marking layer is provided on the surface of the contact body inside the visual tactile sensor; the image acquisition module is used to photograph the marking layer, and output event information when the marking layer is deformed; Alternatively, the contact module further includes a transparent film layer, which is arranged on the surface of the contact body inside the visual-tactile sensor; the image acquisition module is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

7. The visual-tactile sensor according to any one of claims 1 to 6, characterized in that: The visual-tactile sensor further includes an illumination module, which is used to illuminate the contact module in the visual-tactile sensor, so that when the contact module is deformed, the image acquisition module captures the contact module and outputs event information according to the deformation of the contact module.

8. A method for controlling clamping force, characterized in that: include: Controlling a clamping device to clamp an object and move it, wherein the clamping device is provided with a visual-tactile sensor as claimed in any one of claims 1 to 7; In the process of controlling the movement of the clamping device to clamp the object, the clamping force of the clamping device is adjusted in response to event information output by the event camera on the visual-tactile sensor.

9. The clamping force control method according to claim 8, characterized in that: In the process of controlling the movement of the clamping device to clamp the object, adjusting the clamping force of the clamping device in response to event information output by the event camera on the visual tactile sensor includes: Obtaining a unit clamping force for adjusting the clamping force of the clamping device; In the process of controlling the movement of the clamping device to clamp the object, in response to the event information output by the event camera on the visual-tactile sensor, the current clamping force of the clamping device is increased based on the unit clamping force.

10. A clamping force control device, characterized in that: include: A control unit, used for controlling a clamping device to clamp an object and move it, wherein the clamping device is provided with a visual-tactile sensor according to any one of claims 1 to 7; The force adjustment unit is used to adjust the clamping force of the clamping device in response to event information output by the event camera on the visual tactile sensor during the process of controlling the movement of the clamping device to clamp the object.

11. A clamping device, characterized in that: The visual-tactile sensor comprises the visual-tactile sensor as claimed in any one of claims 1 to 7, wherein the visual-tactile sensor is arranged at the contact point between the clamping device and the object clamped by the clamping device.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the clamping force control method according to any one of claims 8 to 9.

13. A storage medium, characterized in that: It comprises a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the clamping force control method according to any one of claims 8 to 9.

Images