Stress analysis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a stress analysis method and device, electronic equipment and a storage medium, event information output by a camera according to a deformation event of a contact module can be obtained, the event information comprises a plurality of events and event data of the events, and the event data at least comprises pixel positions of trigger events; performing displacement analysis according to each event in the event information and event data of the events to obtain displacement information at each pixel position; and for each pixel position, carrying out stress analysis based on the displacement information on the pixel position to obtain a stress analysis result on the pixel position. On the basis, an event camera is arranged in an image acquisition module of the visual tactile sensor, so that a processing module of the visual tactile sensor can carry out stress analysis according to event information output by the event camera due to triggering of deformation of a contact module, and stress identification and feedback are carried out timely and accurately.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a force analysis method, device, electronic device and storage medium. Background Art

[0002] Existing visual tactile sensors use visual imaging technology and deep learning data processing methods to increase the measurement dimension of tactile information, that is, to sense and measure the position, distribution and direction of contact force. However, existing visual tactile sensors have certain limitations, which makes it impossible for existing visual tactile sensors to timely and accurately identify and feedback stress when the contact part is deformed by force. Summary of the invention

[0003] The embodiments of the present application provide a force analysis method, device, electronic device and storage medium. An event camera can be set in the image acquisition module of the visual-tactile sensor so that the processing module of the visual-tactile sensor can perform force analysis based on event information output by the event camera due to the deformation of the contact module, thereby timely and accurately identifying and feeding back stress and improving the user experience.

[0004] In a first aspect, an embodiment of the present application provides a force analysis method, which is applied to a processing module of a visual tactile sensor, wherein a contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force conditions, and the visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the method includes:

[0005] Acquire event information output by the event camera according to the deformation of the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data at least includes a pixel position triggering the event;

[0006] Performing displacement analysis according to each event in the event information and event data of the event to obtain displacement information at each pixel position;

[0007] For each of the pixel positions, force analysis is performed based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0008] In a second aspect, an embodiment of the present application further provides a force analysis device, which is applied to a processing module of a visual tactile sensor, wherein a contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force conditions, and the visual tactile sensor further includes an image acquisition module, and the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the device includes:

[0009] An event information acquisition unit, configured to acquire event information output by the event camera according to the deformation of the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data includes at least a pixel position triggering the event;

[0010] A displacement information acquisition unit, configured to perform displacement analysis according to each event in the event information and event data of the event to obtain displacement information at each pixel position;

[0011] The force analysis unit is used to perform force analysis on each pixel position based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the force analysis methods provided in the embodiments of the present application.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute any step of the force analysis method provided in the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application also provides a computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs any step of the force analysis method provided in the embodiment of the present application.

[0015] The scheme of the embodiment of the present application is adopted and applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object, and the contact module is used to deform according to the force situation. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera, which can obtain event information output by the event camera according to the deformation of the contact module, wherein the event information includes multiple events and event data of the event, and the event data at least includes the pixel position of the triggering event; displacement analysis is performed according to each event in the event information and the event data of the event, and displacement information at each pixel position is obtained; for each pixel position, force analysis is performed based on the displacement information at the pixel position, and the force analysis result at the pixel position is obtained. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the event camera due to the deformation of the contact module, thereby timely and accurately performing stress recognition and feedback, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of an application scenario of the visual-tactile sensor provided in an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of a system architecture for applying the force analysis method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of an embodiment of the force analysis method provided in the embodiments of the present application;

[0020] Figure 4 is a structural schematic diagram of a force analysis device provided in an embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The following disclosure provides many different embodiments or examples to implement different features of the subject matter provided. Specific examples of components and arrangements are described below to make the disclosure concise. Of course, these are only examples and are not intended to be limiting. For example, in the following description, a first feature formed on or formed on a second feature may include an embodiment in which the first feature and the second feature are formed in direct contact, and may also include an embodiment in which an additional feature may be formed between the first feature and the second feature so that the first feature and the second feature may not be in direct contact. In addition, the disclosure may reuse reference numbers and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not inherently dictate the relationship between the various embodiments and / or configurations discussed.

[0024] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0025] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0026] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0027] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0028] It should be noted that the current common tactile sensor solutions are divided into piezoresistive, capacitive, photoelectric, electromagnetic and visual tactile. Among them, visual tactile tactile sensors are visual tactile sensors, which are the forefront and focus of tactile sensor development in recent years. The essence of visual tactile sensors is to use visual imaging technology and deep learning data processing methods to increase the measurement dimension of tactile information, that is, to perform perception measurements on the position, distribution and direction of contact force. Compared with the piezoresistive, capacitive, photoelectric, and electromagnetic tactile perception principles mentioned above, visual tactile sensors will have a more intuitive, clear, and independently decoupled image input, and then establish a mapping relationship between tactile information and image information through calibration training based on reinforcement learning to achieve higher-dimensional tactile measurement. In theory, visual tactile sensors can measure force information in various directions and different distribution positions, and the upper limit of measurement for characteristics such as tangential friction and slip is also higher.

[0029] However, the inventors have found that the existing visual tactile sensors have certain limitations, which are mainly reflected in the low frame rate of the image output by the image acquisition module. When the object contacted by the visual tactile sensor moves rapidly, motion blur will occur, and a clear image cannot be captured, resulting in the response speed and accuracy of the visual tactile sensor being limited by the frame rate. In addition, there is a large amount of redundant data in the image output, which reduces the efficiency of the back-end image processing, and ultimately causes the existing visual tactile sensor to be unable to timely and accurately identify and feedback stress when its contact part is deformed by force.

[0030] Based on this, see Figure 1 The embodiment of the present application provides a visual tactile sensor. The visual tactile sensor is arranged on a clamping device. When the clamping device clamps an object, the visual tactile sensor contacts the object. The visual tactile sensor includes a contact module 11, an image acquisition module 13 and a processing module 14.

[0031] The contact module 11 is used to deform according to the force conditions.

[0032] The image acquisition module 13 is used to photograph the contact module 11 . The image acquisition module 13 includes an event camera. The event camera is used to output event information according to the deformation of the contact module 11 .

[0033] The processing module 14 is used to adjust the clamping force of the clamping device based on the received event information.

[0034] The contact module 11 is a module that is in direct contact with the object clamped by the clamping device. The specific shape of the contact module 11 can be adjusted according to actual conditions. For example, the specific shape of the contact module 11 is circular, square, or rectangular.

[0035] The contact module 11 further includes a contact body and a transparent film layer. The contact body is used to contact the object clamped by the clamping device so that the contact body is deformed according to the force applied to the object clamped by the clamping device. The material of the contact body includes at least one of silicone or latex. The transparent film layer is arranged on the surface of the contact body inside the visual tactile sensor; the image acquisition module 13 is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

[0036] It should be noted that the film layer is transparent, and the thinner the film layer is, the stronger the light transmittance is, and vice versa. The thickness of the film is not limited in the embodiments of the present application.

[0037] By using the transparent film layer, the image acquisition module 13 can more clearly and accurately capture the light intensity change caused by the deformation of the contact module 11, so as to output the corresponding event information more accurately and timely.

[0038] The field of view of the image acquisition module 13 covers the surface of the contact module 11 inside the visual-tactile sensor. When the surface of the contact module 11 inside the visual-tactile sensor is deformed, the deformation of the contact module 11 is detected and corresponding event information is output.

[0039] It is understandable that the image acquisition module 13 in the traditional visual tactile sensor mostly adopts CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor, CIS). When shooting a motion scene through a CIS camera, due to the low output frame rate of the CIS camera, it is often impossible to capture the complete and clear motion information of the object, resulting in the subsequent image processing being unable to accurately analyze the deformation of the visual tactile sensor contact module, thereby causing the visual tactile sensor to be unable to respond quickly and in a timely manner. At the same time, when shooting through a CIS camera, each frame of the image collected by the CIS camera will capture the dynamic area and static area of ​​the object at the same time, and in the application of the visual tactile sensor, the static area in the field of view of the CIS camera belongs to a useless area, so the image captured by the CIS camera has more redundant information, which will cause the amount of data in the back-end image processing to increase and the efficiency to decrease, thereby further slowing down the response speed of the visual tactile sensor.

[0040] The image acquisition module in the visual tactile sensor provided in the embodiment of the present application adopts an event camera (EVS). The event camera is a new type of image sensor based on visual perception. Unlike the CIS camera, the event camera does not perceive the environment by capturing images, but perceives the environment by recording events of changes in light intensity. The working principle of the event camera is to detect changes in light intensity at the pixel level. When the light intensity increases or decreases (the change in light intensity exceeds the set threshold), the pixel triggers the event camera to shoot an event. Among them, the event includes a timestamp, position information and polarity, wherein the timestamp indicates the time when the event occurs, the position information indicates the pixel position where the event occurs, and the polarity indicates whether the light intensity becomes stronger or weaker, which can reflect the direction of movement. More importantly, the event camera only outputs event data in the dynamic area of ​​its field of view, and there is no event in the static area of ​​its field of view, and no event data will be output. That is, the event information in the embodiment of the present application only contains event data in the area where the contact module is deformed, and does not contain event data in other areas.

[0041] At the same time, since event cameras generally define a 2-bit data to represent the state of a pixel based on frame mode acquisition, the pixel states include 00, 01 and 10, where 00 indicates that the corresponding pixel state is no event, 01 indicates that the corresponding pixel state is a positive event (UP event), and 10 indicates that the corresponding pixel state is a negative event (DOWN event), while in traditional CIS cameras, each pixel provides an absolute value of light intensity. Therefore, compared with event cameras, traditional CIS cameras need to use more bits of data to represent each pixel state, such as 10 bits of data to represent a pixel state.

[0042] Therefore, the embodiment of the present application uses the event information output by the event camera for image processing, which greatly reduces redundant data and improves image processing efficiency, thereby making the visual tactile sensor have a faster response speed. In addition, since the event camera detects changes in light intensity, it can detect and output at a higher frame rate than the CIS camera, and can capture clearer and more complete motion images, that is, the visual tactile sensor can respond to changes in the contact module 11 faster and more accurately and output corresponding event information.

[0043] In some embodiments, the image acquisition module 13 includes multiple event cameras, and the field of view formed by the multiple event cameras covers the surface of the contact module 11 inside the visual-tactile sensor; or, the image acquisition module 13 includes at least one pair of image acquisition units, the image acquisition unit includes an event camera and a CIS camera, and the common field of view between the event camera and the CIS camera in the image acquisition unit covers the surface of the contact module 11 inside the visual-tactile sensor.

[0044] Specifically, the image acquisition module 13 may include multiple event cameras, that is, the visual tactile sensor is a binocular visual tactile sensor, or the visual tactile sensor is a multi-visual visual tactile sensor, wherein the fields of view of the multiple event cameras cover the surface of the contact module 11 inside the visual tactile sensor.

[0045] The image acquisition module 13 may also include a CIS camera, and the CIS camera and the event camera are arranged in binocular form. The common field of view of the CIS camera and the event camera covers the surface of the contact module 11 inside the visual tactile sensor.

[0046] The image acquisition module 13 may also include at least one event camera and at least one CIS camera. The event camera and the CIS camera may be arranged in pairs, each event camera corresponds to a CIS camera, and the common field of view between each event camera and its corresponding CIS camera can be summarized to completely cover the surface of the contact module 11 inside the visual tactile sensor.

[0047] Specifically, when the image acquisition module 13 is an image sensor that is a mixture of a CIS camera and an event camera, the mixed image sensor may be one or more, and the setting principle is to be able to completely cover the surface of the contact module 11 inside the visual-tactile sensor.

[0048] It should be noted that the image sensor that is a mixture of a CIS camera and an event camera is called a hybrid vision sensor (Hybrid Vision Sensor, HVS), which integrates a CIS camera and an event camera and supports the simultaneous output of CIS data and event data. It is understandable that, similar to the previous embodiment, the visual tactile sensor using HVS in this embodiment can also identify absolute stress, and calculate the stress under the depth information of the event data according to the depth information and the corresponding stress in the CIS data, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude. In addition, in this embodiment, by using HVS as the image acquisition module 13, unlike binocular or multi-visual visual tactile sensors, there is no need to additionally model the binocular multi-medium light refraction propagation error, which reduces the complexity of the design.

[0049] In some embodiments, the visual-tactile sensor further includes an illumination module 12, which is used to illuminate the contact module 11 in the visual-tactile sensor so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11.

[0050] The lighting module 12 is used to provide uniform lighting in the visual-tactile sensor, so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11 .

[0051] Furthermore, it is understandable that if the clamping device is flexible enough to adjust the magnitude of the force at different parts at the same time, then the embodiment of the present application can also determine which part of the force of the clamping device needs to be increased according to the location where the event occurs. For example, the moving direction of the object can be determined according to the change of the event, and then the force at the corresponding position can be adjusted to achieve stable grasping.

[0052] In summary, an embodiment of the present application proposes a visual tactile sensor, wherein the visual tactile sensor includes a contact module, an illumination module, an image acquisition module and a processing module, and the contact module is in contact with an object. In particular, the present application sets an event camera in the image acquisition module, and sets a transparent film layer inside the visual tactile sensor in the contact module. When receiving the event information output by the event camera, each event in the event information is integrated to obtain the relative displacement size and displacement direction of each corresponding position of the transparent film; then, the relative stress of each corresponding position of the transparent film is determined according to the relative displacement size, and the displacement direction is determined as the direction of the corresponding relative stress. The present application captures the deformation of the contact module based on the event camera in the image acquisition module, and the event information outputted by the application has a high frame rate, so that the relative displacement size and displacement direction of each corresponding position of the transparent film can be calculated completely, clearly and accurately in the subsequent process, and the relative stress and direction of each corresponding position of the transparent film can be determined accordingly; at the same time, the event information does not contain redundant information related to the static area, which reduces the amount of data for subsequent image processing and improves the efficiency of image processing, thereby enabling the visual-tactile sensor to have a faster response speed and timely perform stress recognition and feedback when its contact part is deformed by force.

[0053] To solve the above problems, see Figure 2 , embodiments of the present application provide a force analysis method, device, electronic device, and computer-readable storage medium.

[0054] The stress analysis method provided in the embodiment of the present application can be applied to the system architecture 200. Figure 2 ,like Figure 2 As shown, the system architecture 200 may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0055] Users can use terminal devices 201, 202, 203 to interact with server 205 through network 204 to receive or send messages, etc.

[0056] Terminal devices 201, 202, 203 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Eperts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. The terminal may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. The terminal and the server may perform two-way communication via a network.

[0057] The server 205 may be a server that provides various services, such as a backend server that supports the pages displayed on the terminal devices 201, 202, and 203. The server may be an independent server, or a server network or server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0058] It should be noted that the force analysis method provided in the embodiment of the present application is executed by a server, and accordingly, the force analysis device is arranged in the server.

[0059] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 201, 202, 203 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0060] The following is a detailed description with reference to the accompanying drawings. In this embodiment, the execution subject is an electronic device that can obtain the target video. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0061] It should be noted that the force analysis method is applied to the processing module of the visual tactile sensor, and the visual tactile sensor can be set on a clamping device or other intelligent mechanical equipment. When the clamping device clamps the object, or the intelligent mechanical equipment grabs the object, the contact module of the visual tactile sensor contacts the object, and the contact module is used to deform according to the force situation. The visual tactile sensor also includes an image acquisition module, which is used to shoot the contact module, and the image acquisition module includes an event camera.

[0062] Among them, the visual-tactile sensor can refer to the relevant description of the visual-tactile sensor in the above embodiment, and will not be described in detail in this embodiment of the application.

[0063] Please refer to Figure 3 The specific process of the force analysis method can be as follows: Step 301 to Step 303, wherein:

[0064] Step 301: Acquire event information outputted by a deformation event camera of a contact module, wherein the event information includes a plurality of events and event data of the events, and the event data at least includes a pixel position of a triggering event.

[0065] The event refers to a deformation triggering event of the contact module and a triggering event of a pixel position where the light intensity changes when photographed by the camera.

[0066] The contact module also includes a transparent film layer and a contact body, the contact body is used to contact the object, and the transparent film layer is arranged on the surface of the contact body inside the visual tactile sensor; the image acquisition module is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

[0067] The specific content of the event data can be adjusted according to the actual situation, and the embodiment of the present application does not limit it. For example, the event data includes but is not limited to the pixel position where the light intensity of the event changes, the timestamp of the event, and the polarity of the event.

[0068] It should be noted that the event data also includes events corresponding to pixels whose light intensity has changed compared to the previous frame, which are measured by polarity. Polarity includes positive events and negative events. Positive events are used to indicate that the light intensity of the pixel point that triggered the event has increased, and negative events are used to indicate that the light intensity of the pixel point that triggered the event has decreased.

[0069] Step 302: Perform displacement analysis based on each event in the event information and the event data of the event to obtain displacement information at each pixel position.

[0070] Specifically, in response to receiving event information output by the event camera, the relative displacement magnitude and displacement direction of each corresponding position of the transparent film are obtained according to each event in the event information; the relative stress of each corresponding position of the transparent film is determined according to the relative displacement magnitude, and the displacement direction is determined as the direction of the corresponding relative stress.

[0071] It should be noted that when the contact module of the visual tactile sensor is subjected to force (for example, due to the relative motion of the object in contact), the contact module will be deformed and the light intensity at the corresponding position will change. At this time, the event camera in the image acquisition module will output the event information of the area where the light intensity of the transparent film changes (deformation area) to the processing module. The event information includes the pixel position, timestamp and corresponding polarity where the light intensity changes, and can completely restore the entire process of the deformation of the transparent film, that is, the event data of each frame captured during the deformation of the transparent film. The frame rate of the deformation event image is much higher than the image frame rate output by the CIS camera, and does not contain data in the static area (redundant data). It should be noted that the event data of each frame contains events corresponding to pixels whose light intensity has changed compared to the previous frame, specifically including positive events and negative events. Among them, positive events refer to event data output by pixels whose light intensity increases, and negative events refer to event data output by pixels whose light intensity decreases.

[0072] The embodiment of the present application integrates each event in the event information and considers the polarity of the event (positive event and negative event) to obtain the relative displacement magnitude and displacement direction of each corresponding position of the transparent film, wherein each corresponding position refers to the position on the transparent film corresponding to each event.

[0073] In some embodiments, the event data also includes a timestamp of the event; a displacement analysis is performed based on each event in the event information and the event data of the event to obtain displacement information at each pixel position, including: sorting each event in the event information according to the timestamp to obtain event information to be analyzed, the event information to be analyzed includes each event sorted according to the timestamp; determining optical flow information based on each event in the event data to be analyzed and the event data of the event; determining the displacement information at each pixel position based on the optical flow information.

[0074] There are many ways to determine the optical flow information, for example, a sparse optical flow algorithm (such as Lucas-Kanade algorithm) can be used to determine the optical flow information based on each event in the event data to be analyzed and the event data of the event. For another example, a dense optical flow algorithm can be used to determine the optical flow information based on each event in the event data to be analyzed and the event data of the event.

[0075] Specifically, in some optional embodiments, based on each event in the event data to be analyzed and the event data of the event, the process of determining the optical flow information may include: integrating the events at the same timestamp based on the timestamps of each event in the event data to be analyzed to obtain a frame of event image at each timestamp; calculating the difference in each pixel position between event images that are adjacent in time to obtain the optical flow information.

[0076] Specifically, time alignment is first performed according to the timestamps of each event data, and each event is arranged in chronological order and mapped to a continuous timeline. Then, the processed event data is integrated to obtain event images of each frame, and the optical flow information of each event image is calculated by comparing the differences between consecutive event images.

[0077] It can be understood that the optical flow information corresponding to the event information includes the optical flow information corresponding to each corresponding position of each transparent film. The displacement size and displacement direction of each corresponding position of the transparent film are estimated through the optical flow information.

[0078] By integrating each event in the event information and considering the polarity of the event (positive event and negative event), the relative displacement magnitude and displacement direction of each corresponding position of the transparent film layer are obtained. Each corresponding position refers to the position on the transparent film corresponding to each event.

[0079] The embodiment of the present application preferably uses a sparse optical flow algorithm to calculate the optical flow information corresponding to the event information, which is more adapted to the dynamic range, temporal resolution and other characteristics of the event camera, and can obtain more accurate optical flow information in high-contrast scenes.

[0080] In some optional embodiments, before performing displacement analysis based on each event in the event information and the event data of the event to obtain the displacement information at each pixel position, the event information can also be subjected to event data filtering to eliminate events whose light intensity changes (i.e., brightness changes) are less than a preset change value, so as to remove unstable events caused by noise and improve the overall accuracy of the force analysis method.

[0081] In addition, considering factors such as lens distortion and pixel response non-uniformity of the event camera, the embodiment of the present application can also perform spatial correction based on the event data to ensure the accuracy of the pixel value.

[0082] In some optional embodiments, the process of performing displacement analysis based on each event in the event information and the event data of the event to obtain the displacement information at each pixel position may include: performing displacement analysis based on each event in the event information, the pixel position corresponding to the event, and the polarity of the event to obtain the relative displacement and displacement direction at each pixel position.

[0083] Step 303: For each pixel position, force analysis is performed based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0084] There are many ways to analyze the force, which can be adjusted according to the actual situation and are not limited in the embodiments of the present application.

[0085] For example, for each pixel position, a force analysis is performed based on the displacement information at the pixel position to obtain the force analysis result at the pixel position. The process may include: for each pixel position, using a preset force recognition model, a force analysis is performed based on the relative displacement at the pixel position to obtain the relative stress at the pixel position; according to the displacement direction at the pixel position, the stress direction of the relative stress at the pixel position is determined; and the relative stress at the pixel position and the stress direction of the relative stress are used as the force analysis result at the pixel position.

[0086] Among them, the above-mentioned preset force recognition model refers to a stress recognition model. The stress recognition model can be trained through sample data such as event data under different forces and corresponding displacement sizes collected by the visual-tactile sensor. The specific training process belongs to the conventional method and will not be elaborated here.

[0087] For another example, the force analysis result includes the relative stress at the pixel position and the stress direction of the relative stress; after performing force analysis on each pixel position based on the displacement information at the pixel position to obtain the force analysis result at the pixel position, the method further includes: obtaining a mapping relationship between the relative stress and depth information of the contact module; determining the target depth information recorded by the event camera based on the relative stress and stress direction at each pixel position; and determining the stress information on the contact module based on the target depth information and the mapping relationship.

[0088] Prior to this embodiment, it should be noted that the image acquisition module also includes a CIS camera, and the common field of view between the event camera and the CIS camera covers the surface of the contact module inside the visual-tactile sensor; before obtaining the event information output by the event camera based on the deformation of the contact module, the method also includes: if it is determined that the object has not slipped, obtaining image data output by the CIS camera based on the deformation of the contact module; based on the image data output by the CIS camera, determining the mapping relationship between the relative stress of the contact module and the depth information.

[0089] Specifically, the image acquisition module 13 may further include a CIS camera, and the CIS camera and the event camera are arranged in binocular form. The common field of view of the CIS camera and the event camera covers the surface of the contact module 11 inside the visual tactile sensor.

[0090] It is understandable that the change in the light intensity on the inner surface of the contact module 11 will cause the change in the stress condition, so that the event camera outputs the corresponding event information, but the event camera cannot establish a one-to-one mapping relationship with the actual stress magnitude and can only obtain relative depth information. The embodiment of the present application can combine the CIS camera and the event camera to identify absolute stress and make up for the above shortcomings.

[0091] For example, during the movement of the clamping device clamping the object, if it is determined that the object has not slipped, the image data output by the deformation CIS camera based on the contact module is obtained; based on the image data output by the CIS camera, the mapping relationship between the relative stress of the contact module and the depth information is determined.

[0092] In actual applications, when the object is not moving, a CIS camera is used to count the deformation of the contact module 11 and record the corresponding stress and depth information. Then, when the object is moving, an event camera is used to count the deformation of the contact module 11 and record the corresponding depth information. The stress at each depth information recorded by the event camera is calculated based on the depth information and corresponding stress recorded by the CIS camera, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude.

[0093] The visual tactile sensor using HVS in this embodiment can also identify absolute stress, and calculate the stress under the depth information of the event data according to the depth information and the corresponding stress in the CIS data, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude. In addition, in this embodiment, by using HVS as the image acquisition module, unlike binocular or multi-visual visual tactile sensors, there is no need to additionally model the binocular multi-medium light refraction propagation error, which reduces the complexity of the design.

[0094] The scheme of the embodiment of the present application is adopted and applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object, and the contact module is used to deform according to the force situation. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera, which can obtain event information output by the event camera according to the deformation of the contact module, wherein the event information includes multiple events and event data of the event, and the event data at least includes the pixel position of the triggering event; displacement analysis is performed according to each event in the event information and the event data of the event to obtain displacement information at each pixel position; for each pixel position, force analysis is performed based on the displacement information at the pixel position to obtain the force analysis result at the pixel position. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the event camera due to the deformation of the contact module, so as to timely and accurately perform stress identification and feedback, thereby improving the user experience. In some optional embodiments, the method may also include: determining the motion state of the object contacted by the visual tactile sensor according to the relative displacement size and displacement direction.

[0095] The scheme of the embodiment of the present application is applied to the processing module of the visual tactile sensor, which includes a contact module, an illumination module, an image acquisition module and a processing module, and the contact module is in contact with the object. In particular, the present application sets an event camera in the image acquisition module, and sets a transparent film in the contact module. When receiving the event information output by the event camera, each event in the event information is integrated to obtain the relative displacement size and displacement direction of each corresponding position of the transparent film; then, the relative stress of each corresponding position of the transparent film is determined according to the relative displacement size, and the displacement direction is determined as the direction of the corresponding relative stress. The present application captures the deformation of the contact module based on the event camera in the image acquisition module, and the event information output by the event camera has a high frame rate, so that the relative displacement size and displacement direction of each corresponding position of the transparent film can be calculated completely, clearly and accurately in the future, and the relative stress and direction of each corresponding position of the transparent film can be determined in this way; at the same time, the event information does not contain redundant information related to the static area, which reduces the amount of data for subsequent image processing and improves the efficiency of image processing, so that the visual tactile sensor has a faster response speed, and timely stress recognition and feedback are performed when its contact part is deformed by force.

[0096] This embodiment further provides a force analysis device, which can be integrated into an electronic device, such as a computer device. The computer device can be a terminal, a server, or other device, and this embodiment has no limitation on this.

[0097] For example, Figure 4As shown, the force analysis device is applied to the processing module of the visual tactile sensor, the visual tactile sensor is arranged on the clamping device, when the clamping device clamps the object, the contact module of the visual tactile sensor contacts the object, the contact module is used to deform according to the force situation, the visual tactile sensor also includes an image acquisition module, the image acquisition module is used to shoot the contact module, and the image acquisition module includes an event camera; the force analysis device may include:

[0098] An event information acquisition unit 401 is used to acquire event information outputted by a deformation event camera of a contact module, wherein the event information includes a plurality of events and event data of the events, and the event data includes at least a pixel position of a triggering event;

[0099] The displacement information acquisition unit 402 is used to perform displacement analysis according to each event in the event information and the event data of the event to obtain the displacement information at each pixel position;

[0100] The force analysis unit 403 is used to perform force analysis on each pixel position based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0101] Optionally, in the device provided in the embodiment of the present application, the event data further includes a timestamp of the event. The displacement information acquisition unit 402 includes:

[0102] A sorting subunit, used to sort the events in the event information according to the timestamps to obtain the event information to be analyzed, wherein the event information to be analyzed includes the events sorted according to the timestamps;

[0103] An optical flow information determination subunit, used to determine optical flow information according to each event in the event data to be analyzed and the event data of the event;

[0104] The displacement information determination subunit is used to determine the displacement information at each pixel position according to the optical flow information.

[0105] Optionally, in the device provided in the embodiment of the present application, the optical flow information determination subunit specifically includes:

[0106] According to the timestamps of each event in the event data to be analyzed, the events at the same timestamp are integrated to obtain a frame of event image at each timestamp;

[0107] The difference in each pixel position between event images that are adjacent in time is calculated to obtain the optical flow information.

[0108] Optionally, in the device provided in the embodiment of the present application, the above-mentioned event data also includes the polarity of the event, and the polarity includes positive events and negative events. The positive event is used to indicate that the light intensity of the pixel point that triggers the event increases, and the negative event is used to indicate that the light intensity of the pixel point that triggers the event decreases.

[0109] The displacement information acquisition unit 402 includes:

[0110] The relative displacement determination subunit is used to perform displacement analysis according to each event in the event information, the pixel position corresponding to the event, and the polarity of the event to obtain the relative displacement and displacement direction at each pixel position.

[0111] Optionally, in the device provided in the embodiment of the present application, the force analysis unit 403 includes:

[0112] A relative stress determination subunit is used to perform force analysis based on the relative displacement at each pixel position by using a preset force recognition model to obtain the relative stress at the pixel position;

[0113] A stress direction determination subunit, used to determine the stress direction of the relative stress at the pixel position according to the displacement direction at the pixel position;

[0114] The force analysis result determination subunit is used to take the relative stress at the pixel position and the stress direction of the relative stress as the force analysis result at the pixel position.

[0115] Optionally, in the device provided in the embodiment of the present application, the image acquisition module further includes a CIS camera, and the common field of view between the event camera and the CIS camera covers the surface of the contact module inside the visual-tactile sensor.

[0116] Before the event information acquisition unit 401, the device may further include:

[0117] A displacement information determination unit, used to obtain image data output by a deformation CIS camera based on a contact module if it is determined that the object has not slipped;

[0118] The mapping relationship determining unit is used to determine the mapping relationship between the relative stress of the contact module and the depth information based on the image data output by the CIS camera.

[0119] Optionally, in the device provided in the embodiment of the present application, the above-mentioned force analysis result includes the relative stress at the pixel position, and the stress direction of the relative stress.

[0120] After the force analysis unit 403, the device further includes:

[0121] A mapping relationship acquisition unit, used to acquire a mapping relationship between the relative stress and depth information of the contact module;

[0122] A target depth information determination unit, configured to determine target depth information recorded by the event camera based on the relative stress and the stress direction at each pixel position;

[0123] The stress information determining unit is used to determine the stress information on the contact module based on the target depth information and the mapping relationship.

[0124] Optionally, in the device provided in the embodiment of the present application, the contact module further includes a transparent film layer and a contact body, the contact body is used to contact the object, and the transparent film layer is arranged on the surface of the contact body inside the visual-tactile sensor.

[0125] The image acquisition module is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

[0126] The force analysis device provided in the embodiment of the present application is applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force situation. The visual tactile sensor also includes an image acquisition module. The image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera. The event information output by the event camera according to the deformation of the contact module can be obtained through the event information acquisition unit 401, wherein the event information includes multiple events and event data of the event, and the event data at least includes the pixel position of the triggering event; the displacement information acquisition unit 402 performs displacement analysis according to each event in the event information and the event data of the event to obtain the displacement information at each pixel position; the force analysis unit 403 performs force analysis on each pixel position based on the displacement information at the pixel position to obtain the force analysis result at the pixel position. Based on this, by setting the event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the event camera due to the deformation of the contact module, thereby timely and accurately performing stress recognition and feedback, and improving the user experience.

[0127] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal devices. Alternatively, the electronic device may be a server.

[0128] like Figure 5 As shown, Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0129] The processor 501 is the control center of the electronic device 500, and uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or loading software programs and / or units stored in the memory 502, and calling data stored in the memory 502. The processor 501 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0130] In the embodiment of the present application, the processor 501 in the electronic device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions, such as:

[0131] Acquire event information outputted by a deformation event camera according to the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data includes at least a pixel position of a triggering event;

[0132] Performing displacement analysis according to each event in the event information and the event data of the event to obtain displacement information at each pixel position;

[0133] For each pixel position, a force analysis is performed based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0134] The electronic device provided by the embodiment of the present application is applied to the processing module of the visual tactile sensor, the contact module of the visual tactile sensor contacts the object, the contact module is used to deform according to the force situation, the visual tactile sensor also includes an image acquisition module, the image acquisition module is used to shoot the contact module, the image acquisition module includes an event camera, and can obtain event information output by the event camera according to the deformation of the contact module, wherein the event information includes multiple events and event data of the event, and the event data at least includes the pixel position of the triggering event; displacement analysis is performed according to each event in the event information and the event data of the event, and displacement information at each pixel position is obtained; for each pixel position, force analysis is performed based on the displacement information at the pixel position, and the force analysis result at the pixel position is obtained. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the event camera due to the deformation of the contact module, thereby timely and accurately performing stress recognition and feedback, and improving the user experience.

[0135] Furthermore, various functions implemented by running the application stored in the memory 502 can also be found in the description of the aforementioned embodiments, which will not be repeated here.

[0136] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0137] Optional, such as Figure 5 As shown, the electronic device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 5 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0138] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0139] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0140] The audio circuit 505 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another electronic device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0141] The input unit 506 may be configured to receive input event signal sequences and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0142] The power supply 507 is used to supply power to various components of the electronic device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0143] although Figure 5 Not shown, the electronic device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0144] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0146] To this end, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform any of the force analysis methods provided in the embodiments of the present application. For example, the computer program can perform the following steps of the force analysis method:

[0147] Acquire event information outputted by a deformation event camera according to the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data includes at least a pixel position of a triggering event;

[0148] Performing displacement analysis according to each event in the event information and the event data of the event to obtain displacement information at each pixel position;

[0149] For each pixel position, a force analysis is performed based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

[0150] The computer-readable storage medium provided in the embodiment of the present application is used in a processing module of a visual tactile sensor, wherein the contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force conditions. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera, which can obtain event information output by the event camera according to the deformation of the contact module, wherein the event information includes multiple events and event data of the event, and the event data at least includes the pixel position of the triggering event; displacement analysis is performed according to each event in the event information and the event data of the event, and displacement information at each pixel position is obtained; for each pixel position, force analysis is performed based on the displacement information at the pixel position, and the force analysis result at the pixel position is obtained. Based on this, an event camera is set in the image acquisition module of the visual tactile sensor, so that the processing module of the visual tactile sensor can perform force analysis according to the event information output by the event camera due to the deformation of the contact module, thereby timely and accurately performing stress recognition and feedback, and improving the user experience.

[0151] Furthermore, for the detailed steps of the above method steps, reference can be made to the description in the above embodiments, which will not be repeated here.

[0152] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0153] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0154] Since the computer program stored in the computer-readable storage medium can execute any one of the force analysis methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the force analysis methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0155] According to one aspect of the present application, a computer program product is also provided, including a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the methods provided in various optional implementations of the above embodiments.

[0156] In the above-mentioned force analysis device, computer-readable storage medium, electronic device, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process and beneficial effects of the force analysis device, computer-readable storage medium, computer program product, electronic device and its corresponding units described above can refer to the description of the force analysis method in the above embodiment, and will not be repeated here.

[0157] The force analysis method, device, electronic device, computer-readable storage medium and computer program product provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A stress analysis method, characterized in that: The method is applied to a processing module of a visual-tactile sensor, wherein a contact module of the visual-tactile sensor contacts an object, the contact module is used to deform according to a force condition, the visual-tactile sensor further includes an image acquisition module, the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the method includes: Acquire event information output by the event camera according to the deformation of the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data at least includes a pixel position triggering the event; Performing displacement analysis according to each event in the event information and event data of the event to obtain displacement information at each pixel position; For each of the pixel positions, force analysis is performed based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

2. The stress analysis method according to claim 1, characterized in that: The event data also includes a timestamp of the event; performing displacement analysis according to each event in the event information and the event data of the event to obtain displacement information at each pixel position includes: Sorting each event in the event information according to the timestamp to obtain event information to be analyzed, wherein the event information to be analyzed includes each event sorted according to the timestamp; Determining optical flow information according to each event in the event data to be analyzed and the event data of the event; The displacement information at each pixel position is determined according to the optical flow information.

3. The stress analysis method according to claim 2, characterized in that: The determining of optical flow information according to each event in the event data to be analyzed and the event data of the event includes: According to the timestamps of the events in the event data to be analyzed, the events at the same timestamp are integrated to obtain a frame of event image at each timestamp; The difference in each pixel position between event images that are adjacent in time is calculated to obtain the optical flow information.

4. The stress analysis method according to claim 1, characterized in that: The event data also includes the polarity of the event, and the polarity includes a positive event and a negative event, wherein the positive event is used to indicate that the light intensity of the pixel point triggering the event becomes larger, and the negative event is used to indicate that the light intensity of the pixel point triggering the event becomes smaller; The performing displacement analysis according to each event in the event information and the event data of the event to obtain the displacement information at each pixel position includes: Displacement analysis is performed based on each event in the event information, the pixel position corresponding to the event, and the polarity of the event to obtain the relative displacement and displacement direction at each pixel position.

5. The stress analysis method according to claim 4, characterized in that: The step of performing force analysis on each pixel position based on the displacement information at the pixel position to obtain a force analysis result at the pixel position includes: For each of the pixel positions, a force analysis is performed based on the relative displacement at the pixel position by using a preset force recognition model to obtain a relative stress at the pixel position; Determining a stress direction of a relative stress at the pixel position according to a displacement direction at the pixel position; The relative stress at the pixel position and the stress direction of the relative stress are used as the force analysis result at the pixel position.

6. The stress analysis method according to claim 1, characterized in that: The image acquisition module further comprises a CIS camera, and a common field of view between the event camera and the CIS camera covers a surface of the contact module inside the visual-tactile sensor; Before acquiring the event information output by the event camera of the deformation of the contact module, the method further includes: If it is determined that the object does not slide, acquiring image data output by the CIS camera based on the deformation of the contact module; Based on the image data output by the CIS camera, a mapping relationship between the relative stress and the depth information of the contact module is determined.

7. The stress analysis method according to claim 6, characterized in that: The force analysis result includes the relative stress at the pixel position and the stress direction of the relative stress; after performing force analysis on each pixel position based on the displacement information at the pixel position to obtain the force analysis result at the pixel position, the method further includes: Acquire a mapping relationship between the relative stress and depth information of the contact module; Determining target depth information recorded by the event camera based on the relative stress and the stress direction at each of the pixel positions; Based on the target depth information and the mapping relationship, stress information on the contact module is determined.

8. The stress analysis method according to any one of claims 1 to 7, characterized in that: The contact module comprises a transparent film layer and a contact body, wherein the contact body is used to contact an object, and the transparent film layer is arranged on a surface of the contact body inside the visual tactile sensor; The image acquisition module is used to photograph the transparent film layer and output event information when the transparent film layer is deformed.

9. A force analysis device, characterized in that: The processing module of the visual tactile sensor is applied, wherein the contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera. The device includes: An event information acquisition unit, configured to acquire event information output by the event camera according to the deformation of the contact module, wherein the event information includes a plurality of events and event data of the events, and the event data includes at least a pixel position triggering the event; A displacement information acquisition unit, configured to perform displacement analysis according to each event in the event information and event data of the event to obtain displacement information at each pixel position; The force analysis unit is used to perform force analysis on each pixel position based on the displacement information at the pixel position to obtain a force analysis result at the pixel position.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the force analysis method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: It comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the force analysis method according to any one of claims 1 to 8.

Images