Stress analysis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a stress analysis method and device, electronic equipment and a storage medium. Event information output by a mark array with a deformed contact module can be shot through an event camera; determining deformation information of each mark point in the mark array according to the event information; and based on the deformation information of each mark point, carrying out stress analysis on the mark point to obtain a stress analysis result corresponding to the mark point. On the basis, the event camera is arranged in the image acquisition module of the visual tactile sensor, so that the processing module of the visual tactile sensor can perform stress analysis according to the event information output by the mark array which is shot by the event camera and is deformed by the contact module, thereby performing identification and feedback timely and accurately, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a force analysis method, device, electronic device and storage medium. Background Art

[0002] Existing visual tactile sensors use visual imaging technology and deep learning data processing methods to increase the measurement dimension of tactile information, that is, to sense and measure the position, distribution and direction of contact force. However, existing visual tactile sensors have certain limitations, which makes it impossible for existing visual tactile sensors to timely and accurately identify and feedback the force when the contact part is deformed by force. Summary of the invention

[0003] The embodiments of the present application provide a force analysis method, device, electronic device and storage medium. An event camera can be set in the image acquisition module of the visual-tactile sensor so that the processing module of the visual-tactile sensor can perform force analysis based on the event information output by the marking array of the deformation of the contact module captured by the event camera, thereby performing timely and accurate identification and feedback to improve the user experience.

[0004] In a first aspect, an embodiment of the present application provides a force analysis method, which is applied to a processing module of a visual tactile sensor, wherein a contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force conditions, and the contact module is provided with a marking layer, and the marking layer includes a marking array, and the visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the method includes:

[0005] The event information output by the deformation marker array of the contact module is photographed by the event camera;

[0006] Determining deformation information of each marking point in the marking array according to the event information;

[0007] Based on the deformation information of each of the marking points, a force analysis is performed on the marking point to obtain a force analysis result corresponding to the marking point.

[0008] In a second aspect, an embodiment of the present application further provides a force analysis device, which is applied to a processing module of a visual tactile sensor, wherein a contact module of the visual tactile sensor contacts an object, and the contact module is used to deform according to the force conditions, and the contact module is provided with a marking layer, and the marking layer includes a marking array, and the visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the device includes:

[0009] An event information acquisition unit, configured to capture, through the event camera, event information output by the deformation marker array of the contact module;

[0010] A deformation information determining unit, used to determine the deformation information of each marking point in the marking array according to the event information;

[0011] The force analysis unit is used to perform force analysis on the marking points based on the deformation information of each marking point to obtain the force analysis results corresponding to the marking points.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the force analysis methods provided in the embodiments of the present application.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute any step of the force analysis method provided in the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application also provides a computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs any step of the force analysis method provided in the embodiment of the present application.

[0015] The scheme of the embodiment of the present application is adopted and applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force conditions. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module. The image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera. The event information output by the marking array of the contact module deformation can be shot by the event camera; according to the event information, the deformation information of each marking point in the marking array is determined; based on the deformation information of each marking point, the marking point is subjected to force analysis to obtain the force analysis result corresponding to the marking point. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the marking array of the contact module deformation shot by the event camera, so as to timely and accurately identify and feedback, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of an application scenario of the visual-tactile sensor provided in an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of a system architecture for applying the force analysis method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of an embodiment of the force analysis method provided in the embodiments of the present application;

[0020] Figure 4 It is a structural schematic diagram of a frame of deformation event image provided to the force analysis method in an embodiment of the present application;

[0021] Figure 5 is a structural schematic diagram of a force analysis device provided in an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The following disclosure provides many different embodiments or examples to implement different features of the subject matter provided. Specific examples of components and arrangements are described below to make the disclosure concise. Of course, these are only examples and are not intended to be limiting. For example, in the following description, a first feature formed on or formed on a second feature may include an embodiment in which the first feature and the second feature are formed in direct contact, and may also include an embodiment in which an additional feature may be formed between the first feature and the second feature so that the first feature and the second feature may not be in direct contact. In addition, the disclosure may reuse reference numbers and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not inherently dictate the relationship between the various embodiments and / or configurations discussed.

[0025] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0026] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0029] It should be noted that the current common tactile sensor solutions are divided into piezoresistive, capacitive, photoelectric, electromagnetic and visual tactile. Among them, visual tactile tactile sensors are visual tactile sensors, which are the forefront and focus of tactile sensor development in recent years. The essence of visual tactile sensors is to use visual imaging technology and deep learning data processing methods to increase the measurement dimension of tactile information, that is, to perform perception measurements on the position, distribution and direction of contact force. Compared with the piezoresistive, capacitive, photoelectric, and electromagnetic tactile perception principles mentioned above, visual tactile sensors will have a more intuitive, clear, and independently decoupled image input, and then establish a mapping relationship between tactile information and image information through calibration training based on reinforcement learning to achieve higher-dimensional tactile measurement. In theory, visual tactile sensors can measure force information in various directions and different distribution positions, and the upper limit of measurement for characteristics such as tangential friction and slip is also higher.

[0030] However, the inventors have found that the existing visual tactile sensors have certain limitations, which are mainly reflected in the low frame rate of the image output by the image acquisition module. When the object in contact with the visual tactile sensor moves rapidly, motion blur will occur, and a clear image cannot be captured, resulting in the response speed and accuracy of the visual tactile sensor being limited by the frame rate. In addition, there is a large amount of redundant data in the image output, which reduces the efficiency of the back-end image processing, and ultimately causes the existing visual tactile sensor to be unable to timely and accurately feedback force information and displacement information when the object in contact with it slides rapidly.

[0031] Based on this, see Figure 1 The embodiment of the present application provides a visual tactile sensor. The visual tactile sensor is arranged on a clamping device. When the clamping device clamps an object, the visual tactile sensor contacts the object. The visual tactile sensor includes a contact module 11, an image acquisition module 13 and a processing module 14.

[0032] The contact module 11 is used to deform according to the force conditions.

[0033] The image acquisition module 13 is used to photograph the contact module 11 . The image acquisition module 13 includes an event camera. The event camera is used to output event information according to the deformation of the contact module 11 .

[0034] The processing module 14 is used to adjust the clamping force of the clamping device based on the received event information.

[0035] The contact module 11 is a module that is in direct contact with the object clamped by the clamping device. The specific shape of the contact module 11 can be adjusted according to actual conditions. For example, the specific shape of the contact module 11 is circular, square, or rectangular.

[0036] The contact module 11 includes a contact body and at least one marking layer. The contact body is used to contact the object clamped by the clamping device so that the contact body is deformed according to the force of the object clamped by the clamping device. The material of the contact body includes at least one of silicone or latex. A marking array is provided on each marking layer, and the marking layer is provided on the surface of the contact body inside the visual tactile sensor; the image acquisition module 13 is used to shoot the marking layer and output event information when the marking layer is deformed.

[0037] Among them, the role of the marking layer is to calibrate the deformation information of the sensed contact body through the displacement or deformation degree of the marker. The deformation information contains the attribute characteristics of the contact force. The displacement direction and degree of the marking point represent the distribution and magnitude of the contact force. The area of ​​the marking point is positively correlated with the normal force. It can be understood that by setting a marking layer containing marking points, the image acquisition module 13 can capture the deformation of the contact module 11 more clearly and accurately.

[0038] It should be noted that the single-layer marker array is dense and the single area is small, and the error of image processing is easily confused with the change of its area. Therefore, the contact module 11 can include multiple marker layers, each of which is provided with a marker array, thereby improving the force contrast.

[0039] It should be noted that the shape, distribution density and size of the markers are also closely related to the recognition accuracy. Each marker corresponds to a local area of ​​the contact surface. In theory, the more markers there are, the larger the effective recognition area is, and the accuracy and sensitivity are also improved accordingly. However, high-density markers also bring difficulties to image processing and the characterization of other contact features. Therefore, the embodiments of the present application preferably use a single-layer marker array.

[0040] The field of view of the image acquisition module 13 covers the surface of the contact module 11 inside the visual-tactile sensor. When the surface of the contact module 11 inside the visual-tactile sensor is deformed, the deformation of the contact module 11 is detected and corresponding event information is output.

[0041] It is understandable that the image acquisition module 13 in the traditional visual tactile sensor mostly adopts CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor, CIS). When shooting a motion scene through a CIS camera, due to the low output frame rate of the CIS camera, it is often impossible to capture the complete and clear motion information of the object, resulting in the subsequent image processing being unable to accurately analyze the deformation of the visual tactile sensor contact module, thereby causing the visual tactile sensor to be unable to respond quickly and in a timely manner. At the same time, when shooting through a CIS camera, each frame of the image collected by the CIS camera will capture the dynamic area and static area of ​​the object at the same time, and in the application of the visual tactile sensor, the static area in the field of view of the CIS camera belongs to a useless area, so the image captured by the CIS camera has more redundant information, which will cause the amount of data in the back-end image processing to increase and the efficiency to decrease, thereby further slowing down the response speed of the visual tactile sensor.

[0042] The image acquisition module in the visual tactile sensor provided in the embodiment of the present application adopts an event camera (EVS). The event camera is a new type of image sensor based on visual perception. Unlike the CIS camera, the event camera does not perceive the environment by capturing images, but perceives the environment by recording events of changes in light intensity. The working principle of the event camera is to detect changes in light intensity at the pixel level. When the light intensity increases or decreases (the change in light intensity exceeds the set threshold), the pixel triggers the event camera to shoot an event. Among them, the event includes a timestamp, position information and polarity, wherein the timestamp indicates the time when the event occurs, the position information indicates the pixel position where the event occurs, and the polarity indicates whether the light intensity becomes stronger or weaker, which can reflect the direction of movement. More importantly, the event camera only outputs event data in the dynamic area of ​​its field of view, and there is no event in the static area of ​​its field of view, and no event data will be output. That is, the event information in the embodiment of the present application only contains event data in the area where the contact module is deformed, and does not contain event data in other areas.

[0043] At the same time, since event cameras generally define a 2-bit data to represent the state of a pixel based on frame mode acquisition, the pixel states include 00, 01 and 10, where 00 indicates that the corresponding pixel state is no event, 01 indicates that the corresponding pixel state is a positive event (UP event), and 10 indicates that the corresponding pixel state is a negative event (DOWN event), while in traditional CIS cameras, each pixel provides an absolute value of light intensity. Therefore, compared with event cameras, traditional CIS cameras need to use more bits of data to represent each pixel state, such as 10 bits of data to represent a pixel state.

[0044] Therefore, the embodiment of the present application uses the event information output by the event camera for image processing, which greatly reduces redundant data and improves image processing efficiency, thereby making the visual tactile sensor have a faster response speed. In addition, since the event camera detects changes in light intensity, it can detect and output at a higher frame rate than the CIS camera, and can capture clearer and more complete motion images, that is, the visual tactile sensor can respond to changes in the contact module 11 faster and more accurately and output corresponding event information.

[0045] In some embodiments, the image acquisition module 13 includes multiple event cameras, and the field of view formed by the multiple event cameras covers the surface of the contact module 11 inside the visual-tactile sensor; or, the image acquisition module 13 includes at least one pair of image acquisition units, the image acquisition unit includes an event camera and a CIS camera, and the common field of view between the event camera and the CIS camera in the image acquisition unit covers the surface of the contact module 11 inside the visual-tactile sensor.

[0046] Specifically, the image acquisition module 13 may include multiple event cameras, that is, the visual tactile sensor is a binocular visual tactile sensor, or the visual tactile sensor is a multi-visual visual tactile sensor, wherein the fields of view of the multiple event cameras cover the surface of the contact module 11 inside the visual tactile sensor.

[0047] The image acquisition module 13 may also include a CIS camera, and the CIS camera and the event camera are arranged in binocular form. The common field of view of the CIS camera and the event camera covers the surface of the contact module 11 inside the visual tactile sensor.

[0048] The image acquisition module 13 may also include at least one event camera and at least one CIS camera. The event camera and the CIS camera may be arranged in pairs, each event camera corresponds to a CIS camera, and the common field of view between each event camera and its corresponding CIS camera can be summarized to completely cover the surface of the contact module 11 inside the visual tactile sensor.

[0049] Specifically, when the image acquisition module 13 is an image sensor that is a mixture of a CIS camera and an event camera, the mixed image sensor may be one or more, and the setting principle is to be able to completely cover the surface of the contact module 11 inside the visual-tactile sensor.

[0050] It should be noted that the image sensor that is a mixture of a CIS camera and an event camera is called a hybrid vision sensor (Hybrid Vision Sensor, HVS), which integrates a CIS camera and an event camera and supports the simultaneous output of CIS data and event data. It is understandable that, similar to the previous embodiment, the visual tactile sensor using HVS in this embodiment can also identify absolute stress, and calculate the stress under the depth information of the event data according to the depth information and the corresponding stress in the CIS data, so that the deformation information recorded by the event camera can establish a one-to-one mapping relationship with the actual stress magnitude. In addition, in this embodiment, by using HVS as the image acquisition module 13, unlike binocular or multi-visual visual tactile sensors, there is no need to additionally model the binocular multi-medium light refraction propagation error, which reduces the complexity of the design.

[0051] In some embodiments, the visual-tactile sensor further includes an illumination module 12, which is used to illuminate the contact module 11 in the visual-tactile sensor so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11.

[0052] The lighting module 12 is used to provide uniform lighting in the visual-tactile sensor, so that when the contact module 11 is deformed, the image acquisition module 13 captures the contact module 11 and outputs event information according to the deformation of the contact module 11 .

[0053] Furthermore, it is understandable that if the clamping device is flexible enough to adjust the magnitude of the force at different parts at the same time, then the embodiment of the present application can also determine which part of the force of the clamping device needs to be increased according to the location where the event occurs. For example, the moving direction of the object can be determined according to the change of the event, and then the force at the corresponding position can be adjusted to achieve stable grasping.

[0054] In summary, a visual tactile sensor is proposed in an embodiment of the present application, wherein the visual tactile sensor includes a contact module and an image acquisition module, the contact module contacts the object, a marking layer is provided in the contact module, and a marking array is provided on the marking layer. In particular, the present application sets an event camera in the image acquisition module, and when the event information output by the event camera is received, the contour information of each mark in the marking array is obtained according to the event information, and the deformation degree of each mark is analyzed according to the contour information, and then the force magnitude and force direction of each mark are calculated according to the deformation degree and the preset force model. The event information output by the event camera in the image acquisition module of the present application has a high frame rate, and does not contain redundant information related to the static area, so that the present application can more clearly and completely restore the specific deformation of the relevant mark when the marking layer is deformed, and the data volume of the event information is small, and the efficiency of image processing is also improved, so that the visual tactile sensor has a faster response speed, and can more timely and accurately feedback the force information when the object it contacts slides quickly.

[0055] To solve the above problems, see Figure 2 , embodiments of the present application provide a force analysis method, device, electronic device, and computer-readable storage medium.

[0056] The stress analysis method provided in the embodiment of the present application can be applied to the system architecture 200. Figure 2 ,like Figure 2 As shown, the system architecture 200 may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] Users can use terminal devices 201, 202, 203 to interact with server 205 through network 204 to receive or send messages, etc.

[0058] Terminal devices 201, 202, 203 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Eperts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. The terminal may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. The terminal and the server may perform two-way communication via a network.

[0059] The server 205 may be a server that provides various services, such as a backend server that supports the pages displayed on the terminal devices 201, 202, and 203. The server may be an independent server, or a server network or server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0060] It should be noted that the force analysis method provided in the embodiment of the present application is executed by a server, and accordingly, the force analysis device is arranged in the server.

[0061] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 201, 202, 203 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0062] The following is a detailed description with reference to the accompanying drawings. In this embodiment, the execution subject is an electronic device that can obtain the target video. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0063] It should be noted that the force analysis method is applied to the processing module of the visual tactile sensor, and the visual tactile sensor can be set on a clamping device or other intelligent mechanical equipment. When the clamping device clamps the object, or the intelligent mechanical device grabs the object, the contact module of the visual tactile sensor contacts the object, and the contact module is used to deform according to the force conditions. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module, and the image acquisition module includes an event camera.

[0064] Among them, the visual-tactile sensor can refer to the relevant description of the visual-tactile sensor in the above embodiment, and will not be described in detail in this embodiment of the application.

[0065] Please refer to Figure 3 The specific process of the force analysis method can be as follows: Step 301 to Step 303, wherein:

[0066] Step 301: Use an event camera to capture event information output by a deformation marker array of a contact module.

[0067] Specifically, the event information output by the event camera when photographing the deformation of the marker array of the contact module during the movement process of the clamping device clamping the object or the process of the intelligent mechanical device grasping the object can be obtained.

[0068] Among them, the relevant contents of the contact module and the marking array can refer to the relevant description of the visual tactile sensor in the above embodiment, and will not be repeated in the embodiment of the present application.

[0069] Step 302: Determine the deformation information of each marking point in the marking array according to the event information.

[0070] Among them, the event information includes several events and event data for each event. The specific content of the event data can be adjusted according to the actual situation, and the embodiments of the present application are not limited. For example, the event data includes but is not limited to the pixel position where the light intensity of the event changes, the timestamp of the event, and the polarity of the event. It should be noted that the event data also includes events corresponding to pixels whose light intensity changes compared to the previous frame, which are measured by polarity. Polarity includes positive events and negative events. Positive events are used to indicate that the light intensity of the pixel point that triggered the event becomes larger, and negative events are used to indicate that the light intensity of the pixel point that triggered the event becomes smaller.

[0071] Among them, deformation information refers to relevant information used to measure the degree of deformation of the marking point. The deformation information includes a deformation degree index value and a deformation direction. Among them, the deformation degree can be quantified by changes in the length, angle, area, etc. of the marking point, thereby obtaining a deformation degree index value. It can be understood that there is a certain mapping relationship between the degree of deformation and the force condition of the marking point. The deformation direction refers to the relative direction of the marking point when deformation occurs, which can be adjusted according to actual conditions, and the embodiments of the present application are not limited. For example, the deformation direction is the direction of the marking point relative to the positive direction (from left to right) of the contact module.

[0072] In some optional embodiments, the process of determining the deformation information of each marking point in the marking array according to the event information may include: according to the timestamp of each event in the event information, each event under the same timestamp is combined into a frame of deformation event image to obtain several frames of deformation event images, and the deformation event image is used to indicate the event image constructed by the event in which the contact module is deformed; performing feature extraction on the marking points contained in each deformation event image, and determining the first contour information of the marking points in the corresponding deformation event image according to the obtained features; and determining the deformation information of each marking point according to the first contour information of each marking point.

[0073] The deformation event image is the event image constructed by the events at the same timestamp when the contact module is deformed. For example, see Figure 4 , Figure 4 It is a schematic diagram of a frame of deformation event image provided by an embodiment of the present application.

[0074] The first contour information refers to the contour information of the mark point after deformation in the corresponding deformation event image.

[0075] There are many ways to extract the above features. For example, image processing algorithms such as edge detection and morphological processing are used to extract features of the marker points contained in each deformation event image to obtain the first contour information of each marker point.

[0076] In some optional embodiments, the force analysis method further includes: obtaining second contour information when each marking point in the marking array is in an undeformed state.

[0077] The second contour information refers to the contour information when the marking point is in an undeformed state.

[0078] There are many ways to obtain the second profile information, which can be adjusted according to actual conditions and are not limited in the embodiments of the present application.

[0079] In some optional embodiments, the second contour information may be directly preset, or a static image may be captured by a CIS camera and the second contour information may be determined based on the static image.

[0080] In some optional embodiments, initial contour information of each marking point in the marking array when the contact module is not deformed may be obtained as the second contour information.

[0081] In some optional embodiments, target contour information determined by a deformation event image before a timestamp based on the target deformation event image can be obtained by acquiring each marking point in the marking array as second contour information, wherein the target contour information is determined by extracting features from the marking points contained in the deformation event image before the timestamp of the target deformation event image and based on the obtained features.

[0082] The target deformation event image refers to the deformation event image corresponding to the current frame moment.

[0083] Exemplarily, assuming that there are 10 frames of deformation event images, the 10th frame of deformation event image is the target deformation event image, the second contour information may be the contour information corresponding to the 9th frame of deformation event image, the second contour information may also be the contour information determined based on the deformation information of the 1st to 9th frames of deformation event images, or the second contour information may be the contour information determined by the deformation information of any number of deformation event images selected from the 1st to 9th frames of deformation event images.

[0084] Based on this, in this embodiment, the process of determining the deformation information of each marking point according to the first contour information of each marking point may include: determining the deformation information of the marking point based on the first contour information and the second contour information of each marking point.

[0085] Specifically, based on the first contour information of each marking point, the first feature of the marking point in the target dimension is calculated; based on the second contour information of the marking point, the second feature of the marking point in the target dimension is calculated; based on the first feature and the second feature of the marking point, the feature difference of the marking point in the target dimension is determined, and the deformation information of the marking point is determined.

[0086] The target dimension includes at least one dimension of key point position, area, angle, and shape descriptor.

[0087] For example, when the target dimension is the key point position, for each marker point, based on the contour information, the key points (such as extreme points, inflection points, etc.) on the contour of the marker point can be identified, and the position changes of these points before and after deformation can be compared, and the distance between the key points before and after deformation can be used as its deformation information.

[0088] For another example, when the target dimension is area, the area enclosed by the contour before and after deformation of each marking point can be calculated, and the area change rate or change value can be calculated, and the area change rate or change value can be used as its deformation information.

[0089] For another example, when the target dimension is an angle, the curvature of the principal axis before and after the deformation can be calculated for each marking point, and the curvature of the principal axis can be used as its deformation information.

[0090] For another example, when the target dimension is a shape descriptor, a shape descriptor can be used to quantify the shape change of the contour for each marker point. For example, if the shape of the marker is a circle, a circular descriptor can be used to compare the circularity of the contour before and after the marker is deformed, calculate the change rate or change value of the circularity, and use the change rate or change value of the circularity as the degree of deformation. It is understandable that if the shape of the marker is a rectangle, a rectangular descriptor can be used to complete the above process, and the same applies to other shapes.

[0091] Step 303: Based on the deformation information of each marking point, a force analysis is performed on the marking point to obtain a force analysis result corresponding to the marking point.

[0092] There are many ways to analyze the force, which can be adjusted according to the actual situation and are not limited in the embodiments of the present application.

[0093] For example, when the deformation information includes a deformation degree index value and a deformation direction. Based on the deformation information of each marked point, a force analysis is performed on the marked point to obtain the force analysis result corresponding to the marked point. The process may include: for each marked point, through a preset force model, a force analysis is performed based on the deformation degree index value of each marked point to obtain the force value corresponding to the marked point; for each marked point, based on the deformation direction of the marked point, the force direction of the marked point is determined; and the force value and the force direction are used as the force analysis result corresponding to the marked point.

[0094] Among them, the above-mentioned preset force identification model can be a force identification model constructed based on elastic mechanics, or it can be a force identification model trained based on a machine learning method.

[0095] Exemplarily, the preset force recognition model can be trained through the following steps: simulate the actual application scenario, use the visual tactile sensor to measure the deformation data of the mark under different forces, preprocess the above deformation data, including denoising, data cleaning, data normalization, etc., to improve the quality and reliability of the data; extract key features from the preprocessed data, such as the maximum value of the deformation, the rate of change, the degree of recovery, etc., these features will be used for subsequent model training; according to the nature of the problem and the characteristics of the data, select a suitable mathematical model to describe the relationship between force and deformation, and use experimental data to train the selected model. The training process involves the optimization of model parameters so that the model can predict the force value corresponding to a given deformation as accurately as possible. Among them, common mathematical models include linear models, nonlinear models, neural network models, etc.; the model is verified and adjusted, and finally the force model is established. The scheme of the embodiment of the present application is adopted and applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force conditions. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module. The image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera. The event information output by the marking array of the contact module deformation can be shot by the event camera; according to the event information, the deformation information of each marking point in the marking array is determined; based on the deformation information of each marking point, the marking point is subjected to force analysis to obtain the force analysis result corresponding to the marking point. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the marking array of the contact module deformation shot by the event camera, so as to timely and accurately identify and feedback, and improve the user experience.

[0096] This embodiment further provides a force analysis device, which can be integrated into an electronic device, such as a computer device. The computer device can be a terminal, a server, or other device, and this embodiment has no limitation on this.

[0097] For example, Figure 5 As shown, the force analysis device is applied to the processing module of the visual tactile sensor, the contact module of the visual tactile sensor contacts the object, the contact module is used to deform according to the force condition, the contact module is provided with a marking layer, the marking layer includes a marking array, the visual tactile sensor also includes an image acquisition module, the image acquisition module is used to shoot the contact module, the image acquisition module includes an event camera, and the force analysis device may include:

[0098] An event information acquisition unit 401 is used to capture the event information output by the deformation marker array of the contact module through an event camera;

[0099] A deformation information determining unit 402, configured to determine deformation information of each marking point in the marking array according to the event information;

[0100] The force analysis unit 403 is used to perform force analysis on the marked points based on the deformation information of each marked point to obtain the force analysis results corresponding to the marked points.

[0101] Optionally, in the device of an embodiment of the present application, the above-mentioned event information includes several events and event data of each event, and the event data includes a timestamp of the event.

[0102] In this embodiment, the deformation information determining unit 402 includes:

[0103] The deformation event image determination subunit is used to combine the events with the same timestamp into a frame of deformation event image according to the timestamp of each event in the event information, and obtain a plurality of frames of deformation event images, wherein the deformation event image is used to indicate the event image constructed by the event that the contact module is deformed;

[0104] A first contour information determination subunit is used to extract features of the marker points contained in each deformation event image, and determine first contour information of the marker points in the corresponding deformation event image according to the obtained features;

[0105] The deformation information determining subunit is used to determine the deformation information of each marking point according to the first contour information of each marking point.

[0106] Optionally, in the device of the embodiment of the present application, the force analysis device may further include:

[0107] A second contour information determining unit, used to obtain second contour information when each marking point in the marking array is in an undeformed state;

[0108] Based on this, the above deformation information determines the sub-unit, which specifically includes:

[0109] Based on the first contour information and the second contour information of each marking point, deformation information of the marking point is determined.

[0110] Optionally, in the device of the embodiment of the present application, the second profile information determining unit includes:

[0111] A second contour information first determining subunit is used to obtain initial contour information of each marking point in the marking array when the contact module is not deformed as the second contour information;

[0112] The second determination subunit of the second contour information is used for, or to obtain, for each marking point in the marking array, the target contour information determined by the deformation event image before the timestamp of the target deformation event image as the second contour information, wherein the target contour information is determined by extracting features from the marking points contained in the deformation event image before the timestamp of the target deformation event image and based on the obtained features.

[0113] Optionally, in the device of the embodiment of the present application, the determining the deformation information of the marking point based on the first contour information and the second contour information of each marking point specifically includes:

[0114] Based on the first contour information of each marking point, calculating the first feature of the marking point in the target dimension;

[0115] Based on the second contour information of the marking point, calculating the second feature of the marking point in the target dimension;

[0116] Based on the first feature and the second feature of the marking point, a feature difference of the marking point in the target dimension is determined, and deformation information of the marking point is determined.

[0117] In this embodiment, the target dimension includes at least one dimension of key point position, area, angle, and shape descriptor.

[0118] Optionally, in the device of the embodiment of the present application, the above-mentioned deformation information includes a deformation degree index value and a deformation direction.

[0119] In this embodiment, the force analysis unit 403 includes:

[0120] The force value determination subunit is used to perform force analysis on each marking point based on the deformation index value of each marking point through a preset force model to obtain the force value corresponding to the marking point;

[0121] A force direction determination subunit, used for determining the force direction of each marking point based on the deformation direction of the marking point;

[0122] The force analysis subunit is used to use the force value and force direction as the force analysis result corresponding to the marked point.

[0123] The force analysis device provided in the embodiment of the present application is applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force condition. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera, and the event information acquisition unit 401 can shoot the event information output by the marking array of the contact module through the event camera; the deformation information determination unit 402 determines the deformation information of each marking point in the marking array according to the event information; the force analysis unit 403 performs force analysis on the marking point based on the deformation information of each marking point, and obtains the force analysis result corresponding to the marking point. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the marking array of the contact module that is deformed by the event camera, so as to timely and accurately identify and feedback, thereby improving the user experience.

[0124] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal devices. Alternatively, the electronic device may be a server.

[0125] like Figure 6 As shown, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0126] The processor 501 is the control center of the electronic device 500, and uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or loading software programs and / or units stored in the memory 502, and calling data stored in the memory 502. The processor 501 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0127] In the embodiment of the present application, the processor 501 in the electronic device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions, such as:

[0128] The event information output by the deformation marker array of the contact module is photographed by an event camera;

[0129] Determine the deformation information of each marking point in the marking array according to the event information;

[0130] Based on the deformation information of each marking point, a force analysis is performed on the marking point to obtain the force analysis result corresponding to the marking point.

[0131] The electronic device provided by the embodiment of the present application is applied to the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module, and the image acquisition module is used to shoot the contact module. The image acquisition module includes an event camera, and the event information output by the marking array of the contact module being deformed can be shot by the event camera; according to the event information, the deformation information of each marking point in the marking array is determined; based on the deformation information of each marking point, the marking point is subjected to force analysis to obtain the force analysis result corresponding to the marking point. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the marking array of the contact module being deformed shot by the event camera, so as to timely and accurately identify and feedback, thereby improving the user experience.

[0132] Furthermore, various functions implemented by running the application stored in the memory 502 can also be found in the description of the aforementioned embodiments, which will not be repeated here.

[0133] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0134] Optional, such as Figure 6 As shown, the electronic device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 6 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0135] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0136] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0137] The audio circuit 505 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another electronic device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0138] The input unit 506 may be configured to receive input event signal sequences and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0139] The power supply 507 is used to supply power to various components of the electronic device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0140] although Figure 6 Not shown, the electronic device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0142] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0143] To this end, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform any of the force analysis methods provided in the embodiments of the present application. For example, the computer program can perform the following steps of the force analysis method:

[0144] The event information output by the deformation marker array of the contact module is photographed by an event camera;

[0145] Determine the deformation information of each marking point in the marking array according to the event information;

[0146] Based on the deformation information of each marking point, a force analysis is performed on the marking point to obtain the force analysis result corresponding to the marking point.

[0147] The computer-readable storage medium provided in the embodiment of the present application is used in the processing module of the visual tactile sensor. The contact module of the visual tactile sensor contacts the object. The contact module is used to deform according to the force. The contact module is provided with a marking layer, and the marking layer includes a marking array. The visual tactile sensor also includes an image acquisition module, which is used to shoot the contact module. The image acquisition module includes an event camera, and the event information output by the marking array of the contact module being deformed can be shot by the event camera; according to the event information, the deformation information of each marking point in the marking array is determined; based on the deformation information of each marking point, the marking point is subjected to force analysis to obtain the force analysis result corresponding to the marking point. Based on this, by setting an event camera in the image acquisition module of the visual tactile sensor, the processing module of the visual tactile sensor can perform force analysis according to the event information output by the marking array of the contact module being deformed shot by the event camera, so as to timely and accurately identify and feedback, thereby improving the user experience.

[0148] Furthermore, for the detailed steps of the above method steps, reference can be made to the description in the above embodiments, which will not be repeated here.

[0149] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0150] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] Since the computer program stored in the computer-readable storage medium can execute any one of the force analysis methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the force analysis methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0152] According to one aspect of the present application, a computer program product is also provided, including a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the methods provided in various optional implementations of the above embodiments.

[0153] In the above-mentioned force analysis device, computer-readable storage medium, electronic device, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process and beneficial effects of the force analysis device, computer-readable storage medium, computer program product, electronic device and its corresponding units described above can refer to the description of the force analysis method in the above embodiment, and will not be repeated here.

[0154] The force analysis method, device, electronic device, computer-readable storage medium and computer program product provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A stress analysis method, characterized in that: The method is applied to a processing module of a visual tactile sensor, wherein a contact module of the visual tactile sensor contacts an object, the contact module is used to deform according to a force condition, the contact module is provided with a marking layer, the marking layer includes a marking array, the visual tactile sensor further includes an image acquisition module, the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the method includes: The event information output by the deformation marker array of the contact module is photographed by the event camera; Determining deformation information of each marking point in the marking array according to the event information; Based on the deformation information of each of the marking points, a force analysis is performed on the marking point to obtain a force analysis result corresponding to the marking point.

2. The stress analysis method according to claim 1, characterized in that: The event information includes a plurality of events and event data of each of the events, wherein the event data includes a timestamp of the event; Determining the deformation information of each marking point in the marking array according to the event information includes: According to the timestamp of each event in the event information, each event with the same timestamp is combined into a frame of deformation event image to obtain a plurality of frames of deformation event images, wherein the deformation event image is used to indicate an event image constructed by an event in which the contact module is deformed; Extracting features of the marking points contained in each of the deformation event images, and determining first contour information of the marking points in the corresponding deformation event image according to the obtained features; Deformation information of each of the marking points is determined according to the first contour information of each of the marking points.

3. The stress analysis method according to claim 2, characterized in that: The method further comprises: Acquire second contour information when each marking point in the marking array is in an undeformed state; The step of determining the deformation information of each of the marking points according to the first contour information of each of the marking points comprises: Based on the first contour information and the second contour information of each of the marking points, deformation information of the marking point is determined.

4. The stress analysis method according to claim 3, characterized in that: The obtaining of second contour information when each marking point in the marking array is in an undeformed state includes: Acquire initial contour information of each marking point in the marking array when the contact module is not deformed as second contour information; Alternatively, for each marking point in the marking array, target contour information determined by a deformation event image before the timestamp of the target deformation event image is obtained as second contour information, wherein the target contour information is determined by extracting features from the marking points contained in the deformation event image before the timestamp of the target deformation event image and based on the obtained features.

5. The stress analysis method according to claim 3, characterized in that: The determining the deformation information of the marking point based on the first contour information and the second contour information of each marking point includes: Based on the first contour information of each of the marking points, calculating the first feature of the marking point in the target dimension; Based on the second contour information of the marking point, calculating a second feature of the marking point in the target dimension; Based on the first feature and the second feature of the marking point, a feature difference of the marking point in the target dimension is determined, and deformation information of the marking point is determined.

6. The stress analysis method according to claim 5, characterized in that: The target dimension includes at least one dimension of key point position, area, angle, and shape descriptor.

7. The stress analysis method according to any one of claims 1 to 6, characterized in that: The deformation information includes a deformation degree index value and a deformation direction; The step of performing force analysis on the marking points based on the deformation information of the marking points to obtain force analysis results corresponding to the marking points includes: For each marking point, a force analysis is performed based on the deformation index value of each marking point by using a preset force model to obtain a force value corresponding to the marking point; For each of the marking points, determining the force direction of the marking point based on the deformation direction of the marking point; The force value and the force direction are used as the force analysis result corresponding to the marked point.

8. A force analysis device, characterized in that: The processing module of the visual tactile sensor is applied, wherein the contact module of the visual tactile sensor contacts an object, the contact module is used to deform according to the force, the contact module is provided with a marking layer, the marking layer includes a marking array, the visual tactile sensor also includes an image acquisition module, the image acquisition module is used to photograph the contact module, and the image acquisition module includes an event camera; the device includes: An event information acquisition unit, configured to capture, through the event camera, event information output by the deformation marker array of the contact module; A deformation information determining unit, used to determine the deformation information of each marking point in the marking array according to the event information; The force analysis unit is used to perform force analysis on the marking points based on the deformation information of each marking point to obtain the force analysis results corresponding to the marking points.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the force analysis method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: It comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the force analysis method according to any one of claims 1 to 7.

Images