Apparatus and system for determining forces applied on a deformable material

Abstract

An apparatus includes: a body made of a deformable material; and a plurality of protrusions movably attached to an inner surface of the main body, each of the plurality of protrusions extending in a direction extending from the inner surface toward a center of the main body, and configured to be movable in the direction in response to deformation of the main body.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 507,737, filed on June 13, 2023, entitled “Artificial Skin-based Phygital System for the Cohesion of Human, Robot, and Virtual World,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to an apparatus and system for determining a force exerted on a deformable material. Background Art

[0004] Hands are the primary organs for human touch and tool manipulation because they are the most direct extension of human consciousness. Grasping is also one of the most common human behaviors, which are behaviors that people naturally perform when using hand tools and manipulating objects in the real world. Nowadays, people mostly communicate with the virtual world through joysticks, mice, keyboards, trackpads, etc. With the development of information technology, the industry has launched a variety of VR products that integrate hand manipulation functions. These products usually collect hand manipulation information by using, for example, one or more game handle controllers.

[0005] However, as the need for large amounts of and accurate data collection continues to increase, it is believed that gamepad controllers are no longer an ideal input for VR. For example, it is not possible to use a gamepad controller to determine the forces applied to the controller, such as by shaking hands, and the distribution of the forces. Even for certain devices on the market that attempt to do this by capturing visual data within the device on which the forces are applied, not only are they unable to accurately determine the forces applied to the device and the distribution of the forces, but more than one camera is usually required to capture such data.

[0006] It is expected that new methods, equipment, and systems will help advance the technical needs and industry applications in this field. Summary of the invention

[0007] A device includes: a body made of a deformable material; and a plurality of protrusions movably attached to an inner surface of the body, each of the plurality of protrusions extending in a direction from the inner surface toward a center of the body and configured to be movable in the direction in response to deformation of the body.

[0008] Other embodiments will also be described herein.

[0009] Other benefits and advantages of the disclosed embodiments will be apparent from the description and drawings. These benefits and / or advantages can be obtained individually by various embodiments and features in the description and drawings, and all embodiments and features do not have to be provided in order to obtain one or more of these benefits and / or advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to illustrate various embodiments by way of non-limiting examples only and to explain various principles and advantages according to the current embodiments. In the accompanying drawings, like reference numerals refer to the same or functionally similar elements throughout the individual views, and the accompanying drawings together with the detailed description below are incorporated into and constitute a part of this specification.

[0011] Embodiments of the present disclosure will be better understood and will become apparent to one of ordinary skill in the art through the following written description, which is given by way of example only, in conjunction with the accompanying drawings, in which:

[0012] Figure 1 An exemplary diagram of an apparatus for determining a force exerted on a deformable material is shown, in accordance with certain embodiments of the present disclosure.

[0013] Figure 2 An exemplary diagram showing how displacement of a marker may be detected according to certain embodiments of the present disclosure.

[0014] Figure 3 An exemplary process of how to determine forces and distribution of forces according to certain embodiments of the present disclosure is shown.

[0015] Figure 4 An exemplary diagram of a cylindrical housing of an apparatus for applying finite element mesh analysis is shown in accordance with certain embodiments of the present disclosure.

[0016] Figure 5 An exemplary diagram showing how a finite element mesh may be applied to a device for determining forces exerted on a deformable material in accordance with certain embodiments of the present disclosure.

[0017] Figure 6 Other exemplary diagrams show how a finite element mesh may be applied to a device for determining forces exerted on a deformable material in accordance with certain embodiments of the present disclosure.

[0018] Figure 7 An exemplary diagram illustrating how to obtain training data for use in determining forces exerted on a deformable material in accordance with certain embodiments of the present disclosure.

[0019] Figure 8 Exemplary captured images from an apparatus for use in determining forces exerted on a deformable material are shown in accordance with certain embodiments of the present disclosure.

[0020] Fig. 9 An exemplary three-dimensional (3D) reconstruction of a cylindrical housing of a device according to certain embodiments of the present disclosure is shown.

[0021] Fig.10 An exemplary illustration of an unfolded deformation matrix obtained from a finite mesh analysis of a cylindrical shell is shown in accordance with certain embodiments of the present disclosure.

[0022] Fig.11 An exemplary diagram illustrating a reconstruction of a cylindrical housing according to certain embodiments of the present disclosure is shown.

[0023] Fig.12 An exemplary diagram showing how an apparatus and system for determining a force exerted on a deformable material may be used in accordance with certain embodiments of the present disclosure.

[0024] Fig.13 A schematic diagram illustrating an exemplary computing device suitable for determining a force exerted on a deformable material, according to certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described by way of example only with reference to the accompanying drawings. Like reference numerals and characters in the drawings refer to like elements or equivalents.

[0026] Some portions of the following description are presented explicitly or implicitly in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally considered here to be a self-consistent sequence of steps that leads to a desired result. These steps are steps that require physical manipulation of physical quantities, such as electrical, magnetic or optical signals that can be stored, transmitted, combined, compared and otherwise manipulated.

[0027] Unless specifically stated otherwise, and as will become apparent from the following text, it will be understood that throughout this specification, discussions using terms such as "detect," "estimate," "compare," "receive," "calculate," "determine," "update," "generate," "initialize," "output," "receive," "retrieve," "identify," "distribute," "verify," etc., refer to the actions and processes of a computer system or similar electronic device that manipulate and transform data represented as physical quantities within a computer system into other data similarly represented as physical quantities within a computer system or other information storage, transmission, or display device.

[0028] This specification also discloses a device for performing the operations of the method. Such a device may be specially constructed for the desired purpose, or may include a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various machines may be used with the programs taught herein. Alternatively, more specialized devices may also be constructed to perform the desired method steps. The structure of the computer will become apparent from the following description.

[0029] In addition, this specification also implicitly discloses a computer program, because it is obvious to those skilled in the art that the multiple steps of the method described herein can be implemented by computer code. The computer program is not intended to be limited to any specific programming language and its implementation. It will be understood that various programming languages ​​and their encodings can be used to implement the disclosed teachings contained in this article. In addition, the computer program is not intended to be limited to any specific control flow. The computer program also has many other variations, which can use different control flows without departing from the spirit or scope of the present disclosure.

[0030] In addition, one or more steps of the computer program may be executed in parallel, rather than sequentially. Such a computer program may be stored on any computer readable medium. Computer readable media may include storage devices, such as magnetic or optical disks, memory chips or other storage devices suitable for interfacing with a computer. Computer readable media may also include: hard-line media, such as those exemplified in the Internet system; or wireless media, such as those exemplified in the GSM mobile phone system. The computer program, when loaded and executed on such a computer, may effectively produce a device that implements the steps of the preferred method.

[0031] Exemplary Embodiments

[0032] Various embodiments of the present disclosure relate to apparatus and systems for determining forces exerted on a deformable material.

[0033] Definition of terms

[0034] In the present disclosure, a device is disclosed, which includes a body having a three-dimensional shape, the body consisting of two bases, the two bases being joined by a surface around the body, for example, the surface joins the edge around one base with the edge around the other base. The shape of the two bases can be circular, so that the surface is a curved surface and the body can be a cylindrical body or a body with other suitable shapes. In one embodiment, the shape of the two bases can be square, hexagonal, octagonal and other similar shapes, and the two bases can have curved corners. Similarly, although the cylindrical body is mainly referred to in the present disclosure, it will be understood that the body can also be square, hexagonal, octagonal or other similar shapes with curved corners, and the surface can also be constructed accordingly. The centers of the bases overlap each other to form an axis or straight line passing through the center of the body. The curved surface can be straight from one base to the other along the length of the cylindrical body. In one embodiment, the curved surface can be ergonomically shaped, for example, to conform to the shape of a hand holding the cylindrical body or other similar shapes. In the present disclosure, the inner surface of the cylindrical body refers to a curved surface arranged inside the cylindrical body and facing the center of the cylindrical body. Similarly, the outer surface of the cylindrical body refers to a curved surface arranged outside the cylindrical body. In the present disclosure, the cylindrical body has a first end and a second end (e.g., each of the two bases mentioned above).

[0035] A deformable material refers to a material that can change shape due to, for example, a force acting on the material. Such a material can be elastic so that after the force acting on the material is removed, the material can return to its original shape. The material can be artificial skin, rubber, silicone or other similar materials. In one embodiment, the cylindrical body of the present disclosure is made of a deformable material. In another embodiment, a portion of the cylindrical body (e.g., the outer surface and / or the inner surface) is made of a deformable material.

[0036] A projection refers to an object that bulges out or protrudes from its surroundings. In the present disclosure, a plurality of projections are arranged on the inner surface of a cylindrical body, and each projection extends in a direction from the inner surface toward the center of the cylindrical body. A plurality of projections can be movably attached to the inner surface, and configured to be able to move in this direction in response to the deformation of the cylindrical body (e.g., when a force is applied to the outer surface of the cylindrical body). Each projection can also have a color at the end (e.g., the end of the projection that moves toward the center of the cylindrical body or away from the center of the cylindrical body due to the force applied to the cylindrical body), and the color is different from the color of the inner surface, which can advantageously improve the accuracy of monitoring the displacement of a plurality of projections. The colors of a plurality of projections can also be different from each other. The projection can be straight and can be made of a deformable material. In one embodiment, the projection can be made of a material different from the cylindrical body. In one embodiment, the length of each of the plurality of protrusions is proportional to the distance of each of the plurality of protrusions from the first end of the cylindrical body, which advantageously enables a single visual sensor (eg, disposed at the first end) to monitor the displacement of the plurality of protrusions.

[0037] A dome refers to a generally hemispherical structure that may include a base that is circular, square, triangular, or other similar shapes. In the present disclosure, the cylindrical body may include a plurality of holes arranged at the outer surface of the cylindrical body, each of the plurality of holes extending through each of the plurality of protrusions. In addition, a plurality of domes may be arranged on the outer surface of the cylindrical body, wherein each of the plurality of domes covers each of the plurality of holes. The plurality of domes may be made of a material that can change shape due to, for example, a force acting on the dome, and such material may be elastic so that it can return to its original shape after the force acting on the dome is removed. Therefore, the dome may be made of a deformable material or may be made of a different material but having the above-mentioned properties.

[0038] In the present disclosure, a marker refers to a thin, hair-like structure (e.g., thinner than a protrusion so that it can pass through a hole) that is attached to a dome at one end and extends from the dome to a hole covered by the dome. The hole extends through the protrusion, and the marker is configured to move along the hole in response to a deformation of the dome (e.g., due to a force acting on the dome). Thus, there are multiple holes (one hole passing through each protrusion) and multiple markers in the cylindrical body. Each marker may also have a color at the other end (e.g., the end of the marker that moves into or out of the hole due to a force acting on the dome) that is different from both the color of the inner surface and the color of the corresponding protrusion, which advantageously can improve the accuracy of monitoring the displacement of multiple markers. The colors of multiple markers may also be different from each other. Each marker may be straight and may be made of a deformable material or a material different from the dome. In one embodiment, the length of each of the multiple marks is proportional to the distance of each of the multiple protrusions from the first end of the cylindrical body, which advantageously enables a single visual sensor (e.g., a visual sensor arranged at the first end) to monitor the displacement of multiple marks.

[0039] A visual sensor (e.g., such as a thermal imager, an infrared camera, a red-green-blue (RGB) camera, a high-definition (HD) camera, and other similar visual sensors) can be arranged at one end (e.g., a first end) of the cylindrical body and configured to monitor any displacement of the plurality of protrusions and / or markers. Sensor data (e.g., images, videos, and other similar data) obtained from the visual sensor can be used by the system to determine the length of the displacement of each of the plurality of protrusions captured by the visual sensor, and determine the force applied to the cylindrical body based on a comparison of the length of the displacement with historical data. The historical data can be training data obtained through a controlled setting, where a force is applied at one or more locations of the cylindrical body, and the results of the displacement are compared with ground truth data (e.g., a measurement of the actual force applied for each displacement).

[0040] A light source may be used to provide illumination within the cylindrical body to facilitate monitoring of the displacement of the plurality of protrusions and / or markings by the visual sensor. The light source may be cylindrical and extend from a second end of the cylindrical body (e.g., opposite to the first end where the visual sensor is disposed) along the center of the cylindrical body. The light source may include a polarizing film covering a surface of the cylindrical light source to filter out any unwanted reflected light, thereby advantageously improving the accuracy of monitoring the displacement of the plurality of markings and / or protrusions by the visual sensor.

[0041] Figure 1An exemplary illustration of a device 100 for determining a force applied to a deformable material according to multiple embodiments of the present disclosure is shown. The device 100 may include a cylindrical body 128. In this example, the curved surface 106 of the cylindrical body 128 is straight along the length from the first end 102 to the second end 104 of the cylindrical body 128. In one embodiment, the curved surface can be an ergonomic shape, for example, to conform to the shape of a hand holding the cylindrical body 128 or other similar shapes. The curved surface 106 of the cylindrical body 128 can be made of a deformable material. The deformable material can be artificial skin, rubber, silicone, or other similar materials. In one embodiment, the entire cylindrical body 128 can be made of a deformable material. In another embodiment, a portion of the cylindrical body 128 (e.g., the outer surface and / or inner surface of the curved surface 106 of the cylindrical body 128) can be made of a deformable material.

[0042] The cylindrical body 128 may include a plurality of protrusions 110 arranged on the inner surface 108 of the cylindrical body 128, each protrusion 110 extending in a direction from the inner surface toward the center of the cylindrical body 128. The plurality of protrusions 110 may be movably attached to the inner surface and configured to be able to move in the direction in response to deformation of the cylindrical body 128, for example, the cylindrical body can be deformed when a force is applied to the outer surface of the cylindrical body 128. Each protrusion 110 may also have a color at an end 112 (e.g., an end of the protrusion 110 that moves toward or away from the center of the cylindrical body due to a force applied to the cylindrical body), which is different from the color of the inner surface 108, which can advantageously improve the accuracy of monitoring the displacement of the plurality of protrusions 110. The colors of the plurality of protrusions 110 may also be different from each other. The protrusions 110 may be straight and may be made of a deformable material. In one embodiment, the protrusions 110 may be made of a material different from the cylindrical body 128.

[0043] In addition, a plurality of holes 114 may be arranged at the curved surface 128, each of the plurality of holes 114 extending through a corresponding protrusion of the plurality of protrusions 110. In addition, a plurality of domes 116 may be arranged at the curved surface 128 (e.g., on the outer surface of the cylindrical body 128), wherein each of the plurality of domes 116 covers a corresponding hole of the plurality of holes 114. The plurality of domes 116 may be made of a material that can change shape due to, for example, a force acting on the dome, and the material may be elastic so that it can return to its original shape after the force acting on the dome is removed. Therefore, the dome 116 may be made of a deformable material or a different material having the above-mentioned properties.

[0044] The device 100 may also include a plurality of markers 118, which are thin, hair-like structures (e.g., thinner than the protrusions 110 so that they can pass through the holes 114). Each marker 118 may be attached to the dome 116 at one end and extend from the dome 116 to the hole 114 covered by the dome 116. Each hole 114 may extend through a corresponding protrusion of the plurality of protrusions 110, and each marker 118 may be configured to move along the corresponding hole 114 in response to deformation of the corresponding dome 116 (e.g., due to forces acting on the dome 116). Each marker 118 may also have a color at the other end 120 (e.g., the end of the marker 118 that moves into or out of the hole 114 due to forces acting on the corresponding dome 116) that is different from the color of the inner surface 108 and the color of the corresponding protrusion 110, which may advantageously improve the accuracy of monitoring the displacement of the plurality of markers 118. The colors of the plurality of markers 118 may also be different from each other. For example, the end 120 of each mark 118 can be red, the end 112 of each protrusion 110 can be green, and the inner surface 108 can be white. It will be understood that the colors can also have other combinations and variations. In addition, each mark 118 can be straight and can be made of a deformable material or a different material from the dome 116.

[0045] A visual sensor 122 (such as a camera) may be disposed at the first end 102 of the cylindrical body 128, which may be configured to monitor any displacement of the plurality of protrusions 110 and / or the markings 118. Sensor data (e.g., images, videos, and other similar data) obtained from the visual sensor 122 may be used by the system to determine the length of the displacement of each of the plurality of protrusions 110 captured by the visual sensor 122, and to determine the force applied to the cylindrical body 128 based on a comparison of the length of the displacement with historical data. In one embodiment, the length of each of the plurality of protrusions 110 may be proportional to the distance of each of the plurality of protrusions 110 from the first end 102 of the cylindrical body 128, which advantageously enables a single visual sensor 122 disposed at the first end 102 to monitor the displacement of the plurality of protrusions 110. In one embodiment, the length of each mark 118 in the plurality of marks 118 is proportional to the distance of each mark 118 in the plurality of marks 118 from the first end of the cylindrical body, which advantageously enables the single visual sensor 122 disposed at the first end 102 to monitor the displacement of the plurality of marks 118. This design can effectively avoid mutual occlusion of the marks 118 and the protrusions 110 due to deformation of the cylindrical body 128 and / or the dome 116, thereby improving the accuracy of optical tracking by the visual sensor 122. In one embodiment, each colored end of the mark 118 and the protrusion 110 can be tilted and face the visual sensor 122 to facilitate the visual sensor 122 to capture the displacement of the protrusion 110 and the mark 118.

[0046] A light source 124 may be used to provide illumination within the cylindrical body 128 to facilitate the visual sensor to monitor any displacement of the plurality of protrusions 110 and / or markers 118. The light source 124 may be cylindrical to provide uniform light on the surface of the plurality of protrusions 110 and markers 118, and may extend from the second end 104 of the cylindrical body 128 (e.g., opposite the first end 102 where the visual sensor 122 is disposed) along the center of the cylindrical body 128. The light source 124 may include a polarizing film 126 covering the surface of the cylindrical light source 124 to filter out any unwanted reflected light, thereby advantageously improving the accuracy of the visual sensor 122 in monitoring the displacement of the plurality of markers 110 and / or protrusions 118.

[0047] Figure 2An exemplary diagram 200 is shown of how displacement of a marker may be detected in accordance with multiple embodiments of the present disclosure. A visual sensor 202 may be disposed at one end (e.g., at the first end 102 of the cylindrical body 128) to monitor displacement of three markers 206, 212, and 224 with the aid of illumination provided by a light source (e.g., the light source 124 in the cylindrical body 128). For simplicity, only three groups of markers are shown here. Since no force is applied to the domes 204 and 222, the markers 206 and 224 do not move, and thus the ends 216 and 226 of the markers 206 and 224 do not move toward or away from the respective ends of the respective protrusions 208 and 228. When a force 220 is applied to the dome 210, the marker 212 moves (e.g., with reference to FIG. 214 ). Figure 1 , moves along the corresponding hole 114 in the corresponding protrusion 110, and further moves toward the center of the cylindrical body 128), so that the end 214 of the mark 212 approaches the end of the corresponding protrusion 218, and then protrudes through the end of the corresponding protrusion 218. Therefore, the displacement of the end 214 of the mark 212 caused by the movement of protruding through the end of the corresponding protrusion 218 is captured by the visual sensor 202.

[0048] In one embodiment, the end 214 of the mark may not need to extend from the corresponding protrusion 218 in order for the visual sensor 202 to detect its movement. For example, as shown in the diagram 200, each colored end of the marks 206, 212 and 224 and each colored end of the protrusions 208, 218 and 228 can be tilted and facing the visual sensor 202 to facilitate the visual sensor 202 to capture the displacement of the protrusions and marks. Therefore, due to the tilted characteristics of the ends of each mark and protrusion, the colored end 214 of the mark 212 extends beyond the end of the protrusion 218 but is still located within the protrusion 218 (e.g., does not extend beyond the end of the protrusion 218), so it is visible to the visual sensor 202, and thus the movement of the end of the protrusion 218 can be detected by the visual sensor 202. It is also possible that the projections 208, 218, and 228 may all move (e.g., each displaced a different length) due to the force 220 applied to the dome 210, but by detecting the end 214, it can be determined at which location of the device the force was applied (e.g., due to the displacement of the end 214 (e.g., the displacement is greater), thereby determining that the force 220 is applied to the dome 210). Therefore, the sensitivity of the device in detecting and determining the force applied to the cylindrical body is advantageously increased. It will be understood that the projections 208, 218, and 228 can move (e.g., refer to Figure 1, further moving toward the center of the cylindrical body 128), this is due to the force applied on the curved surface around and near the corresponding domes 204, 210 and 222, but the force is not directly applied to the domes 204, 210 and 222, so that the corresponding marks 206, 212 and 224 will not move within the corresponding protrusions 208, 218 and 228, which enables a more accurate force distribution map to be generated.

[0049] Figure 3 An exemplary process 300 of how to determine forces and force distributions according to multiple embodiments of the present disclosure is shown. At stage 302, a force is applied to the device 100, such as by applying a handshake around the cylindrical body 128 as shown in this illustration or by other similar force application methods. As shown in this illustration, the displacement of one or more protrusions and / or one or more marks due to the applied force can be captured by a visual sensor such as a camera (e.g., arranged at the first end 102 of the device 100). At stage 304, one or more images of the multiple marks and protrusions captured from the camera (e.g., indicating the displacement due to the force applied in stage 302) can be used as input to an image segmentation algorithm. Under illumination from a light source (e.g., light source 124), the marks and protrusions of different colors (e.g., blue, green, red, or other similar colors) are configured to have a significant color contrast with the inner surface of the cylindrical body (e.g., white or other similar colors). Therefore, an image segmentation algorithm (e.g., a color segmentation algorithm or other similar algorithm) can be used to extract the pixel coordinates of different markers and protrusions (e.g., the x-axis and y-axis pixel coordinates of each marker and protrusion in one or more images captured by the camera). Then, through morphological calculations, the three-dimensional (3D) coordinates of each protrusion and marker can be obtained in stage 306, and in stage 308, the displacement matrix of the markers and protrusions can be calculated.

[0050] With the displacement matrix as input, a force distribution map can be reconstructed (e.g., as shown in stage 312), which can provide information about the forces and the distribution of the forces applied to the cylindrical body during stage 302. To this end, an artificial intelligence (AI)-based force reconstruction algorithm consisting of a physical model and a deep neural network can be used to process the displacement matrix, such as a deep neural network (DNN) assisted by a finite element model (FEM) in stage 310. The physical model can include equations based on, for example, the Reissner-Mindlin Flat Shell Theory or other similar theories, using dynamic models, physical equations, geometric equations, and nodal degree of freedom (DoF) equations, such as those described below. The FEM shell of the cylindrical body (including multiple nodes or cells that constitute the curved surface of the cylindrical body) can be constructed by traversing the outer surface of the cylindrical body based on the DoF until each node or cell is covered, for example, according to Figure 4 400 showing a node 402, wherein the traversal 412 around the four corners 404, 406, 408 and 410 of the node 402 is in a counter-clockwise direction and according to Figure 5 500, where each cell or node around the cylindrical body is traversed (e.g., traversed along the path in a counterclockwise direction around a central axis representing the center of the cylindrical body). Each node or cell can represent the location of each dome (and therefore each protrusion and mark) of the cylindrical body. An exemplary finite element model (FEM) of the cylindrical body is shown in Figure 6 600 is shown in FIG.

[0051] For example, in a physical model between nodal displacement and force magnitude, a shell (e.g., a curved surface representing a cylindrical body, see also Figure 6 600) is discretized into a flat shell model based on a four-node rectangular shell, for example. According to the Reisner-Mindlin flat shell theory, the coordinates of node O in the local coordinate system (see also Figure 4 The diagram 400 and Figure 5 The diagram 500) can be represented as:

[0052] u′(x′,y′,z′)=u′0(x′,y′)-z′θ 0x′ (x′,y′)

[0053] v′(x′,y′,z′)=v′0(x′,y′)-z′θ 0y′ (x′,y′)

[0054] w′(x′,y′,z′)=w′0(x′,y′)

[0055] where u′0, v′0 and w′0 are the displacements of node O along the local directions x′, y′ and z′ respectively; θ 0x′ and θ 0y′ are the rotation angles in the local planes x′z′ and y′z′ respectively; the local displacement vector of node O is d′0=[u′0,v′0,w′0,θ 0x′ ,θ 0y′ ] T . Therefore, the displacement field inside the rectangular element can be expressed by the following equation:

[0056]

[0057] Where N i =diag(N i ,N i ,N i ,N i ,N i ) represents the matrix of the shape function, where ξ,η∈[-1 1], ξ 1,4 =-1,η 1,2 =-1,ξ 2,3 =1,η 3,4 = 1. Related strain and stress can be written as follows:

[0058]

[0059] in, and are the membrane strain matrix, the bending strain matrix, and the transverse shear strain matrix, respectively, and are given by:

[0060]

[0061] The constitutive matrix is ​​given by:

[0062]

[0063] Where E, v and t are elastic modulus, Poisson's ratio and shell thickness respectively, κ is the shear correction factor and is equal to 5 / 6 for isotropic materials. By applying the virtual work principle and constitutive equation (3), we can have:

[0064]

[0065] Substituting equations (1) and (2) into equation (4), the equilibrium equation of the element is:

[0066] k′ e u′ e =f′e (5)

[0067] Among them, f′ e represents the local force vector combining the surface load and the node load, and the local stiffness matrix k′ e It can be obtained by:

[0068]

[0069] in:

[0070]

[0071] Then the global system (e.g., a cylindrical housing or body representing the device) can be described as:

[0072] u e =L i (φ)·u′ e

[0073] f e =L i (φ)·f′ e

[0074]

[0075] The transformation matrix L i The equation for (φ) (e.g., the displacement matrix) is:

[0076]

[0077] Wherein, the angle φ of an element is the angle between the global x-axis and the normal vector of the element. It will be appreciated that other variations of the above equations may also be used to obtain the displacement matrix.

[0078] Therefore, after obtaining the displacement matrix, the physical model can be used, for example, based on equation F p =K·U (as shown above, see equation 5) to calculate the initial force distribution F p , where K is the stiffness matrix in the physical model and U is the obtained displacement matrix. It will be appreciated that other similar physical models and equations may also be used for FEM-assisted DNN. In the FEM-assisted DNN of stage 310, the model error e that may exist in the physical model is compensated by the DNN (e.g., ResNet or other similar DNN) according to the equation e=g(U), where g() represents the DNN and U is the obtained displacement matrix. Therefore, the final force distribution map can be calculated by the following equation: F=F p +e=K·U+g(U) The above calculation process can be completed within, for example, 0.1 seconds to dynamically display the force distribution diagram.

[0079] like Figure 7 As shown, the DNN can be trained with multiple displacement matrices (obtained from multiple training iterations) and ground truth data of force distribution. For example, in order to collect training data for the DNN, a three-DoF test platform 704 with controllable position and angle can be used in the first stage 702 to apply force at each point of the cylindrical body 706 (e.g., each node or unit of the cylindrical body 706). The three-DoF may include angular degrees of freedom around the cylindrical body 706 (e.g., by rotation 708 of the cylindrical body 706), planar position degrees of freedom (e.g., by translation 710 along the plane of the test platform 704), and clamping distance degrees of freedom (e.g., by up and down movement of the clamping device 712), which are input into the DoF of the node equation. The test platform 704 can be equipped with a standard force sensor 716, which is configured to simultaneously record the actual force distribution map when the force 714 is applied to the cylindrical body 706 to be used as ground truth data. In the next stage 718, a corresponding plurality of images 720 captured from a visual sensor such as a camera (e.g., arranged at an end of the cylindrical body 706, such as arranged at the first end 102 of the cylindrical body 128), a corresponding plurality of marker displacement matrices 722 generated based on the plurality of images 720, and corresponding ground truth data 724 obtained from the force sensor 716 are collected to establish a large amount of training data. In the next stage 726, the DNN is trained based on the data obtained in the stage 718. The entire process of collecting data and training the DNN can be automated and completed in a few hours (depending on the central processing unit (CPU) used). Therefore, based on the trained DNN, the distribution of residual forces can be determined in response to a force residual map (e.g., a map including the difference between the observed and predicted values ​​of the force applied to the cylindrical body) applied to the cylindrical body 128 of the device 100, and the results are displayed by a distributed force map by combining the force map of the physical model with the force residual map. Advantageously, the DNN predicts the error between the above physical model and the ground truth data to provide an improved system for determining the forces and distribution of forces applied to the cylindrical body. In this section, the function of the DNN is to predict the error between the above physical model and the ground truth data.

[0080] When the cylindrical body is grasped by hand, most of the force is usually distributed on the fingertips and flexor pollicis brevis, which is consistent with the force characteristics of grip force and indicates that the results (e.g. Figure 3The distributed force obtained by the force reconstruction algorithm based on artificial intelligence is accurate in the case of at least second-order continuity on the curved surface of the cylindrical body, so the obtained distributed force has high-resolution accuracy. At the same time, the optical tracking of the marker has high robustness. Therefore, it is advantageous that high resolution and high robustness for distributed force reconstruction can be achieved simultaneously.

[0081] In another application example of the device 100, the visual sensor 122 may capture Figure 8 800 in FIG. 1 , which shows the displacement of the plurality of markings 118 and the protrusions 110 due to the forces exerted on the cylindrical body 128. Fig. 9 As shown in diagram 900 of , a 3D reconstruction of the cylindrical body 128 (eg, deformed by forces applied to the cylindrical body 128) may be generated based on the captured images. Fig.10 As shown in diagram 1000 of , the 3D reconstruction may be processed to form an unfolded deformation matrix. Fig.11 As shown in the diagram 1100, based on historical data (e.g., from Figure 7 The deformation matrix is ​​analyzed using the training data obtained from the training setting 700 to obtain the image reconstruction.

[0082] Fig.12 An exemplary diagram 1200 is shown of how the device and system for determining forces applied to a deformable material may be utilized in accordance with various embodiments of the present disclosure. For example, the device and system may be applied to interaction with virtual reality and augmented reality (e.g., by using the device as a controller), medical assessment and rehabilitation (e.g., for injured hands), scientific training assessment (e.g., for sports, exercise, bodybuilding, and other similar applications), and robotic sensing (e.g., by using the device as part of a robot / machine or as a controller for interacting with a robot / machine or other similar applications).

[0083] Fig.13 A schematic diagram of an exemplary computing device suitable for determining a force exerted on a deformable material is shown.

[0084] Fig.13 An exemplary computing device 1300, hereinafter interchangeably referred to as a computer system 1300, is shown, wherein one or more such computing devices 1300 may be used as a system for determining a force applied to a deformable material and may perform at least the following steps: Figure 3 and Fig.11 The following description of computing device 1300 is provided by way of example only and is not intended to be limiting.

[0085] like Fig.13As shown, the example computing device 1300 includes a processor 1304 for executing software routines. Although a single processor is shown for clarity, the computing device 1300 may also include a multi-processor system. The processor 1304 is connected to a communication infrastructure 1306 for communicating with other components of the computing device 1300. The communication infrastructure 1306 may include, for example, a communication bus, a crossbar, or a network.

[0086] The computing device 1300 also includes a main memory 1308, such as a random access memory (RAM), and a secondary memory 1310. The secondary memory 1310 may include, for example, a storage drive 1312, which may be a hard drive, a solid state drive, or a hybrid drive; and / or a removable storage drive 1314, which may include a tape drive, an optical drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive, or a memory card), etc. The removable storage drive 1314 reads from and / or writes to a removable storage medium 1318 in a known manner. The removable storage medium 1318 may include a magnetic tape, an optical disk, a non-volatile memory storage medium, etc., which is read and written by the removable storage drive 1314. It will be understood by those skilled in the art that the removable storage medium 1318 includes a computer-readable storage medium having computer-executable program code instructions and / or data stored therein.

[0087] In alternative implementations, the secondary memory 1310 may additionally or alternatively include other similar mechanisms for allowing computer programs or other instructions to be loaded into the computing device 1300. For example, such mechanisms may include, for example, a removable storage unit 1322 and an interface 1320. Examples of removable storage units 1322 and interfaces 1320 include program cartridges and cartridge interfaces (such as those found in video game console devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, removable solid-state storage drives (such as USB flash drives, flash memory devices, solid-state drives, or memory cards), and other removable storage units 1322 and interfaces 1320 that allow software and data to be transferred from the removable storage unit 1322 to the computer system 1300.

[0088] The computing device 1300 also includes at least one communication interface 1324. The communication interface 1324 allows software and data to be transmitted between the computing device 1300 and an external device via a communication path 1326. In various embodiments of the present disclosure, the communication interface 1324 allows data to be transmitted between the computing device 1300 and a data communication network (such as a public data or private data communication network). The communication interface 1324 can be used to exchange data between different computing devices 1300, which form part of an interconnected computer network. Examples of the communication interface 1324 may include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial port, a parallel port, a printer, a GPIB, an IEEE 1394, an RJ45, a USB), an antenna with associated circuits, and the like. The communication interface 1324 may be wired or wireless. The software and data transmitted through the communication interface 1324 are in the form of signals, which may be electronic signals, electromagnetic signals, optical signals, or other signals that can be received by the communication interface 1324. These signals are provided to the communication interface via a communication path 1326.

[0089] like Fig.13 As shown, computing device 1300 also includes a display interface 1302 and an audio interface 1332 , wherein display interface 1302 performs operations for presenting images or videos to an associated display 1330 , and audio interface 1332 performs operations for playing audio content through associated speakers 1334 .

[0090] As used herein, the term "computer program product" may refer in part to removable storage media 1318, removable storage unit 1322, a hard disk installed in storage drive 1312, or a carrier wave that transfers the software to communication interface 1324 via communication path 1326 (wireless link or cable). Computer-readable storage media refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and / or data to computing device 1300 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray disc, hard drive, ROM or integrated circuit, solid-state storage drive (such as USB flash drive, flash memory device, solid-state drive or memory card), hybrid drive, magneto-optical disk, or computer-readable card (such as PCMCIA card), etc., whether such devices are internal or external to computing device 1300. Examples of temporary or intangible computer-readable transmission media that may also be involved in providing software, applications, instructions and / or data to computing device 1300 include radio or infrared transmission channels, as well as a network connection to another computer or networked device, and the Internet or an intranet (including email transmissions and information recorded on websites), etc.

[0091] Computer programs (also referred to as computer program code) are stored in the main memory 1308 and / or the secondary memory 1310. The computer program may also be received via the communication interface 1324. Such computer programs, when executed, enable the computing device 1300 to perform one or more features of the embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 1304 to perform the features of the embodiments described above. Thus, such computer programs represent controllers of the computer system 1300.

[0092] The software may be stored in a computer program product and loaded into the computing device 1300 using the removable storage drive 1314, the storage drive 1312, or the interface 1320. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system 1300 via the communication path 1326. The software, when executed by the processor 1304, causes the computing device 1300 to perform the necessary operations as a system for determining forces applied to a deformable material, to perform processes, to perform calculations, to generate 3D reconstructions and force maps, and as such Figures 3 to 11 In one implementation, a system for determining a force applied to a deformable material may include a device 100, a visual sensor located at a first end 102 of a cylindrical body 128, and monitoring displacements of a plurality of protrusions 110 in response to deformation of the cylindrical body 128. The system may be configured to: determine a length of displacement of each of the plurality of protrusions 110 and / or markers 118 captured by the visual sensor; and determine the force applied to the device 100 based on a comparison of the length of the displacement with historical data. The system may also be configured to calculate a distribution of the force applied to the device 100 based on the force determined for each of the plurality of protrusions 110 and / or markers 118.

[0093] It should be understood that Fig.13 The embodiments of the computing device 1300 are presented by way of example only to explain the operation and structure of the system for determining the force applied to a deformable material. Therefore, in some embodiments, one or more features of the computing device 1300 may be omitted. Moreover, in some embodiments, one or more features of the computing device 1300 may be combined together. Additionally, in some embodiments, one or more features of the computing device 1300 may be separated into one or more component parts.

[0094] Those skilled in the art will appreciate that many changes and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the scope of the present disclosure as broadly described. Therefore, the present embodiments should be considered in all aspects as illustrative and not restrictive.

Claims

1. A device, comprising: a main body, said main body being made of a deformable material; as well as A plurality of protrusions are movably attached to the inner surface of the body, and each of the plurality of protrusions extends in a direction from the inner surface toward the center of the body and is configured to be movable in the direction in response to deformation of the body.

2. The device according to claim 1, wherein The length of each of the plurality of protrusions is proportional to the distance of each of the plurality of protrusions from the first end of the body.

3. The apparatus of claim 1, further comprising a color at an end of each of the plurality of protrusions, the color being different from a color of the inner surface of the body to facilitate monitoring displacement of the plurality of protrusions.

4. The apparatus according to claim 1, further comprising: a plurality of holes disposed at an outer surface of the body, and each hole of the plurality of holes extending through each protrusion of the plurality of protrusions; a plurality of domes located on the outer surface of the body, each dome of the plurality of domes covering each hole of the plurality of holes; as well as A plurality of markers are attached to the plurality of domes at one end, each marker of the plurality of markers extending from each dome of the plurality of domes into each hole of the plurality of holes, wherein each marker of the plurality of markers is configured to be movable along each hole of the plurality of holes in response to deformation of each dome of the plurality of domes.

5. The device according to claim 4 further includes a color at the other end of each of the multiple marks, which is different from the color at the end of each of the multiple protrusions and the color of the inner surface of the body, so as to facilitate monitoring the displacement of the multiple marks and the multiple protrusions.

6. The device according to claim 4, wherein: The plurality of domes are made of a deformable material.

7. The apparatus of claim 2, further comprising a vision sensor located at the first end of the body for monitoring displacement of the plurality of protrusions in response to deformation of the body.

8. The apparatus of claim 7, further comprising a cylindrical light source extending from a second end of the body through a center of the body, the second end being opposite to the first end. 9 . The apparatus of claim 8 , the cylindrical light source further comprising a polarizing film covering a surface of the cylindrical light source.

10. The device according to claim 1, wherein: The deformable material is artificial skin.

11. A system for monitoring a force applied to a deformable material, the system comprising: The device according to claim 1; a visual sensor located at the first end of the body for monitoring displacement of the plurality of protrusions in response to deformation of the body; at least one processor; as well as at least one memory, the at least one memory comprising computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, enable the system to at least: determining a length of a displacement of each of the plurality of protrusions captured by the vision sensor; as well as Based on a comparison of the length of the displacement with historical data, the force applied to the device is determined.

12. The system of claim 11, further configured to calculate a distribution of forces exerted on the device based on the forces determined for each of the plurality of protrusions.

13. The system according to claim 11, wherein: The length of each of the plurality of protrusions is proportional to the distance of each of the plurality of protrusions from the first end of the body.

14. The system of claim 11, further comprising a color at an end of each of the plurality of protrusions, the color being different from a color of the inner surface of the body to facilitate monitoring displacement of the plurality of protrusions.

15. The system according to claim 11, further comprising: a plurality of holes disposed at an outer surface of the body, each hole of the plurality of holes extending through each protrusion of the plurality of protrusions; a plurality of domes located on the outer surface of the body, each dome of the plurality of domes covering each hole of the plurality of holes; as well as A plurality of markers are attached to the plurality of domes at one end, each marker of the plurality of markers extending from each dome of the plurality of domes into each hole of the plurality of holes, wherein each marker of the plurality of markers is configured to be movable along each hole of the plurality of holes in response to deformation of each dome of the plurality of domes.

16. The system according to claim 15 further includes a color at the other end of each of the plurality of marks, the color being different from the color at the end of each of the plurality of protrusions and the color of the inner surface of the body, wherein the visual sensor is configured to monitor the displacement of the plurality of marks and the plurality of protrusions based on the displacement of the corresponding colors.

17. The system according to claim 15, further configured to: determining a length of a displacement of each of the plurality of marks captured by the vision sensor; and Based on a comparison of the length of the displacement with historical data, the force applied to the device is determined.

18. The system of claim 17, further configured to calculate a distribution of forces applied to the device based on the forces determined for each of the plurality of markers.

19. The system of claim 11, further comprising a cylindrical light source extending through a center of the body from a second end of the body, the second end being opposite the first end.

20. The system of claim 19, the cylindrical light source further comprising a polarizing film covering a surface of the cylindrical light source.

Images