Multi-dimensional force decoupling method based on marker point tracking

Abstract

The application relates to the technical field of tactile sensing, and discloses a multi-dimensional force decoupling method based on marker point tracking. The method uses a camera to track the area change and centroid displacement field of a marker point on a flexible skin surface, realizes the calculation and decoupling of multi-dimensional force through vector operation mathematical analysis, and specifically comprises the following steps: calculating the area and displacement field of the marker point through a visual image; the area change can be used to realize the measurement of normal pressure; the displacement field is differentiated to obtain a surface deformation shear strain field, and the curl of the shear strain field can be used to obtain the size and direction of shear force and the information of a normal torque. The application provides a real-time multi-dimensional force decoupling measurement method based on a physical model and a mathematical analysis method, has the characteristics of self-decoupling and universality, does not depend on large-scale training data, reduces the calculation complexity and computing power cost, and the algorithm is compatible with different specifications of hardware forms and can be used for multi-dimensional force sensing in multiple fields such as robots, medical devices and human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of tactile sensing technology, specifically to a method for real-time decoupling of multidimensional forces in complex scenarios based on camera tracking of displacement and area changes of marker points on flexible skin. Background Technology

[0002] In recent years, with the rapid development of robotics technology, intelligent robots have been increasingly widely used in various fields such as industry, medicine, and daily life. Touch, as one of the core capabilities of robots in perceiving their external environment, plays an irreplaceable role in humanoid robots and dexterous manipulation. Through tactile perception, robots can accurately acquire key physical properties of objects such as hardness, temperature, texture, and shape, thereby achieving more precise operation and environmental adaptability, significantly improving the robot's ability to operate in complex scenarios.

[0003] Visor-tactile sensors utilize high-resolution cameras to capture the deformation of elastic bodies and model it using computer vision algorithms to generate high-precision tactile data such as six-dimensional force / torque and stress distribution, achieving micron-level detection and intelligent feature extraction. Compared to traditional electronic sensors, it offers advantages such as high information density, strong environmental robustness, and low cost. Its non-contact optical measurement avoids complex circuitry, balancing high resolution and sensitivity. This technology shows significant potential in fields such as industrial precision assembly, medical surgical robots, and aerospace. With advancements in algorithms and flexible materials, it is becoming a key enabling technology driving the intelligent and precise development of robots.

[0004] Despite the significant advantages of visual-tactile sensors in robotics perception, their practical application still faces numerous technical challenges. First, high computational complexity limits real-time performance. Relying on computer vision algorithms and deep learning neural networks to process high-resolution images requires substantial computation, making it difficult to meet millisecond-level real-time feedback requirements. This can lead to control instability in high-speed robot operations or human-robot interaction scenarios. Second, the nonlinear properties of elastomer materials significantly limit their application, making direct mathematical modeling difficult. Furthermore, long-term use can lead to aging and creep, altering deformation characteristics and affecting long-term measurement stability, increasing maintenance costs. Additionally, optical imaging is sensitive to changes in ambient light and is susceptible to interference from shadows, reflections, and other factors, potentially causing failure under complex lighting conditions. Finally, multi-dimensional contact force reconstruction technology is still underdeveloped. Existing algorithms have limited ability to model complex deformations (such as shear and torsion), struggle to eliminate crosstalk between axial forces, and suffer from poor real-time performance. These issues directly impact the application of visual-tactile sensors in robotics. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multidimensional force decoupling method based on marker point tracking. The method calculates the centroid displacement field of the marker point using tactile images, performs spatial differentiation on the displacement field to obtain the surface deformation shear strain field, and achieves real-time decoupling of multidimensional forces through generalized vector calculation and calibration of the shear strain field.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A multidimensional force decoupling method based on marker point tracking includes:

[0008] The displacement field and area change rate of the array of marked points on the surface of flexible skin are acquired using a camera after contact occurs.

[0009] Calculate the normal force of contact based on the rate of change of the area of ​​the marked points:

[0010] The shear force at contact is calculated based on the curl field of the displacement field, and the direction of the shear force is analyzed based on the curl field distribution.

[0011] The torsional force at contact is calculated based on the center of curl of the curl field, and the direction of the torsional force is analyzed based on the distribution of the curl field.

[0012] In one embodiment, acquiring the displacement field and area change rate of the flexible skin surface marker array after contact occurs via a camera specifically includes:

[0013] By using a camera to image the marked points on the surface of flexible skin, and tracking the position of the centroid and the size of the area of ​​the marked points in real time, a three-dimensional vector field of the marked points is obtained. ,in, Mark the x-axis and y-axis displacements of the points respectively. Composition of the displacement field , This represents the rate of change of the area of ​​the marked point (implying z-axis displacement information).

[0014] In actual calculations, linear interpolation can be used to subdivide the generalized displacement field at the marked points into a grid. The interpolation grid is as follows:

[0015] ;

[0016] Represents grid nodes physical coordinates Integer indices representing the grid nodes along the x-axis and y-axis. This represents the maximum index of the grid along the x-axis. This represents the maximum index of the grid along the y-axis.

[0017] In one embodiment, calculating the contact normal force based on the rate of change of the area of ​​the marked points specifically includes:

[0018] ;

[0019] in, The rate of change of area, For the contact normal pressure, The area of ​​the marked point when there is no initial contact. The area of ​​the marked point after the contact occurs. This represents the coefficients of a quadratic curve.

[0020] In one embodiment, the step of calculating the contact shear force based on the curl field of the displacement field and analyzing the shear force direction based on the curl field distribution specifically includes:

[0021] Shear force for:

[0022] ;

[0023] in, Indicates the mean curl. Indicates the maximum curl. For displacement field The curl field, Represents the vector differential operator. It conforms to The number of grids, These are the coefficients of the conic section;

[0024] The direction of shear force, as analyzed from the curl field distribution, is as follows: the line connecting the centers of the positive and negative crescent-shaped symmetrical curl regions generated when the shear force is applied rotates 90° counterclockwise and passes through the midpoint of the line.

[0025] In one embodiment, the step of calculating the torsional force of the contact based on the center of curl of the curl field and analyzing the direction of the torsional force based on the curl field distribution specifically includes:

[0026] ;

[0027] in, For torsional force, For the curl field, The first-order curve coefficient represents the center of curl of the curl field. for:

[0028]

[0029] This represents the maximum index of the grid along the x-axis. This represents the maximum index of the grid in the y-direction. Indicates at grid points The curl field value at that point, i.e., the vector field curl at that point These represent the coordinates of the curl center of the curl field;

[0030] The direction of torsional force, as analyzed by the curl field distribution, is as follows: within a circular curl region, positive curl is counterclockwise rotation, negative curl is clockwise rotation, and the central axis of rotation passes through the center of curl.

[0031] Furthermore, the rate of change of the area of ​​the marker points refers to the average rate of change of the area of ​​the marker points in the contact area.

[0032] Furthermore, This is obtained by calibrating contact force data.

[0033] In one embodiment, when multiple contact areas exist, the displacement field is used. divergence field Separate multiple contact regions and independently apply the multidimensional force decoupling method as described in claim 1 to each contact region.

[0034] The measurement of normal force, shear force, and torsional force in this invention has a self-decoupling characteristic.

[0035] Compared with the prior art, the beneficial technical effects of the present invention are:

[0036] (1) The miniaturized visual-tactile sensor proposed in this invention uses a standard imaging module, a multi-functional board, a 3D printed shell and a modular assembly design. It is easy to manufacture, has a low cost, and communicates wirelessly with the host computer. It is portable, has a variety of application scenarios and strong expandability.

[0037] (2) The real-time multidimensional contact force decoupling algorithm based on the vector calculation and mathematical analysis of the displacement field of the marker point proposed in this invention has high resolution, high accuracy, good real-time performance, simple calibration process and is applicable to sensing hardware of different sizes, and has strong versatility.

[0038] (3) The multi-dimensional contact force decoupling algorithm proposed in this invention can be extended to complex scenarios with multiple contact forces to realize the real-time decoupling function of multi-point contact forces.

[0039] In summary, the real-time multidimensional force decoupling method proposed in this invention separates and decouples different components of multidimensional contact forces based on the principle of linear superposition of forces. It obtains the contact force distribution using a vector mathematical analysis method driven by a physical model. The method exhibits outstanding performance, strong scalability, strong versatility, and robustness to hardware structure. It is expected to promote the widespread application of visual-tactile sensors in intelligent robots, intelligent manufacturing, medical and health care, and human-computer interaction, thereby promoting the development of related fields and achieving good social and economic benefits. Attached Figure Description

[0040] Figure 1 This is a flowchart of a real-time multi-point multi-dimensional contact force decoupling method according to the present invention.

[0041] Figure 2 This is an exploded view of the structure of a visual-tactile sensor according to the present invention.

[0042] Figure 3 This is a schematic diagram illustrating the positive pressure measurement principle of the present invention.

[0043] Figure 4 This is a schematic diagram of the contact force calibration platform of the present invention.

[0044] Figure 5 This is a schematic diagram illustrating the positive pressure measurement principle and calibration results of the present invention.

[0045] Figure 6 This is a schematic diagram illustrating the shear force measurement principle and calibration results of the present invention.

[0046] Figure 7 This is a schematic diagram illustrating the torsional force measurement principle and calibration results of the present invention.

[0047] Figure 8 This is the principle of decoupling shear force and torsional force in this invention.

[0048] Figure 9 This is a schematic diagram of the multi-point contact force decoupling principle of the present invention.

[0049] Figure 10 This demonstrates the decoupling results of normal pressure and shear force in this invention.

[0050] Figure 11 This demonstrates the decoupling results of the normal force and torsional force of the present invention.

[0051] Figure 12 This demonstrates the decoupling results of shear force and torsional force in this invention.

[0052] Figure 13 This is a schematic diagram illustrating the evaluation of the shear force direction calculation effect of the present invention.

[0053] Figure 14 This is a demonstration of the contact force results during the human-computer interaction experiment of the present invention.

[0054] Figure 15 This presents the experimental results for estimating the viscosity of viscous substances by stirring, as described in this invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] The following will refer to Figures 1 to 15 Embodiments of the present invention are described in detail below. While specific embodiments of the invention are shown in the accompanying drawings, it should be understood that these descriptions are exemplary only and not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0057] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0058] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0059] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operating procedures to further explain the technical solutions of the present invention.

[0060] refer to Figure 2 This embodiment provides a visual-tactile sensor 1 for high-resolution real-time contact force reconstruction, including: a flexible skin module 100, a control module, an illumination imaging module, a wireless communication module, a power supply module, a support 200, and a housing 400.

[0061] The outer casing 400 consists of four parts: a tactile sensing part 410, a camera printed circuit board mounting base part 420, a heat dissipation battery compartment part 430, and a battery compartment cover 440, all of which are 3D printed from black light-absorbing nylon material. The tactile sensing part 410 has a top dimension of 25mm × 25mm and an opening dimension of 20mm × 20mm, representing the sensing area. The tactile sensing part 410, the camera printed circuit board mounting base part 420, the heat dissipation battery compartment part 430, and the battery compartment cover 440 are respectively secured to each other using M1.6 round-head screws 1001, 1004, and 1005. The camera printed circuit board mounting base part 420 and the heat dissipation battery compartment part 430 have pre-drilled TYPEC holes for the circuit board 700.

[0062] The support 200 is fixed to the support mounting platform of the tactile sensing part housing 410 by epoxy resin adhesive.

[0063] The flexible skin module 100 comprises a skin protective layer 110, a marker layer 120, and an elastomer layer 130. The elastomer layer 130 is made of mixed silicone and cast within the cavity formed by the support 200 and the tactile sensing portion shell 410, with its upper surface slightly raised above the tactile sensing portion shell 410, forming a slightly curved surface. The marker layer is made by applying silicone using a mask; after the silicone solidifies, it adheres to the upper surface of the elastomer layer 130, forming a marker array with a diameter of 0.5 mm and a spacing of 0.5 mm. Finally, a 0.5 mm thick ultra-black light-absorbing skin protective layer 110 is applied to the top layer, and its perimeter in contact with the shell is fixed with adhesive.

[0064] The illumination and imaging module consists of an LED light strip 300 and a camera module 500. The LED light strip is bonded to a 2 mm deep square recess in the tactile sensing housing 410 using epoxy resin adhesive. The power cable is led out from the bottom of 410 through a pre-drilled hole. The camera module 500 is fixed to the camera printed circuit board mounting base housing 420 using M3.0 round-head screws 1002. The LED light strip 300 provides a single white light source for the camera 500 to capture images of the marking layer 120. The light is scattered into approximately uniform illumination by the support 200. The distance between the camera and the marking layer 120 is determined by the field of view and the size of the sensing area. Adjusting the camera focal length allows the camera to clearly capture the marking image. An aluminum alloy heat sink 600 is fixed to the bottom of the camera 500 to assist in heat dissipation.

[0065] The control module and wireless communication module are both integrated on board 700. Board 700 is fixed to the camera printed circuit board mounting base housing 420 using M2.0 round head screws (31003). Heat sinks are attached to both the main control module and the WIFI module. The camera 500 is connected to the 30-pin flexible circuit board of board 700 via a flexible flat cable. The power supply line of the LED light strip 300 is connected to the 3.3V power supply SH1.0 terminal of the board. The invention also includes a mounting position 800 for a cooling fan.

[0066] The power supply module consists of a built-in 5V USB 2.0 Type-C port and a 5V lithium battery charging / discharging module 900. After input to the board, the voltage is boosted or bucked to the required specified voltage by a DC-DC module. The lithium battery leads are connected to the onboard SH1.0 terminal through through holes on the heat dissipation battery compartment 430.

[0067] refer to Figure 2 Based on the aforementioned hardware, the precise displacement field of the marker layer 120 can be obtained through image acquisition. Based on this displacement field, this embodiment provides a real-time multidimensional contact force decoupling algorithm, see [link to relevant documentation]. Figure 1 :

[0068] refer to Figure 5 When the sensing surface is subjected to a positive pressure At that time, the distance between the contact area marker and the lens This will decrease, resulting in a smaller area of ​​the marked points in the image. Increase. (Reference) Figure 3 Assuming the radius of the marked point is , diameter is And there was no horizontal displacement during this process, the indentation depth was The distance between the lens and the CMOS imaging plane is The diameters of the image spots before and after pressing are respectively The areas are respectively Known The following relationship exists:

[0069] ;

[0070] Furthermore, finite element simulation shows that the normal pressure With indentation depth The following relationship exists:

[0071] ;

[0072] From the above two equations, we can obtain:

[0073]

[0074] Let the rate of change of the area of ​​the marked point be: ,make The above formula can then be expressed as:

[0075] (1)

[0076] refer to Figure 4 In order to calibrate and test the accuracy of the multi-dimensional force measurement of the visual-tactile sensor in this embodiment, a high-precision multi-axis displacement loading platform was built, including a six-dimensional force sensor 2 and a displacement loading stage 3. The visual-tactile sensor 1 was fixed on the loading platform, and the contact force was changed by loading displacements along different axes.

[0077] An 8 mm cylindrical indenter is used to apply a vertical downward displacement of 0–3 mm to the sensing surface, with a step size of 0.5 mm. The average value of the rate of change of area within the contact region is taken. The coefficients of the aforementioned quadratic curve are determined as follows: .

[0078] In practical calculations, this invention can use linear interpolation to subdivide the generalized displacement field at the marked points into a grid, thereby obtaining the interpolated shear strain field. Using the interpolated shear strain field Related vector field curl To calculate the shear force in the x-axis and y-axis directions (unified as) and torsional force .

[0079] refer to Figure 6 When a shear force is applied At this time, two symmetrical crescent-shaped curl regions, one positive and one negative, are generated on both sides of the line of action of the force. The curl of the two regions is symmetrical about the line of action of the force, with the left side being positive and the right side being negative.

[0080] Take the mean curl as follows:

[0081]

[0082] in , It conforms to The number of grids.

[0083] Shear force With the mean curl The relationship can be approximated by a quadratic function:

[0084] (2)

[0085] An 8 mm cylindrical indenter was used to apply a preload of 0.5 mm to 3 mm to the sensor, with a step size of 0.5 mm. A shear displacement of 0 mm to 5 mm was applied at each indentation depth, with a step size of 0.1 mm. The process was stopped if slippage occurred. The calibrated quadratic curve coefficients are as follows: Based on the curl field distribution, the direction of shear force is analyzed as follows: the line connecting the weighted geometric centers of the two crescent-shaped symmetrical curl regions (one positive and one negative) rotates 90° counterclockwise and passes through the midpoint of the line connecting the two centers.

[0086] refer to Figure 7 When a torsional force is applied At this time, a circular curl region is generated around the center of the torque, with counterclockwise being positive and clockwise being negative.

[0087] Take the center of curl as follows:

[0088]

[0089] Torsional force With the center of curl curl value The relationship can be approximated by a linear function:

[0090] (3)

[0091] An 8 mm cylindrical indenter was used to apply preloads of 1.5 mm to 3 mm to the sensor, with a step size of 0.5 mm. A shear displacement of 0° to 15° was applied at each indentation depth, with a step size of 5°. The process was stopped if slippage occurred. The calibrated first-order curve coefficients are as follows: Based on the distribution of the curl field, the direction of the torsional force can be analyzed: within a circular curl region, positive curl is counterclockwise torsion, negative curl is clockwise torsion, and the center of the torque is the weighted geometric center of the curl region.

[0092] refer to Figure 8 Simultaneously loading At this time, since the elastic sensing layer is approximately in the linear elastic stage within the preload range of 0 mm to 3 mm, the principle of linear superposition of forces still applies. Therefore, this situation can be regarded as a single loading. The superposition model also applies to the curl field. First, calculate the curl center. and past The line of symmetry, using The resulting curl field The symmetry of the negative curl component can be used to... By performing symmetry and adding it to the coupled positive curl region, the two basic models mentioned above are decoupled.

[0093] refer to Figure 9 Based on single-point contact force decoupling, this embodiment further proposes multi-point contact force measurement: setting a threshold close to 0. , will be greater than or equal to The elements are assigned a value of 1, and the rest are 0. A breadth-first search is used to obtain all connected components. After deleting small noise regions, the equivalent geometric center position of each region is calculated, and this point is taken as the contact center position. Then, the elements within this region are... This serves as a threshold for a second, more precise calculation of the contact edge location. This process is repeated until all areas have been divided, resulting in the final list of contact areas. Represents the divergence field The standard deviation is calculated. The contact force distribution in each contact region is calculated using the multidimensional force decoupling method described above, iterating through all contact regions.

[0094] In one embodiment, Figure 10 The paper demonstrates the real-time decoupling and measurement effect of the contact force decoupling algorithm of this invention on both normal pressure and shear force when they are applied simultaneously.

[0095] In one embodiment, Figure 11 The paper demonstrates the real-time decoupling and measurement effect of the contact force decoupling algorithm of the present invention on both normal pressure and torsional force when they are applied simultaneously.

[0096] In one embodiment, Figure 12 The paper demonstrates the real-time decoupling and measurement effect of the contact force decoupling algorithm of the present invention on the simultaneous application of shear force and torsional force.

[0097] In one embodiment, Figure 13 The effectiveness of the contact force decoupling algorithm of this invention in measuring the shear force direction is demonstrated.

[0098] In one embodiment, Figure 14 The invention demonstrates how a miniaturized sensor can be mounted on a gripper to provide tactile sensing. When the gripper grasps a test tube and interacts with a person, interference signals such as collisions do not affect the stability of the grip. The gripper will only release when a person applies a stable gripping force.

[0099] In one embodiment, Figure 15 The process of stirring a viscous liquid while holding a glass rod is demonstrated. The sensor can provide key contact force information to enable the robotic arm to adapt its stirring speed by detecting the peak torque.

[0100] The above embodiments are preferred embodiments of the present invention. Those skilled in the art can make various modifications or improvements based on these embodiments. Without departing from the overall concept of the present invention, all such modifications or improvements should fall within the scope of protection claimed by the present invention. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0101] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

Claims

1. A multidimensional force decoupling method based on marker point tracking, characterized in that, include: The displacement field and area change rate of the array of marked points on the surface of flexible skin are acquired using a camera after contact occurs. Calculate the normal force of contact based on the rate of change of the area of ​​the marked points: The shear force at contact is calculated based on the curl field of the displacement field, and the direction of the shear force is analyzed based on the curl field distribution, specifically including: Shear force for: ;in, Indicates the mean curl. Indicates the maximum curl. For displacement field The curl field, Represents the vector differential operator. It conforms to The number of grids, It is the coefficient of the quadratic curve; the direction of shear force according to the curl field distribution analysis is: the line connecting the centers of the positive and negative crescent-shaped symmetrical curl regions generated when the shear force is applied rotates 90° counterclockwise and passes through the midpoint of the line; The torsional force at contact is calculated based on the center of curl of the curl field, and the direction of the torsional force is analyzed based on the distribution of the curl field.

2. The multidimensional force decoupling method based on marker point tracking according to claim 1, characterized in that, The step of acquiring the displacement field and area change rate of the marker array on the flexible skin surface after contact occurs via camera specifically includes: imaging the markers on the flexible skin surface using the camera, tracking the centroid position and area of ​​the markers in real time, and obtaining the three-dimensional vector field of the markers. ,in, These represent the x-axis displacement and y-axis displacement of the marked point, respectively. Composition of the displacement field , The area change rate of the marked point.

3. The multidimensional force decoupling method based on marker point tracking according to claim 1, characterized in that, The calculation of the contact normal force based on the area change rate of the marked points specifically includes: ； in, The rate of change of area, For the contact normal pressure, The area of ​​the marked point when there is no initial contact. The area of ​​the marked point after the contact occurs. This represents the coefficients of a quadratic curve.

4. The multidimensional force decoupling method based on marker point tracking according to claim 1, characterized in that, The calculation of the torsional force at contact based on the center of curl of the curl field, and the analysis of the direction of the torsional force based on the curl field distribution, specifically includes: ； in, For torsional force, For the curl field, The first-order curve coefficient represents the center of curl of the curl field. for: This represents the maximum index of the grid along the x-axis. This represents the maximum index of the grid in the y-direction. Indicates at grid points The curl field value at that point, i.e., the vector field curl at that point These represent the coordinates of the curl center of the curl field; The direction of torsional force, as analyzed by the curl field distribution, is as follows: within a circular curl region, positive curl is counterclockwise rotation, negative curl is clockwise rotation, and the central axis of rotation passes through the center of curl.

5. The multidimensional force decoupling method based on marker point tracking according to claim 1, characterized in that, When multiple contact areas exist, through the displacement field divergence field Separate multiple contact areas and independently apply the multidimensional force decoupling method as described in claim 1 to each contact area.

6. The multidimensional force decoupling method based on marker point tracking according to any one of claims 1-5, characterized in that, The measurement of normal force, shear force, and torsional force has self-decoupling characteristics.

Images