Haptic perception devices and methods, force field inversion modeling, and load measurement systems and methods

Abstract

The application discloses a kind of tactile perception device and method, force field inversion modeling and load measurement system and method, which comprises: force magnetic conversion unit and magnetic-optical conversion unit;Force magnetic conversion unit is used to occur mechanical deformation under the action of external load, and converts mechanical deformation into spatial magnetic field change;Magnetic-optical conversion unit is used to generate optical information for characterizing load in response to the spatial magnetic field change.Using the application scheme, the device integration complexity can be reduced, and the accuracy of tactile perception and the adaptability to the measurement environment are improved.

Description

Technical Field

[0001] This invention relates to the field of tactile sensing, specifically to a tactile sensing device and method, as well as a force field inversion modeling system and method, and a load measurement system and method. Background Technology

[0002] Tactile perception is a crucial means for robots to acquire surface morphology, local stiffness, friction state, force direction, and contact stability when interacting with their environment. To achieve high-precision, full-field, and long-term stable tactile measurements, current research typically follows three technical routes: electrical measurement, visual readout, or magnetic field measurement. Piezoresistive and capacitive sensors offer advantages such as fast response, mature devices, and ease of arraying, but are susceptible to noise, drift, and crosstalk. Pure vision-based tactile sensors can provide rich full-field image information, but often rely on transparent elastomers, marker layers, and stable optical paths, requiring high standards for long-term packaging and usage conditions. Magnetotactile sensors exhibit good environmental robustness, but using high-density magnetic sensor arrays for full-field perception leads to complex wiring, difficult packaging, and increased costs.

[0003] Most existing multimodal sensing solutions employ a backend approach of superimposing and fusing multiple types of sensor signals. For example, a sparse Hall array works in conjunction with an image acquisition module, using a Hall sensor to provide a local absolute magnetic field reference, and then machine learning methods are used to reconstruct the three-dimensional force and displacement fields. While this approach has strong backend expressive capabilities, the key constraints for system implementation still lie in multimodal acquisition and data fusion. This requires the introduction of electronic arrays, leading to a complex system structure and making it difficult to simultaneously achieve accuracy, full-field performance, and stability. Summary of the Invention

[0004] One aspect of this invention provides a tactile sensing device and method to reduce device integration complexity, improve the accuracy of tactile sensing, and enhance adaptability to the measurement environment.

[0005] Another aspect of the present invention provides a force field inversion modeling system and method based on a tactile sensing device, and a load measurement system and method, which combine the environmental robustness of magnetic sensing and the information density advantages of visual readout, thereby improving the load inversion accuracy and the robustness of load prediction results.

[0006] On one hand, embodiments of the present invention provide a tactile sensing device, the device comprising: a force-magnetic conversion unit and a magneto-optical conversion unit; The force-magnetic conversion unit is used to undergo mechanical deformation under external load and convert the mechanical deformation into changes in the spatial magnetic field. The magneto-optical conversion unit is used to generate optical information characterizing the load in response to changes in the spatial magnetic field.

[0007] Optionally, the force-magnetic conversion unit and the magneto-optical conversion unit are arranged adjacent to each other, stacked, or spaced apart.

[0008] Optionally, the load includes: normal force, tangential force, or the resultant force of normal force and tangential force; the mechanical deformation includes any one or more of the following: compression, bending, shearing, tension; the spatial magnetic field change includes any one or more of the following changes: intensity, direction, gradient, periodic texture, distribution pattern.

[0009] Optionally, the force-magnetic conversion unit includes a magnetic flexible body.

[0010] Optionally, the magnetic flexible body is a single-layer structure or a multi-layer composite structure; the single-layer structure is any one of the following: magnetic powder doped flexible magnetic film; magnetic array flexible body; the multi-layer composite structure includes at least one magnetic layer and one flexible layer.

[0011] Optionally, the optical information includes any one or more of the following: color, brightness, texture, stripes, diffraction pattern, polarization pattern, reflection spectrum, transmission spectrum, luminous intensity, luminous wavelength, and display pattern.

[0012] Optionally, the magneto-optical conversion unit includes a magnetic field-responsive optical imaging layer.

[0013] Optionally, the material or structure of the magnetic field-responsive optical imaging layer is any one of the following: magnetically responsive photonic crystal, structural color material containing superparamagnetic particles, magnetically responsive liquid crystal material, magneto-optical thin film, magnetically responsive fluorescent material, or perovskite quantum dot material.

[0014] On the other hand, embodiments of the present invention also provide a tactile sensing method, the method comprising: To acquire information about the spatial magnetic field generated by mechanical deformation caused by external loads; The space magnetic field information is converted into optical information used to characterize the load.

[0015] On the other hand, embodiments of the present invention also provide a force field inversion modeling system based on the tactile sensing device, the system comprising: An optical information acquisition module is used to acquire optical information generated by the tactile sensing device under external load. The load information acquisition module is used to synchronously collect the load data and displacement response data at the corresponding time of the optical information; The dataset construction module is used to associate the optical information, the load data, the displacement response data, and the target area mask label to generate a force-optical dataset. The force-optical dataset includes an input tensor, load data corresponding to the input tensor, displacement response data, and a target area mask label. The target area mask label is used to characterize the spatial location of the optical information corresponding to the contact area or the extended perturbation area containing the contact boundary neighborhood. The training module is used to jointly train the trainable parameters in the neural network-based force field inversion model using the force-light dataset, with the input tensor as input and the corresponding load data, displacement response data and target area mask labels as supervision signals, to obtain the force field inversion model.

[0016] Optionally, the load data and the displacement response data are output synchronously by the mechanical testing machine.

[0017] Optionally, the neural network-based force field inversion model includes: a feature extraction network, a target region localization network, a transformation module, a feature fusion network, and a regression prediction network; The feature extraction network is used to extract image spatial features based on the input tensor; The target region localization network is used to generate a contact region prediction mask based on the difference image in the input tensor, and to determine the local stress region based on the contact region prediction mask. The transformation module is used to perform local fast Fourier transform and / or Gabor filtering in the local stress region to extract magnetically encoded demodulation features from periodic or textured optical images. The feature fusion network is used to fuse the image spatial features, contact area prediction mask features, and magnetic coding demodulation features to obtain fused features; The regression prediction network is used to map the fused features into load prediction values ​​and / or displacement response prediction values.

[0018] Optionally, the optical information acquisition module is specifically used to acquire optical images generated by the tactile sensing device under external load; the input tensor includes: a reference image when no force is applied, a force image when force is applied, and a difference image characterizing the local changes caused by the force; the input tensor is obtained by stitching the reference image, the force image, and the difference image in the channel dimension, and when the image is an RGB image, each image includes multiple color channels.

[0019] Optionally, the target area positioning network includes a U-Net network; The U-Net network is used to output a contact region prediction mask based on the differential image, and to determine the target region based on the contact region prediction mask.

[0020] Optionally, the total loss used when training the force field inversion model includes load regression loss, displacement response regression loss, and contact region segmentation loss, wherein the contact region segmentation loss is used to characterize the difference between the contact region prediction mask and the target region mask label, and includes binary cross-entropy loss and Dice loss.

[0021] On the other hand, embodiments of the present invention also provide a force field inversion modeling method based on the tactile sensing device, the method comprising: The system collects optical information generated by the tactile sensing device under external load, and simultaneously collects load data and displacement response data at the corresponding time of the optical information, and obtains or generates a mask label for the target area. The optical information, load data, displacement response data, and target area mask labels are correlated to generate a force-optical dataset; Using the force-optical dataset, with the optical information or an input tensor composed of a reference image, a force image, and a difference image as input, and with the corresponding load data, displacement response data, and target region mask labels as supervision signals, the trainable parameters in the neural network-based force field inversion model are jointly trained to establish a mapping relationship from optical information to load and displacement response, and supervise the target region localization network to output a contact region prediction mask. Local fast Fourier transform and / or Gabor filtering are performed within the local force region defined by the contact region prediction mask to extract magnetically encoded demodulation features. The magnetically encoded demodulation features are then fused with image spatial features and contact region prediction mask features and input into a regression prediction network to obtain the force field inversion model.

[0022] On the other hand, embodiments of the present invention also provide a load measurement system based on the tactile sensing device, the system comprising: A force field inversion model trained using the sensory data from the aforementioned tactile sensing device; An optical information acquisition module is used to acquire the optical information generated by the tactile sensing device under external load. The prediction module is used to input the optical information into the force field inversion model and determine the load based on the output of the force field inversion model.

[0023] On the other hand, embodiments of the present invention also provide a load measurement method based on the tactile sensing device, the method comprising: Acquire the optical information generated by the tactile sensing device under external load; The optical information is input into a neural network-based force field inversion model trained by the tactile sensing device, and the load is determined based on the output of the force field inversion model.

[0024] On the other hand, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when run by a computer, executes the steps of the force field inversion modeling method or the steps of the load measurement method.

[0025] The tactile sensing device and method provided in this invention uses a force-magnetic conversion unit and a magneto-optical conversion unit as core structures. The force-magnetic conversion unit undergoes mechanical deformation under external load and converts this deformation into a change in the spatial magnetic field. The magneto-optical conversion unit responds to this change in the spatial magnetic field, generating optical information to characterize the external load. This allows the external load to be sequentially converted into changes in the spatial magnetic field and optical response, thus providing a better reflection of the actual load situation. Using this tactile sensing device, tactile information directly reflecting the external load can be obtained without requiring a Hall array or other electronic magnetic sensitive array as a necessary front end. This not only reduces the difficulty of device integration and packaging thickness but also avoids the limitations imposed by rigid electronic units on flexible tactile interfaces, making it more suitable for thin, lightweight, and flexible configurations.

[0026] Furthermore, the tactile sensing device provided in this embodiment of the invention moves the magnetic field design forward to the magnetomechanical conversion unit body. Through the coordinated design of magnetization method, particle distribution, interlayer combination, and structural geometry, the distinguishability of normal force, tangential force, and composite load is improved from the sensing mechanism level, resulting in a relatively simple structure. Moreover, by utilizing the magneto-optical conversion unit for high-resolution readout across the entire field, it combines the environmental robustness of magnetic sensing with the information density advantages of visual readout, and can be widely applied in fields such as robot tactile sensing, grasping stability assessment, slip detection, and surface property recognition.

[0027] The force field inversion modeling system and method, and load measurement system and method based on the above-mentioned tactile sensing device provided in this invention can reduce the complexity of the sensing acquisition end and multimodal synchronous fusion compared with the scheme that requires the participation of additional electric and magnetic field arrays. Furthermore, the accuracy and robustness of load inversion can be improved by fusing contact area positioning and magnetic coding demodulation features. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a tactile sensing device provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process by which a force-magnetic conversion unit, implemented using a magnetic flexible body in an embodiment of the present invention, converts an external load into a change in a spatial magnetic field. Figure 3This is a flowchart illustrating the preparation process of the magnetic powder-doped flexible magnetic thin film, which serves as the force-magnetic conversion unit in this embodiment of the invention. Figure 4 This is a flowchart illustrating the fabrication process of the flexible magnet array, which serves as the force-magnetic conversion unit, in an embodiment of the present invention. Figure 5 This is a schematic diagram of several typical magnetization methods of the force-magnetic conversion unit in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the process by which a magnetically responsive photonic crystal, serving as a magneto-optical conversion unit, converts a change in magnetic field into a change in optical response in an embodiment of the present invention. Figure 7 This is another structural schematic diagram of the tactile sensing device provided in an embodiment of the present invention; Figure 8 This is a flowchart of a tactile perception method provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a force field inversion modeling system provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of a force field inversion model in an embodiment of the present invention; Figure 11 This is a flowchart of a force field inversion modeling method provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of a load measurement system provided in an embodiment of the present invention; Figure 13 This is a flowchart of a load measurement method provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] To address the problems of existing tactile sensing schemes based on magneto-optical fusion requiring the introduction of electronic arrays and the use of multiple sensors with single functions for electrical measurement, visual readout, or magnetic field measurement, resulting in complex system structure and poor environmental adaptability, this invention provides a tactile sensing device that enables external loads to be sequentially converted into changes in spatial magnetic field and optical response. Optical information can better reflect the actual situation of the load.

[0031] like Figure 1 The diagram shown is a structural schematic of a tactile sensing device provided in an embodiment of the present invention.

[0032] The tactile sensing device 100 includes: a force-magnetic conversion unit 11 and a magneto-optical conversion unit 12. Wherein: The force-magnetic conversion unit 11 is used to undergo mechanical deformation under external load and convert the mechanical deformation into changes in the spatial magnetic field; The magneto-optical conversion unit 12 is used to generate optical information characterizing the load in response to changes in the spatial magnetic field.

[0033] In some embodiments, the force-magnetic conversion unit 11 and the magneto-optical conversion unit 12 may be arranged adjacently, stacked, or spaced apart, and the embodiments of the present invention do not limit this.

[0034] The load may include: normal force, tangential force, or the resultant force of normal force and tangential force; Mechanical deformation can be compression, bending, shearing, tension, or a combination of these.

[0035] The change in the spatial magnetic field can be any one or more of the following: intensity, direction, gradient, periodic texture, or distribution pattern.

[0036] In practical implementation, the force-magnetic conversion unit 11 can be implemented by a magnetic flexible body, and the magneto-optical conversion unit 12 can be represented by a magnetic field-responsive optical imaging layer. It should be understood that the magnetic flexible body and optical imaging layer are only used to illustrate the specific structural forms of the aforementioned force-magnetic conversion unit 11 and magneto-optical conversion unit 12, and do not constitute a limitation on the material, shape, number of layers, assembly method, or optical response mechanism of the force-magnetic conversion unit 11 and magneto-optical conversion unit 12. Any structure capable of generating a change in spatial magnetic field under stress deformation can be used as the force-magnetic conversion unit 11; any material or structure capable of generating a detectable optical response change under the action of a magnetic field can be used as the magneto-optical conversion unit 12.

[0037] like Figure 2 The diagram shown is a schematic representation of the process by which a force-magnetic conversion unit, implemented using a magnetic flexible body in an embodiment of the present invention, converts an external load into a change in a spatial magnetic field.

[0038] When the force-magnetic conversion unit 11 is not subjected to an external load, the surrounding spatial magnetic field is in a preset initial distribution state, such as... Figure 2 As shown in (a), under the action of the normal force Fz, the local area of ​​the force-magnetic conversion unit 11 bends, thereby causing a change in the magnetic field of the neighborhood space, as shown in (a). Figure 2 As shown in (b), under the action of tangential force Fx, the entire force-magnetic conversion unit 11 undergoes shear deformation, thereby causing changes in the magnetic field of the neighborhood space, as shown in Figure (b). Figure 2 As shown in (c). Changes in the magnetic field in the neighborhood space can be changes in magnetic field strength, direction, gradient, or distribution pattern, which are not limited in the embodiments of this application.

[0039] In practice, the magnetic flexible body can be a single-layer structure or a multi-layer composite structure.

[0040] Among them, single-layer structures include, for example, magnetic powder-doped flexible magnetic thin films or flexible bodies with magnet arrays.

[0041] A non-limiting method for preparing magnetic powder-doped flexible magnetic thin films is as follows: Magnetic powder (such as NdFeB, neodymium iron boron) is incorporated into an elastic matrix (such as PDMS, polydimethylsiloxane). After mixing, degassing, casting, and curing, a bendable magnetic composite film is formed. Then, directional magnetization or zoned magnetization is applied by a specific external magnetic field to form a preset magnetic field texture. A specific preparation process is as follows: Figure 3 As shown: First, the PDMS basic components and curing agent are mixed at a mass ratio of 10:1. Then, NdFeB magnetic powder is added and stirred thoroughly to obtain a uniform magnetic composite precursor. After degassing, the precursor is poured into a mold or cast onto a substrate to form a film. After the material has cured, a specific spatially distributed external magnetic field is applied to the film to complete magnetization, forming a preset magnetization direction and initial magnetic field texture. If necessary, additional flexible layers, protective layers, or buffer layers can be laminated above or below the magnetic layer to form a multilayer magnetic flexible body, realizing customized designs with different ranges, stiffnesses, and deformation modes. The magnetization method can be selected according to the target function, such as uniform magnetization, partitioned magnetization, mask-assisted magnetization, or array magnetization, to achieve different purposes such as high-sensitivity normal sensing, directional shear sensing, or local hotspot encoding.

[0042] A non-limiting method for fabricating a flexible magnet array is as follows: Several tiny permanent magnets are embedded in a flexible substrate according to predetermined polarity, position, and spacing to obtain a permanent magnet array structure with locally concentrated and controllable magnetic field distribution. A specific fabrication process is as follows: Figure 4 As shown: First, the PDMS substrate is prepared and the mold is completed. Then, micro permanent magnets are placed in the mold according to the preset positions and polarities. If necessary, positioning grooves, jigs, or templates can be used to ensure that the magnet spacing and orientation are consistent. Then, an elastomer precursor liquid is poured into the mold and cured to form a flexible structure with an internally embedded permanent magnet array. By changing the size, thickness, arrangement period, and polarity direction of the magnets, an initial magnetic field with specific spatial characteristics can be constructed. Furthermore, flexible layers or buffer layers can be superimposed on one or both sides of the permanent magnet array to adjust the overall bending stiffness and local compressive displacement, which is particularly suitable for scenarios requiring strong magnetic field output, high-contrast imaging, and large measurement range.

[0043] In practical design, the structural parameters of magnetic flexible bodies should be comprehensively considered around the goal of "deformation generating significant magnetic field differences." Different structural parameters should be designed according to different application requirements. For example, if applied to the detection of minute normal pressure, a thinner continuous magnetic film can be used, and the vertical field gradient can be increased, causing a significant change in the central field strength due to local indentation. If applied to tangential force and slip recognition, anisotropic magnetization or arrayed magnetic hot spots can be introduced, making the lateral displacement appear as a distinguishable offset trajectory on the image. If both normal and tangential information need to be considered simultaneously, a composite design combining centripetal magnetization and periodic magnetization can be used, reflecting the three-dimensional contact state through image symmetry changes, boundary distortion, and local brightness shifts. If it is necessary to further expand the measurement range, a low-modulus flexible layer, buffer layer, or cavity support structure can be added outside or between the magnetic layers to increase the reversible deformation range, improve the stress buffering process, and prevent the single-layer magnetic body from prematurely entering a local saturation or rigidity-limited state under large loads.

[0044] The multilayer composite structure includes at least one magnetic layer and one flexible layer. Examples of the multilayer composite structure include: a magnetic layer + a flexible layer, a magnetic layer + a flexible layer + a magnetic layer, or a composite stack of flexible layers with different moduli and different magnetization layers, etc. This embodiment of the invention does not limit the specific type of structure.

[0045] Furthermore, additional buffer layers and / or spacer layers can be added to the multilayer composite structure to expand the overall compressible or bendable deformation, adjust the neutral plane position, reduce local stress concentration, and improve the deformation range and measurement range. The combination of multiple magnetic layers facilitates the construction of more complex spatial magnetic field gradients, differential magnetic field responses, and direction-selective encoding, thereby enhancing the ability to distinguish between normal forces, tangential forces, and composite loads. The former is characterized by its thinness, continuity, and good fit, while the latter features high magnetic field strength, clear modes, and high design freedom. The composite multilayer structure combines the advantages of adjustable range, high magnetic field encoding freedom, and adaptability to different packaging requirements.

[0046] In the force-magnetic conversion unit, the design principle of the magnetic flexible body is to improve the distinguishability of load types by designing the magnetic pole direction.

[0047] Figure 5 The diagram shows several typical magnetization methods of the force-magnetic conversion unit in an embodiment of the present invention.

[0048] Reference Figure 5 In this embodiment of the invention, for a single-layer continuous magnetic thin film, the following three magnetic field design ideas can be preferentially adopted: Halbach magnetization (or Halbach array), centripetal magnetization, and periodic alternating magnetization.

[0049] (1) Halbach magnetization (or Halbach array): By setting magnetization directions that rotate sequentially in different regions of the thin film, the magnetic field is concentrated and enhanced on one side of the imaging layer, while a weaker leakage magnetic field is formed on the back side, thereby improving the magneto-optical imaging contrast and single-sided readout efficiency. Its core mechanism is that the magnetic field is "squeezed" to the target side by a specific magnetization direction array, achieving the effect of significantly enhancing the magnetic field on one side and almost canceling it out on the other side.

[0050] By employing orthogonal double-layer sinusoidal magnetization, a triaxial magnetic field characterization with direction selectivity can be constructed beneath the magnetic thin film: Lateral displacement (tangential load): mainly changes the magnetic field ratio and And their spatial distribution. Among them, The magnitude of the magnetic flux density vector. , These are the magnetic induction intensity vectors. exist directional components, , Reflecting the magnetic field in The relative proportion of directions (direction angle information).

[0051] Normal displacement (normal load): primarily changes the total magnetic field strength .

[0052] Based on the above differences, tangential and normal loads can be solved separately, thus achieving front-end self-decoupling of the three-dimensional load. Increasing the magnetization period... It can expand the effective displacement range of self-decoupling and reduce the error of three-dimensional force estimation. Therefore, Halbach magnetization is not only a magnetization method, but also a magnetic field coding strategy that can simultaneously control sensitivity, linear range and decoupling stability through period parameters.

[0053] (2) Centripetal magnetization: The magnetization vector points from the edge to the center, forming a radial magnetic field distribution with centrosymmetric characteristics, thus exhibiting different response laws to normal indentation force and tangential disturbance. Its three-dimensional decoupling mechanism is as follows: In the ideal centrosymmetric magnetization model, the magnetic field components , and With magnetic thin films , , The displacements in different directions have an analytical inverse relationship, from which triaxial displacement decoupling parameters can be constructed. Furthermore, for practically fabricable folded, centrally magnetized thin films, the magnetization of their upper and lower surfaces can be equivalent to a series of toroidal currents, and a spatial magnetic field distribution model can be established based on the Biot-Savart law. Since the main difference between the actual folded magnetization model and the ideal model lies in… Therefore, a linear correction term is introduced. and To reduce the deviation between the ideal model and the actual magnetic field distribution, a model is constructed based on the corrected magnetic field components. , and Decoupling characteristic quantity with dominant sensitivity to directional displacement , and The definition is as follows: ; in, , , These are the magnetic induction intensity vectors. exist , directional components, They represent exist , and The compensation coefficient in the calculation. Transverse shear mainly triggers or Normal pressure mainly excites This allows the normal force and the two orthogonal tangential forces to achieve natural three-dimensional decoupling within the working range.

[0054] Furthermore, the centripetal magnetization structure can extend the effective working distance to over 20 mm and significantly reduce the impact of installation axial deviation on the sensing results.

[0055] (3) Periodic alternating magnetization: By constructing a periodic magnetic field gradient, local compression, stretching and shear deformation are manifested in the imaging image as different stripe migration, patch shift or brightness changes.

[0056] Furthermore, design parameters can be coordinated and controlled. For example, in practical design, the sensitivity of deformation and magnetic field coupling can be controlled by material modulus, film thickness, magnetic particle content, interlayer thickness, magnetization depth, and the bonding state of each layer interface, so that the magnetic field disturbance caused by force is sufficiently significant, while avoiding loss of direction discrimination ability due to local magnetic saturation. In addition, the initial magnetic field strength and spatial gradient can be designed in reverse by combining the optimal operating magnetic field range of the magneto-optical conversion unit 12 used, so as to ensure that the optical image has high contrast, good stability, and repeatable feature evolution.

[0057] For a discrete permanent magnet flexible array, the distribution characteristics of local magnetic field hotspots can be controlled by adjusting the size, position, polarity direction and spacing of the magnets, so as to form a high-contrast, traceable and easy-to-identify pattern code.

[0058] For multi-layered composite structures, additional flexible layers, buffer layers, or gradient modulus layers can be introduced between or outside the magnetic layers to amplify displacement differences caused by normal compression, bending, or shearing, delay local magnetic saturation, improve the force transmission path, and enhance the measurement range, structural durability, and distinguishability of different load modes while maintaining high sensitivity.

[0059] By employing alternating magnetic poles or an array of permanent magnets, each permanent magnet unit can form stable, repeatable, and distinguishable local magnetic field peaks, gradient centers, or periodic textures in space. These features can essentially serve as physical markers in machine learning.

[0060] By combining a designable magnetized front end with machine learning, super-resolution tactile perception at the array unit level can be achieved. Furthermore, based on a self-decoupling model, the displacement of each unit can be further mapped to an array-level three-dimensional force distribution. Therefore, the arrayed permanent magnet not only acts as a magnetic field source but also provides: position priors, locally encoded tags, and traceable feature centers. These features can be used for contact area localization, slip trajectory tracking, local deformation recognition, and multi-point force inversion, thereby reducing the reliance on manual labeling in traditional optical image recognition.

[0061] For multilayer magnetic flexible bodies, additional flexible layers, buffer layers, or gradient modulus layers can be introduced between the magnetic layers to amplify the displacement differences caused by normal compression, bending, and shearing, respectively, so that the sensor can further expand the measurable deformation range and mechanical range while maintaining high sensitivity.

[0062] Figure 5 The magnetization method shown can be with Figure 2The force-magnetic conversion mechanism shown in the figure enables the magnetic flexible body to form a spatial magnetic field change with directionality, hierarchy and codedness under the action of external contact, and finally realizes high-resolution tactile information reading through the magneto-optical conversion mechanism.

[0063] In this embodiment of the invention, the magneto-optical conversion unit 12 can be implemented by a magnetic field-responsive optical imaging layer, specifically using, but not limited to, the following materials or structures: magneto-responsive photonic crystals, structural color materials containing superparamagnetic particles, magneto-responsive liquid crystal materials, magneto-optical thin films, magneto-responsive fluorescent materials, perovskite quantum dot materials, and other materials or structures capable of generating detectable optical response changes under the influence of a magnetic field. Among these, structural color materials containing superparamagnetic microparticles or nanoparticles can be assembled into chain-like, columnar, or ordered periodic structures under the influence of a magnetic field, thereby forming a photonic crystal with a specific particle spacing, reflecting different colors of light through diffraction or structural color effects.

[0064] like Figure 6 The diagram shown is a schematic representation of the process by which a magnetically responsive photonic crystal, serving as a magneto-optical conversion unit, converts a change in magnetic field into a change in optical response in an embodiment of the present invention.

[0065] As the magnetic field strength changes from weak to strong, the arrangement of magnetic particles inside the magnetic response photonic crystal changes, and the color of the reflected light also changes accordingly.

[0066] It should be noted that, in specific implementation, it can be Figure 1 The force-magnetic conversion unit 11 and the magneto-optical conversion unit 12 are integrated into a single housing, serving as a tactile sensor to perform force-magnetic conversion and magneto-optical conversion functions. This tactile sensor can be used in scenarios such as robot end effectors, robotic arm grasping feedback, flexible fingertip sensing, contact state recognition, and complex surface tactile characterization, generating optical information capable of characterizing external tactile information (i.e., external loads). The optical information may include, but is not limited to, any one or more of the following: color, brightness, texture, stripes, diffraction patterns, polarization patterns, reflection spectrum, transmission spectrum, luminous intensity, luminous wavelength, and display patterns.

[0067] In some embodiments, such as Figure 7 The diagram shown is another structural schematic of the tactile sensing device provided in an embodiment of the present invention.

[0068] In this embodiment, the tactile sensing device 100 includes a force-magnetic conversion unit 11, a magneto-optical conversion unit 12, an illumination unit 13, and an image acquisition unit 14 arranged in sequence. For ease of use, the above-mentioned units can be housed within the housing 10.

[0069] The force-magnetic conversion unit 11 undergoes mechanical deformation (such as compression, bending, or shearing) under external load and converts the mechanical deformation into changes in the spatial magnetic field. The magneto-optical conversion unit 12 responds to the change in the spatial magnetic field and generates optical information to characterize the load, such as generating a corresponding optical pattern; The illumination unit 13 is used to provide uniform and stable illumination conditions for the magneto-optical conversion unit 12; The image acquisition unit 14 is used to acquire and output the optical information (such as the display image) generated by the magneto-optical conversion unit 12.

[0070] In a non-limiting embodiment, the magneto-optical conversion unit 12 and the magnetic flexible body, which serves as the force-magnetic conversion unit 11, can be integrated into a single structure, for example, by attaching the magneto-optical conversion unit 12 to one side of the magnetic flexible body. The illumination unit 13 and the image acquisition unit 14 can be fixed to the housing 10.

[0071] Furthermore, for ease of installation, a support unit (not shown) can be provided inside the housing 10 to fix the lighting unit 13 and the image acquisition unit 14 onto the support unit.

[0072] It should be noted that the lighting unit 13 and image acquisition unit 14, which play an auxiliary role, can be selected and set according to the application scenario, and this embodiment of the present invention does not limit them.

[0073] Accordingly, embodiments of the present invention also provide a tactile sensing method, such as... Figure 8 The diagram shown is a flowchart of a tactile sensing method provided in an embodiment of the present invention, which includes the following steps: Step 801: Obtain information on the spatial magnetic field generated by the mechanical deformation caused by the external load.

[0074] Step 802: Convert the space magnetic field information into optical information for characterizing the load.

[0075] The tactile sensing device and method provided in this invention are based on force-magnetic conversion and magneto-optical conversion mechanisms. The force-magnetic conversion unit undergoes compression, bending, shearing, stretching, or local deformation under external normal force, tangential force, or combined load, and converts this mechanical deformation into changes in the intensity, direction, gradient, periodic texture, or distribution pattern of the spatial magnetic field. The magneto-optical conversion unit responds to these changes in the spatial magnetic field and converts them into optical information characterizing the load. This optical information includes, for example, color, brightness, texture, stripes, diffraction patterns, polarization patterns, reflection spectra, transmission spectra, luminescence intensity, luminescence wavelength, or display patterns. Thus, external load information is transformed into collectable and analyzable optical information through a continuous "force-magnetic-optical" conversion path. Using this optical information, force field inversion modeling can be achieved, and subsequently, a learning-based inversion model incorporating prior physical knowledge can be used to achieve efficient and accurate measurement of external loads.

[0076] like Figure 9 The diagram shown is a structural schematic of a force field inversion modeling system based on a tactile sensing device 100 provided in an embodiment of the present invention.

[0077] The force field inversion modeling system 900 includes the following modules: The optical information acquisition module 91 is used to acquire optical information generated by the tactile sensing device 100 under external load. The load information acquisition module 92 is used to synchronously acquire the load data and displacement response data corresponding to the optical information at the same time. The dataset construction module 93 is used to associate the optical information, the load data, the displacement response data, and the target area mask label to generate a force-optical dataset; the force-optical dataset includes an input tensor, load data corresponding to the input tensor, displacement response data, and target area mask label; Training module 94 is used to jointly train the trainable parameters in the neural network-based force field inversion model using the force-light dataset, with the input tensor as input and the corresponding load data, displacement response data and target area mask labels as supervision signals, to obtain the force field inversion model.

[0078] In a non-limiting embodiment, the load data and the displacement response data can be output synchronously by a mechanical testing machine. That is, during data acquisition, the mechanical testing machine applies different forces (which can be normal force, tangential force, or the resultant force of normal and tangential forces) to the tactile sensing device 100, and the mechanical testing machine simultaneously outputs load data and displacement response data. On one hand, the load information acquisition module 92 synchronously acquires these load data and displacement response data; on the other hand, the optical information acquisition module 91 synchronously acquires the optical information generated by the tactile sensing device 100.

[0079] In one non-limiting embodiment, the optical information acquisition module 91 can acquire optical images generated by the tactile sensing device 100 under external load.

[0080] In one optional embodiment, the force-optical dataset further includes target region mask labels. These target region mask labels characterize the contact area in an optical image under external load, or characterize an extended perturbation region containing a contact boundary neighborhood; wherein the extended perturbation region can be obtained from a contact region prediction mask through morphological dilation or boundary expansion. The target region mask labels can be generated through manual annotation, automatic annotation based on differential image thresholding, synchronous generation based on indenter geometry and loading position, or projection based on the relationship between the loading trajectory of the mechanical testing machine and image coordinate calibration.

[0081] Based on the time-synchronized optical information, load data, displacement response data, and target area mask labels, a corresponding force-optical dataset can be obtained. Using this force-optical dataset, a force field inversion model can be constructed through a machine learning process.

[0082] In this embodiment of the invention, a force field inversion model based on a neural network can be used. For example... Figure 10 The diagram shown is a structural schematic of a force field inversion model in an embodiment of the present invention.

[0083] The force field inversion model 20 may include: a feature extraction network 21, a target region localization network 22, a transformation module 23, a feature fusion network 24, and a regression prediction network 25. Wherein: Feature extraction network 21 is used to extract spatial features of the image based on the input tensor; The target region localization network 22 is used to output a contact region prediction mask based on the input tensor or difference image, and to determine the local stress region or ROI (Region of Interest) based on the contact region prediction mask. The transformation module 23 is used to perform local fast Fourier transform (FFT) and / or Gabor filtering demodulation in the local stress area to obtain magnetic code demodulation features; Feature fusion network 24 is used to fuse the image spatial features, contact area prediction mask features, and magnetic coding demodulation features to obtain fused features; The regression prediction network 25 is used to map the fused features into load prediction values ​​and / or displacement response prediction values.

[0084] It should be noted that the specific structure of the regression prediction network and the corresponding prediction parameters can be determined according to actual needs, and this embodiment of the invention does not limit this.

[0085] For example, in some embodiments, the regression prediction network may consist of a multilayer perceptron, a fully connected network, or a lightweight convolutional regression head, and the multiple prediction values ​​may include load prediction values ​​and displacement response prediction values.

[0086] In one alternative embodiment, the regression prediction network can further expand the corresponding output branches when local force field labels, simulation labels, slip labels, or height / indentation displacement spatial labels are introduced; when no corresponding labels are introduced, the spatial force field distribution, slip probability, or height field is not considered as a necessary output or necessary supervision item.

[0087] It should be noted that the force field inversion referred to in this paper includes at least the inversion of the overall three-dimensional load components, and can be extended to the inversion of the spatial force field distribution when local force field labels are available. When the load data is the overall load data output by the mechanical testing machine, the supervision signals of the force field inversion model mainly correspond to the overall normal load, overall tangential load, overall displacement response, and target area mask labels. The output of the force field inversion model can be the overall load prediction value and the displacement response prediction value, and the contact area prediction mask is output through the target area positioning network.

[0088] In an alternative embodiment, a lightweight attention enhancement module can also be set between the feature fusion network and the regression prediction network to adjust the fusion weights of different feature channels or different spatial locations, but this module is not a core necessary structure of the force field inversion model.

[0089] Taking the magnetic field-responsive optical imaging layer in the tactile sensing device 100 as an example to realize the magneto-optical conversion unit 12 and acquire the optical image of the magnetic field-responsive optical imaging layer, the process of constructing a force field inversion model through machine learning in the embodiment of the present invention will be described in detail below.

[0090] First, a reference image formed by the magnetic field-responsive optical imaging layer under zero load and a force image formed after applying a load are acquired. A difference image is constructed based on the reference image and the force image, while simultaneously obtaining load data, displacement response data, and target region mask labels. Then, an input tensor feature extraction network composed of the reference image, force image, and difference image is used to obtain image spatial features, and a contact region prediction mask is output through a target region localization network. Within the ROI defined or expanded by the contact region prediction mask, magnetic coding demodulation features are extracted using local FFT and / or Gabor filtering. Finally, the image spatial features, mask features, and magnetic coding demodulation features are fused and input into a regression prediction network. After training, a force field inversion model incorporating physical priors is obtained. This model can output the magnitude, direction, and displacement response of the contact force based on a single frame or consecutive frames of imaging images.

[0091] For example, in a non-limiting embodiment, the input tensor can be formed by concatenating a reference image at zero load, a force image after load application, and a difference image (obtained from the force image and the reference image) along the channel dimension; when the above images are RGB images, each image can contain multiple color channels. Meanwhile, referencing... Figure 10 The feature extraction network 21 can use a convolutional neural network to extract image spatial features, and the target region localization network 22 can use a U-Net network to output a contact region prediction mask based on the differential image, and determine the local stress region accordingly. The transformation module 23 performs local fast Fourier transform (FFT) and / or Gabor filtering demodulation within the local stress region to extract magnetically encoded demodulation features from the periodic or textured image. The magnetically encoded demodulation features can include any one or more of the following: local amplitude, frequency, phase, direction changes, etc. Subsequently, the feature fusion network 24 fuses the image spatial features extracted by the convolutional neural network, the contact region prediction mask features output by the U-Net, and the magnetically encoded demodulation features extracted by the transformation module 23, and the regression prediction network 25 determines the mapping relationship between the fused features and the load and displacement response to obtain a force field inversion model based on the neural network.

[0092] In embodiments employing Halbach magnetization or periodic magnetic encoding structures, this approach reduces background interference from indiscriminate full-map regression, allowing the model to focus on actual contact perturbations, and enhances the ability to distinguish between normal and tangential loads by utilizing the phase, amplitude, and direction information of periodic magnetic encoding. The specific process is as follows: (1) Image input and differential modeling Let the reference image corresponding to the tactile sensing device when it is not under force (i.e., zero load) be... The force diagram after the force is applied (i.e., after the load is applied) is as follows Both can be grayscale images or RGB multi-channel images.

[0093] First, construct the difference image. : ; To reduce illumination drift and camera exposure differences, the difference images can be further processed. Normalization is performed to obtain the normalized difference image. : ; in, To prevent stable terms with excessively small denominators.

[0094] Furthermore, regarding the reference image Force image Difference image after normalization Image registration, illumination normalization, reference field elimination, and scale unification are performed to obtain the input tensor of the convolutional neural network. This input tensor can be obtained by stitching together a reference image, a force image, and a difference image along the channel dimension. When the images are RGB images, each image can contain multiple color channels. This input simultaneously includes the initial magnetic field texture, the current force state, and information on local changes caused by the force.

[0095] (2) U-Net contact area mask localization and extended ROI generation

[0096] To improve the accuracy of mechanical quantity estimation, the target region localization network may include a U-Net network. The U-Net network is used to output a contact region prediction mask based on the differential image, and to determine the target region based on the contact region prediction mask.

[0097] During the training phase, the difference between the contact region prediction mask output by the U-Net network and the target region mask label is used to form the contact region segmentation loss; during the inference phase, the contact region prediction mask is used to determine the local stress region, so that the subsequent FFT and / or Gabor filtering demodulation mainly acts on the stress disturbance region.

[0098] Based on the contact area prediction mask, an extended ROI can be obtained through thresholding, connected component filtering, morphological dilation, or safety boundary expansion.

[0099] The extended ROI retains both the main contact area and the deformation transition information near the contact boundary, which can provide sufficient context for subsequent frequency domain demodulation.

[0100] The convolutional neural network and the U-Net network are trainable neural network modules; the convolutional neural network is used to extract spatial features of the image, and the U-Net network is used to output a contact area prediction mask.

[0101] Local FFT and / or Gabor filtering demodulation is a deterministic physical demodulation module, which itself does not contain network parameters that need to be learned through backpropagation.

[0102] The local amplitude, frequency, phase, and direction changes of the demodulated output of local FFT and / or Gabor filtering are used as magnetic coding demodulation features to participate in subsequent feature fusion.

[0103] (3) Local FFT and / or Gabor filter magnetic coding demodulation

[0104] In embodiments employing orthogonal double-layer Halbach magnetic films, because the orthogonal double-layer Halbach magnetic films form approximately periodic magnetic codes in two directions, the imaging texture can be visualized within the target area. It can be approximated as the superposition of two orthogonal carriers: ; in, Represents pixels The spatial coordinate vector; and These are two sets of orthogonal carrier vectors (i.e., the spatial frequencies of the texture). Dot product; It is the linear dominant term in the phase; , These are the local amplitudes of the first and second carrier groups, respectively; , These are the local phases of the first and second carrier groups, respectively. For background items.

[0105] By applying Gabor filter kernels or local FFT filters matching the aforementioned carrier wave vectors to the image within the target region, the complex response can be obtained: ; ; in, Represents linear convolution; In order to be with the first A complex Gabor filter kernel that is matched to the carrier direction and spatial frequency, or a complex response corresponding to the main frequency component can be obtained by local FFT bandpass filtering; Indicates amplitude response, This indicates the phase response.

[0106] Subtracting the phase and amplitude of the force state from the reference state yields the local magnetic coding perturbation: ; in, and These represent the local amplitudes before and after the force is applied, respectively; wrap(·) is the phase wrapping function, wrapping to (-π,π]; and These represent the local phases before and after the force is applied.

[0107] Under the small displacement approximation, the phase change satisfies: ; in, Indicates the first iOne orthogonal carrier wave vector; This represents a two-dimensional displacement vector field.

[0108] If only the first... Displacement components in each carrier direction Then it can be approximated as: ; in, Indicates the first The modulus of a carrier wave vector.

[0109] Therefore, local amplitude changes usually reflect changes in magnetic field strength and are related to normal compression, while local phase changes usually reflect fringe migration and are related to tangential displacement or shear disturbance.

[0110] Furthermore, the phase gradient can also be calculated. coherence Compared to crosstalk This is used to evaluate the demodulation reliability and the degree of coupling between the two orthogonal directions.

[0111] (4) Image spatial features, mask features and structural prior features

[0112] Optionally, in addition to physical demodulation features, convolutional neural networks can also extract edge, texture transfer, local brightness redistribution, and nonlinear deformation features from images, denoted as image spatial features; the contact area prediction mask and its intermediate layer features output by the U-Net network can be denoted as mask features.

[0113] Optionally, in some embodiments, prior structural features of the tactile sensing device 100 can also be acquired. For example, in a non-limiting embodiment, the prior structural features include a double-layer Halbach magnetic film, a 90° spatial phase difference, a magnetization period, a flexible body thickness, interlayer contact / fixation relationships, and an effective response range of the imaging layer. Utilizing prior information allows the network to not only fit image labels during training but also be constrained by the magnetic encoding direction, period, and interlayer mechanical relationships. This enables the force field inversion model to adapt to various structural types of tactile sensing devices 100, improving the robustness of the force field inversion model.

[0114] (5) Feature fusion

[0115] Image spatial features, contact region prediction mask features, magnetic coding demodulation features, and optional structural prior features are concatenated, weighted, or convolved to obtain fused features. : ; in, This represents the spatial features of the image extracted by the convolutional neural network; This indicates the contact area prediction mask or its features; ; Represents prior structural features; Ψ( ) represents a feature fusion function that incorporates both data-driven complex image representations and interpretable physical quantities from magnetic coding demodulation.

[0116] (6) Fusion feature regression inversion

[0117] The fused features are input into a regression prediction network, which can be composed of a multilayer perceptron, a fully connected network, or a lightweight convolutional regression head, to establish a nonlinear mapping relationship between the fused features and the load and displacement response, and to output three-dimensional load components and / or displacement response.

[0118] The regression prediction network is used to map the fused features into load prediction values ​​and displacement response prediction values. The load prediction values ​​may include a normal force component and two orthogonal tangential force components, and the displacement response prediction values ​​may include normal displacement and / or tangential displacement.

[0119] In one alternative embodiment, the corresponding prediction output can be expanded by further introducing local force field labels, simulation labels, slip labels, or height / indentation displacement spatial labels; without the introduction of such labels, the spatial force field distribution, slip probability, or height field are not required outputs or required supervision items.

[0120] Therefore, the core output of the model can focus on the overall three-dimensional load and displacement response, reducing the reliance on unlabeled spatial prediction tasks.

[0121] (7) Loss function design

[0122] During training, the total loss function of the force field inversion model includes load regression loss, displacement response regression loss, and contact region segmentation loss, which can be expressed as: ; in, , , , Represents the binary cross-entropy loss. Indicates Dice loss; , and These are the weights of the corresponding loss terms.

[0123] in, Represents the true load vector. Represents the load prediction vector. Represents the true displacement response. This represents the predicted displacement response value. Indicates the target area mask label. This indicates the predicted mask for the contact area.

[0124] The contact region segmentation loss includes binary cross-entropy loss and Dice loss, which are used to constrain the difference between the contact region prediction mask and the target region mask label.

[0125] In an alternative embodiment, regularization terms such as phase smoothing and non-contact region constraints can be added under the condition of having explicit physical formulas, calibration parameters or simulation labels, but such regularization terms are not considered as necessary loss terms in this embodiment.

[0126] (8) Training and online reasoning.

[0127] During the training phase, loading force, displacement, and imaging images are simultaneously acquired using a mechanical testing machine to obtain or generate target region mask labels and perform timestamp alignment to form sample pairs. The trainable parameters of the convolutional neural network, U-Net network, and regression prediction network are obtained by minimizing the total loss mentioned above. The local FFT and / or Gabor filtering demodulation modules themselves do not contain network parameters that need to be learned through backpropagation.

[0128] During the inference phase, only real-time imaging images need to be acquired and a differential input constructed with a reference image. This allows for the sequential completion of contact area prediction mask generation, ROI magnetic coding demodulation, feature fusion construction, and regression prediction, ultimately yielding... Predicted values ​​and displacement response predictions.

[0129] It should be noted that, in this embodiment of the invention, the force field inversion includes the inversion of the overall three-dimensional load components, and when a spatial force field label is provided, it can be extended to the spatial force field distribution inversion.

[0130] Accordingly, embodiments of the present invention also provide a force field inversion modeling method based on a tactile sensing device 100, such as... Figure 11 The diagram shown is a flowchart of a force field inversion modeling method provided in an embodiment of the present invention, which includes the following steps: Step 1101: Collect optical information generated by the tactile sensing device under external load, and simultaneously collect load data and displacement response data at the corresponding time of the optical information, and obtain or generate a mask label for the target area.

[0131] Step 1102: Associate the optical information, the load data, the displacement response data, and the target area mask label to generate a force-optical dataset; the force-optical dataset includes the input tensor, the corresponding load data, the displacement response data, and the target area mask label.

[0132] Step 1103: Using the force-light dataset, with the input tensor as input and the corresponding load data, displacement response data and target area mask labels as supervision signals, the trainable parameters in the neural network-based force field inversion model are jointly trained to obtain the force field inversion model.

[0133] The force field inversion modeling system and method based on the above-mentioned tactile sensing device provided in this invention can reduce the complexity of the sensing acquisition end and multimodal synchronous fusion, and improve the model inversion accuracy and robustness by fusing contact area localization and magnetic coding demodulation features.

[0134] Accordingly, embodiments of the present invention also provide a load measurement system based on the above-described tactile sensing device, such as... Figure 12 This is a schematic diagram of the load measurement system.

[0135] The load measurement system 1200 includes: a force field inversion model 20 trained using sensory data from a tactile sensing device 100, an optical information acquisition module 91, and a prediction module 1203. Wherein: Optical information acquisition module 91 is used to acquire optical information generated by the tactile sensing device under external load; The prediction module 1203 is used to input the optical information into the force field inversion model 20 and determine the load based on the output of the force field inversion model.

[0136] Accordingly, embodiments of the present invention also provide a load measurement method based on the above-described tactile sensing device 100, such as... Figure 13 As shown, the load measurement method includes the following steps: Step 1301: Acquire the optical information generated by the tactile sensing device under external load.

[0137] Step 1302: Input the optical information into the force field inversion model based on the neural network obtained by training the sensory data of the tactile sensing device, and determine the load based on the output of the force field inversion model.

[0138] The load measurement system and method based on the above-mentioned tactile sensing device provided in this embodiment of the invention can improve the accuracy and robustness of load prediction results compared with schemes that require additional electromagnetic field arrays.

[0139] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon, the computer program being executable during runtime. Figure 10 or Figure 12The method shown may include some or all of the steps. The storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0140] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data provider to another website, computer, server, or data provider via wired or wireless means.

[0141] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0145] The integrated unit implemented as a software functional unit described above can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute some steps of the methods described in the various embodiments of this application.

[0146] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A force field inversion modeling system based on a tactile sensing device, characterized in that, The tactile sensing device includes: a force-magnetic conversion unit and a magneto-optical conversion unit; the force-magnetic conversion unit is used to undergo mechanical deformation under external load and convert the mechanical deformation into a change in spatial magnetic field; the magneto-optical conversion unit is used to generate optical information characterizing the load in response to the change in spatial magnetic field. The system includes: An optical information acquisition module is used to acquire optical information generated by the tactile sensing device under external load. The load information acquisition module is used to synchronously collect load data and displacement response data of the contact area at the corresponding moment of the optical information; A dataset construction module is used to associate the optical information, the load data, the displacement response data, and the target region mask label to generate a force-optical dataset. The force-optical dataset includes an input tensor, load data and displacement response data corresponding to the input tensor, and a target region mask label. The input tensor includes a reference image when no force is applied, a force image when force is applied, and a difference image representing the local changes caused by the force. The target region mask label is used to represent the spatial location of the optical information corresponding to the contact area or the extended perturbation area containing the contact boundary neighborhood. The training module is used to jointly train the trainable parameters in the neural network-based force field inversion model using the force-light dataset, with the input tensor as input and the corresponding load data, displacement response data and target area mask labels as supervision signals, to obtain the force field inversion model. The neural network-based force field inversion model includes: a feature extraction network, a target region localization network, a transformation module, a feature fusion network, and a regression prediction network; The feature extraction network is used to extract image spatial features based on the input tensor; The target region localization network is used to generate a contact region prediction mask based on the difference image in the input tensor, and to determine the local stress region based on the contact region prediction mask. The transformation module is used to perform local fast Fourier transform and / or Gabor filtering in the local stress region to extract magnetically encoded demodulation features from periodic or textured optical images. The feature fusion network is used to fuse the image spatial features, contact area prediction mask features, and magnetic coding demodulation features to obtain fused features; The regression prediction network is used to map the fused features into three-dimensional load prediction values ​​and / or displacement response prediction values.

2. The force field inversion modeling system according to claim 1, characterized in that, The force-magnetic conversion unit is arranged adjacent to, stacked on top of, or spaced apart from the magneto-optical conversion unit.

3. The force field inversion modeling system according to claim 1, characterized in that: The load includes: normal force, tangential force, or the resultant force of normal force and tangential force; The mechanical deformation includes any one or more of the following: compression, bending, shearing, and tension; The changes in the spatial magnetic field include any one or more of the following: intensity, direction, gradient, periodic texture, and distribution pattern.

4. The force field inversion modeling system according to claim 1, characterized in that, The force-magnetic conversion unit includes a magnetic flexible body.

5. The force field inversion modeling system according to claim 4, characterized in that, The magnetic flexible body is a single-layer structure or a multi-layer composite structure; the single-layer structure is any one of the following: magnetic powder doped flexible magnetic film; magnetic array flexible body; the multi-layer composite structure includes at least one magnetic layer and one flexible layer.

6. The force field inversion modeling system according to claim 1, characterized in that, The optical information includes any one or more of the following: color, brightness, texture, stripes, diffraction pattern, polarization pattern, reflection spectrum, transmission spectrum, luminous intensity, luminous wavelength, and display pattern.

7. The force field inversion modeling system according to claim 1, characterized in that, The magneto-optical conversion unit includes a magnetic field-responsive optical imaging layer.

8. The force field inversion modeling system according to claim 7, characterized in that, The material or structure of the magnetic field-responsive optical imaging layer is any one of the following: magnetically responsive photonic crystal, structural color material containing superparamagnetic particles, magnetically responsive liquid crystal material, magneto-optical thin film, magnetically responsive fluorescent material, or perovskite quantum dot material.

9. The force field inversion modeling system according to claim 1, characterized in that, The load data and the displacement response data are output synchronously by the mechanical testing machine.

10. The force field inversion modeling system according to claim 1, characterized in that, The optical information acquisition module is specifically used to acquire the optical image generated by the tactile sensing device under external load. The input tensor is obtained by concatenating the reference image, the force image, and the difference image along the channel dimension. When the image is an RGB image, each image includes multiple color channels.

11. The force field inversion modeling system according to claim 10, characterized in that, The target area positioning network includes a U-Net network; The U-Net network is used to output a contact region prediction mask based on the differential image, and to determine the target region based on the contact region prediction mask.

12. The force field inversion modeling system according to claim 11, characterized in that, The total loss used when training the force field inversion model includes load regression loss, displacement response regression loss, and contact region segmentation loss. The contact region segmentation loss is used to characterize the difference between the contact region prediction mask and the target region mask label, and includes binary cross-entropy loss and Dice loss.

13. A force field inversion modeling method based on a tactile sensing device, characterized in that, The tactile sensing device includes: a force-magnetic conversion unit and a magneto-optical conversion unit; the force-magnetic conversion unit is used to undergo mechanical deformation under external load and convert the mechanical deformation into a change in spatial magnetic field; the magneto-optical conversion unit is used to generate optical information characterizing the load in response to the change in spatial magnetic field. The method includes: The system collects optical information generated by the tactile sensing device under external load, and simultaneously collects load data and displacement response data at the corresponding time of the optical information, and obtains or generates a mask label for the target area. The optical information, load data, displacement response data, and target area mask labels are correlated to generate a force-optical dataset; Using the force-optical dataset, with the optical information or an input tensor composed of a reference image, a force image, and a difference image as input, and with the corresponding load data, displacement response data, and target region mask labels as supervision signals, the trainable parameters in the neural network-based force field inversion model are jointly trained to establish a mapping relationship from optical information to load and displacement response, and supervise the target region localization network to output a contact region prediction mask. Local fast Fourier transform and / or Gabor filtering are performed within the local force region defined by the contact region prediction mask to extract magnetically encoded demodulation features. The magnetically encoded demodulation features are then fused with image spatial features and contact region prediction mask features and input into a regression prediction network to obtain the force field inversion model.

14. A load measurement system, characterized in that, The system includes: A force field inversion model, which is trained using the force field inversion modeling method based on a tactile sensing device as described in claim 13; An optical information acquisition module is used to acquire the optical information generated by the tactile sensing device under external load. The prediction module is used to input the optical information into the force field inversion model and determine the load based on the output of the force field inversion model.

15. A load measurement method, characterized in that, The method includes: Acquire the optical information generated by the tactile sensing device under external load; The optical information is input into a neural network-based force field inversion model trained using the force field inversion modeling method based on a tactile sensing device as described in claim 13, and the load is determined based on the output of the force field inversion model.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a computer, performs the steps of the force field inversion modeling method of claim 13, or the steps of the load measurement method of claim 15.

Images