Tactile camera and related devices

Abstract

This invention provides a tactile camera and related devices. The tactile camera includes: a camera, a transparent elastomer, a light source, and a reflective layer. The reflective layer covers the near-surface of the transparent elastomer and is used to control the specular peak width and amplitude, separate specular reflection and diffuse reflection, and ensure that the specular / diffuse ratio is no greater than 0.5. The camera is located behind the transparent elastomer and is used to capture optical images reflected by the reflective layer and transmitted through the transparent elastomer. The transparent elastomer is located in front of the light source and is used to generate corresponding elastic deformation based on the surface texture, shape, and contact force of an object when in contact with it. The light source is a multi-light source structure including a co-located light source, used to continuously emit light towards the transparent elastomer. This invention controls the specular peak width and amplitude through the reflective layer, facilitates the separation of specular reflection and diffuse reflection, and improves imaging effects.

Description

Technical Field

[0001] This invention relates to the fields of visual-tactile sensing and computational imaging technology, specifically to a tactile camera and related devices. Background Technology

[0002] Tactile cameras typically estimate surface normals and morphology by placing a reflective layer on the surface / near the surface of a transparent elastomer and acquiring a small number of images under multi-directional illumination. In visual-tactile imaging, surface reflection usually consists of two parts: diffuse reflection (relatively uniform in all directions) and specular reflection (bright spots or bands appearing in a certain direction, i.e., "highlights"). Previously, the specular component was often considered an interference, but when the reflective properties of the reflective layer are properly controlled, the specular component can actually bring practical value: edges and details are more prominent: the specular component is particularly sensitive to slight surface tilts, making boundaries, edges, and fine textures "brighter," facilitating the segmentation of contact areas, boundary location, and identification of minor defects. Light touches are visible: under slight pressure or shallow contact, diffuse reflection changes are often not obvious; specular highlights still show visible changes with contact, helping to stably detect contact and normal changes under low loads. Separating "morphology signals" from "illumination effects": In multi-directional or multi-channel imaging, specular and diffuse reflection exhibit different response trends. Utilizing both types of information together makes it easier to separate the true morphology from brightness fluctuations, resulting in more robust normal / morphology solutions. Reducing the impact of shadows and environmental fluctuations: Specular highlights provide a stable brightness reference, reducing interference from shadows, ambient light, or light source output fluctuations, and improving cross-scene consistency. Facilitating self-inspection and maintenance: The position and width of highlights are sensitive to material conditions and can serve as self-inspection signals for reflective layer aging, contamination, or light source attenuation, facilitating routine maintenance and rapid review. Directly aiding upper-level tasks: In grasping and hand-held operations, highlight movement quickly reflects changes in the contact point's normal and potential slippage direction; in appearance / pressure inspection, small protrusions and scratches are more easily "highlighted." Simple implementation: No complex components are required; by simply selecting and adjusting the reflective layer material and optimizing its surface treatment, a moderate and controllable specular component can be obtained, compatible with existing multi-directional / multi-channel acquisition workflows.

[0003] Existing technologies often struggle to stably separate diffuse and specular components under conditions of few frames when dealing with strong specular response and near-field geometry, leading to reconstruction errors. Summary of the Invention

[0004] The main objective of this invention is to provide a tactile camera and related devices that control the peak width and amplitude of the specular surface through a reflective layer, facilitate the separation of specular reflection and diffuse reflection, and improve imaging performance.

[0005] To achieve the above objectives, one aspect of the present invention provides a tactile camera, comprising: a camera, a transparent elastomer, a light source, and a reflective layer; The reflective layer covers the near-surface of the transparent elastomer and is used to control the specular peak width and amplitude, separate specular reflection and diffuse reflection, and make the specular / diffuse ratio no greater than 0.5. The camera, located behind the transparent elastomer and the light source, is used to capture optical images reflected by the reflective layer and transmitted through the transparent elastomer. The transparent elastomer is located in front of the light source and is used to generate corresponding elastic deformation when it comes into contact with an object, based on the object's surface texture, shape, and the force applied during contact. The light source is a multi-light source structure including a co-located light source, located between the camera and the transparent elastomer, and is used to continuously emit light into the transparent elastomer.

[0006] In some embodiments, the thickness of the reflective layer is 5. 125μm; The roughness window is 0.08. 0.15; With an incident angle of 45°, the mirror peak width is 2°. 14°; The normal reflectivity is not less than 0.6.

[0007] In some embodiments, the light source is a multi-light source structure including a co-located light source.

[0008] In some embodiments, the angle between the emission direction of the co-located light source and the camera's line of sight at the calibration origin is no greater than 5°. The maximum included angle at any point within the field of view is no greater than 8°.

[0009] In some embodiments, the number of other light sources besides the co-located light source is greater than or equal to 2; The other light source is at an angle of 35° to the visual axis. 60°; The angle between the directions of any two light sources is greater than or equal to 60°.

[0010] To achieve the above objectives, another aspect of this invention proposes a tactile image acquisition method, applied to a tactile camera, comprising acquiring tactile images using a time-division multiplexing approach, specifically including: The independent frames of the co-located light source, the independent frames of other light sources, and the dark field frames are obtained by using time-division or color-division acquisition methods. Match individual frames and dark frames according to timestamps; Based on the calibration of the tactile camera, establish the geometric / radiative relationship between near-field illumination and imaging, and perform intensity and geometric normalization for each observation point; For independent frames and dark-field frames at the same observation point, the light source direction is matched according to the timestamp, and the diffuse reflection intensity in different directions is aggregated using a robust algorithm to obtain a baseline map dominated by diffuse reflection. Input the baseline map dominated by diffuse reflection into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

[0011] In some implementations, the method of acquiring independent frames of the co-located light source, independent frames of other light sources, and dark field frames using time-division or color-division acquisition includes: In different time slices, a co-located light source and at least two other light sources are illuminated, or in the same time slice, at least two other light sources with distinguishable wavelengths are illuminated simultaneously.

[0012] To achieve the above objectives, another aspect of the present invention provides a tactile image acquisition system, applied to a tactile image acquisition method, comprising: The image acquisition module is used to acquire independent frames of co-located light sources, independent frames of other light sources, and dark field frames using time-division or color-division acquisition methods. The image matching module is used to match individual frames and dark frames according to their timestamps; The data calibration module is used to establish the geometric / radiative relationship between near-field illumination and imaging based on the calibration of the tactile camera, and to perform intensity and geometric normalization for each observation point; The robust aggregation module is used to match the light source direction by timestamp for independent frames and dark field frames at the same observation point, and aggregate the diffuse reflection intensity in different directions using a robust algorithm to obtain a baseline map dominated by diffuse reflection. The near-field photometric analysis module is used to input the baseline map dominated by diffuse reflection into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

[0013] To achieve the above objectives, another aspect of the present invention provides an electronic device, including: a processor and a memory, wherein the memory is used to store computer program code, the computer program code including computer instructions, and the processor executes the computer program to implement a tactile image acquisition method.

[0014] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described tactile image acquisition method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements a tactile image acquisition method.

[0016] The beneficial effects of the camera of the present invention are as follows: the present invention covers the near surface of the transparent elastomer with a reflective layer, which enables the mirror peak width and amplitude to be controlled, facilitates separation from diffuse reflection, and improves the imaging effect.

[0017] The beneficial effects of the method of the present invention are as follows: The present invention adopts a minimized time-division / color-division acquisition organization, which can select time-division or color-division acquisition methods or use them together according to the requirements of frame rate, energy and complexity, and achieve robust separation and reconstruction without relying on polarization and complex hardware. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a tactile camera structure provided in an embodiment of the present invention; Figure 2 A flowchart of a tactile image acquisition method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the light source direction and conjugate orientation provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of time-division and color-division acquisition organization provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0023] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0024] Existing technologies often struggle to stably separate diffuse and specular components under conditions of few frames when dealing with strong specular response and near-field geometry, leading to reconstruction errors.

[0025] Based on this, the main objective of the present invention is to provide a tactile camera and related devices, which aim to control the peak width and amplitude of the mirror, facilitate the separation of mirror reflection and diffuse reflection, and improve the imaging effect.

[0026] This invention provides a tactile camera and related devices, belonging to the field of visual-tactile sensing and computational imaging. Specifically, it relates to the design of a reflective layer and a multi-source illumination structure for near-field photometric stereo reconstruction, as well as acquisition and reconstruction methods supporting time-division or color-division methods. It is applicable to robot end effectors, wearable tactile sensing, human-computer interaction, and end-effector sensing for endoscopic / minimally invasive instruments. The following embodiments illustrate this further, first describing a tactile camera from one embodiment of the invention.

[0027] Figure 1 This is a schematic diagram of a tactile camera structure provided in an embodiment of the present invention. Please refer to [link / reference]. Figure 1 The present invention provides a tactile camera, comprising: a camera 50, a transparent elastomer 20, a light source and a reflective layer 10. The working distance of the tactile camera is 30-80 mm, which represents the typical distance from the light source / camera to the contact surface being measured.

[0028] The reflective layer 10 covers the near-surface of the transparent elastomer 20 and is used to control the specular peak width and amplitude, separating specular reflection and diffuse reflection.

[0029] It is easy to understand that the reflective layer 10 is located in the area near the surface of the transparent elastomer 20 but not completely exposed to the external environment; the reflective layer 10 is represented as... Figure 1 The section underlined in bold represents a tactile camera, a sensing device that acquires geometric / texture information of the contact area through imaging with a camera. It typically consists of a transparent elastomer 20, a reflective layer 10, a light source, and a camera 50. Furthermore, this invention provides a reflective layer with specific reflective properties, allowing control over the specular peak width and amplitude, and facilitating the separation of specular and diffuse reflection.

[0030] No. The brightness under street lighting is expressed as:

[0031] Diffuse reflection term follows The change is approximately linear, with mirror terms. (such as existing technology) (or micro-surface nuclei) have a very steep curve near the highlight, which is related to derivative Large, therefore for the same small normal perturbation The changes are greater, thus increasing sensitivity ("sensitive" to subtle shape changes), and therefore the sensitivity is even higher when there is a mirror component.

[0032] The camera 50 is located behind the transparent elastomer 20 and the light source, and is used to capture optical images reflected by the reflective layer 10 and transmitted through the transparent elastomer 20.

[0033] It should be noted that in the imaging assembly of the present invention, the camera is arranged behind the transparent elastomer 20 and also behind the light source. Its core function is to capture the optical image reflected by the reflective layer and then transmitted through the transparent elastomer 20. Further, as an optional implementation, the light emitted by the light source is reflected by the target object to the reflective layer 10. The reflective layer 10 reflects the light and couples it into the interior of the transparent elastomer 20. After being transmitted along a preset path within the transparent elastomer 20, the light is finally projected onto the photosensitive element of the camera. The camera acquires and converts the projected optical image to form image data that can be used for subsequent processing. This arrangement effectively utilizes the light transmission characteristics of the transparent elastomer 20 and the directional reflection function of the reflective layer 10, reducing light loss during transmission and ensuring that the image captured by the camera has high clarity and accuracy.

[0034] The transparent elastomer 20 is located in front of the light source and is used to generate corresponding elastic deformation when in contact with an object, based on the object's surface texture, shape, and the force applied during contact.

[0035] It is foreseeable that the transparent elastomer 20 in this invention is arranged in front of the light source, and the transparent elastomer 20 is made of a material with good light transmittance and elasticity. Specifically, when the transparent elastomer 20 comes into contact with the target object, it can generate corresponding elastic deformation according to the texture, geometry, and magnitude of the force applied during the contact process. This elastic deformation can not only conform to the uneven structure of the object's surface, but also reflect the difference in contact force through the change in the degree of deformation. At the same time, due to its good light transmittance, it will not hinder the light emitted by the light source from penetrating and acting on the object's surface, providing a precise morphological basis for subsequent optical image acquisition and analysis.

[0036] The light source is located between the camera 50 and the transparent elastomer 20, and is used to continuously emit light into the transparent elastomer 20.

[0037] Those skilled in the art will understand that the light source in this invention is arranged between the camera 50 and the transparent elastomer 20, and the light source can be a light-emitting element with stable light-emitting characteristics. Specifically, the light source continuously emits light into the transparent elastomer 20; the emitted light can penetrate the transparent elastomer 20 and act on the surface of the target object in contact with the elastomer, providing a stable and sufficient optical basis for the reflection of light by the subsequent reflective layer 10, the transmission of optical signals by the transparent elastomer 20, and the capture of optical images by the camera, ensuring the continuity and uniformity of light throughout the imaging process. In some embodiments, the light source includes several directional light sources, wherein L0 is a co-located light source 40, the emission direction of which is almost coincident with the camera's line of sight (quasi-coaxial), used to provide a stable baseline and geometric reference, at least two other light sources (L1…Lk, k≥2), and at least two other light sources 30, distributed in different incident directions, used to introduce directional differences to achieve mirror / diffuse separation and normal solution. This invention only requires one co-located light source 40 and at least two other light sources 30, without the need for a ring lamp or polarizing element. Specifically, the co-located light source 40 and other light sources 30 of the present invention can achieve synergistic light emission. The co-located light source 40 can ensure the concentration and directionality of light to meet the transmission requirements of the transparent elastomer for incident light. At the same time, the combination and arrangement of multiple light sources can achieve flexible adjustment of the light coverage and light intensity, thereby adapting to the lighting requirements of target objects with different surface textures and shapes. This ensures that the light can penetrate the transparent elastomer evenly and fully and act on the surface of the object, providing stable and reliable lighting conditions for subsequent optical image acquisition.

[0038] Mirrored Neck and This involves strong coupling with parameters such as roughness and Fresnel density. Using linear assumptions for the solution introduces systematic biases, while nonlinear fitting is susceptible to local extrema due to initial values ​​and noise. When the specular highlight is very sharp, often only the path "near the mirror" is significantly bright, while other paths are nearly dark. This makes the solution rely primarily on a very small number of channels; once these channels experience slight saturation, shadowing, or brightness shifts, the estimation will fluctuate significantly. The mirror surface is mainly affected by... Control; Around The mirror image direction may produce similar brightness patterns. Combined with factors such as occlusion / reflection, it's easy for different normal directions to give similar observations, meaning the solution is not unique. The mirror surface steepens the "slope-brightness" relationship, but also makes the equations more nonlinear, overly dependent on a few channels, and introduces symmetry ambiguity, thus reducing accuracy and robustness. Furthermore, when BRDF is nearly isotropic, the mirror term exhibits a "common change" of the same shape across all channels:

[0039] in, Changes with direction. These are common items with the same shape for all channels.

[0040] After normalizing the coefficients through calibration, perform difference or ratio calculations:

[0041] The specular "common term" is canceled out, leaving mainly linear information from diffuse reflection, thus: reducing nonlinear effects: solving for a more stable form closer to linear PS; suppressing channel imbalance: specular reflection no longer dominates a single channel, significantly reducing the impact of saturation / micro-drift on the results; alleviating ambiguity: brightness differences are mainly caused by different The geometric projection determines that the confusion caused by mirror symmetry is reduced. Traditional photometric stereo methods rely on the reflection characteristics of unknown objects, which need to be solved inversely using photometric stereo. Here, however, we pre-design the reflection characteristics of the reflective layer and reduce the calculation error while retaining the high sensitivity of the co-located light source.

[0042] Furthermore, the thickness of the reflective layer 10 is 5 mm. 125μm.

[0043] In some embodiments, the thickness of the reflective layer 10 may also be equivalent to the optical thickness.

[0044] The mirror / diffuse ratio is no greater than 0.5.

[0045] In some embodiments, the mirror / diffuse ratio is the ratio of diffuse reflection weight to specular weight; a smaller ratio indicates that the specular component is more dominant. In the operating wavelength range (e.g., 520±10 nm or 625±10 nm), the mirror / diffuse ratio threshold (… To ensure that the "positive specular residual" can be separated from the non-uniformity and noise of diffuse reflection, the specular / diffuse ratio formula is expressed as:

[0046] Among them, the system is non-uniform. ,redundancy .

[0047] The roughness window is 0.08. 0.15.

[0048] In some embodiments, the roughness window is a scale parameter in the microsurface model; the smaller the value, the narrower the mirror lobe. In the working wavelength range (e.g., 520±10 nm or 625±10 nm), the roughness window ( The formula, derived from the measured mirror flap width, is expressed as:

[0049] Among them, if the difference angle is taken as Measurement, then .

[0050] With an incident angle of 45°, the mirror peak width is 2°. 14°.

[0051] In some embodiments, the specular peak width is the half-peak width of the specular highlight in the angular domain, characterizing the "width / narrowness" of the highlight. In the operating wavelength range (e.g., 520±10 nm or 625±10 nm), derived from the target roughness, the specular peak width formula is expressed as:

[0052] Among them, corresponding The direction is approximately twice that value.

[0053] The normal reflectivity is not less than 0.6.

[0054] In some embodiments, normal reflectance is the reflection intensity or reflectance index under normal (incident angle ≈ 0°) conditions. Lower limit of normal reflectance ( In the operating wavelength range (e.g., 520±10 nm or 625±10 nm), the photon budget is inversely calculated from the minimum SNR required for reconstruction, as expressed by the formula:

[0055] Among them, shot-noise is the preferred method. (If reading noise is significant, use) Take the larger one.

[0056] In this invention, the roughness window and the specular flap width are a pair of mutually computable quantities: first measure the FWHM, then use an approximation to replace it with... (Or vice versa). The mirror / diffuse ratio is determined using a margin where "the components must be clearly distinguishable," with an upper limit roughly... (The system is non-uniform) and safety factor (Give a number like 0.2). The lower bound of normal reflectivity is inversely derived from the photon budget of the required SNR for reconstruction. Smaller (Narrower lobes) make the sensor more sensitive, but the mirror peaks are sharper and more prone to saturation, which also increases the effective range. This changes the mirror / manga ratio requirement and improves... It will simultaneously increase both diffuse reflection and specular reflection. If polarization is used, it will also change the equivalent mirror / diffuse ratio, so they need to be re-evaluated together. / FWHM mainly depends on the coating process and microstructure, wavelength / polarization, and incident observation geometry; the mirror / diffuse ratio is affected by material ratio and interface flatness, channel crosstalk, and system non-uniformity, etc. The reflective layer window of this invention is clearly defined, and with a small number of parameter constraints, the mirror / diffuse components can be distinguished under a few frame conditions.

[0057] Further, refer to Figure 3 The angle between the emission direction of the co-positioned light source 40 and the camera's line of sight at the calibration origin is no greater than 5°.

[0058] The maximum included angle at any point within the field of view is no greater than 8°.

[0059] It is easy to understand that the calibration origin is a reference point defined during geometric / radiative calibration (such as the center of the field of view or a specific point on the reference plane).

[0060] Further, refer to Figure 3 The number of other light sources 30 besides the co-located light source 40 is greater than or equal to 2.

[0061] The angle between the other light source 30 and the visual axis is 35° to 60°.

[0062] The angle between the directions of any two light sources is greater than or equal to 60°.

[0063] It includes at least one pair of light sources conjugate about the visual axis.

[0064] It is easy to understand that, relative to the viewing axis, the two light sources are a pair of directions that are symmetrical about 0° in the plane orientation. The viewing axis is the direction of the central ray of the camera optical system. This invention does not limit the ring structure or fixed trajectory. The light source can be mounted on any bracket / housing, as long as the above-mentioned angle and quantity constraints are met.

[0065] Figure 2 This is a flowchart of a tactile image acquisition method provided in an embodiment of the present invention, with reference to... Figure 2 The present invention also provides a tactile image acquisition method, applied to the above-mentioned tactile camera. The tactile image acquisition method includes, but is not limited to, steps S100 to S500.

[0066] Step S100 uses time-division or color-division acquisition methods to acquire independent frames of co-located light sources, independent frames of other light sources, and dark field frames.

[0067] Optionally, time-division acquisition is a method of sequentially lighting up light sources in different time slots, while color-division acquisition is a method of simultaneously lighting up light sources of different wavelengths in the same time slot and separating the channels by relying on camera multi-channels or filters.

[0068] Step S200: Match independent frames and dark frames according to timestamps.

[0069] Optionally, the timestamp clock domains of independent frames and dark field frames are unified, and precise pairing is achieved by minimizing the time difference and threshold constraints. The specific process is as follows: First, the acquisition timestamps of each frame in the independent frame sequence and the dark field frame sequence are extracted and sorted in ascending order of timestamps; then, for each independent frame to be matched, the timestamp difference is calculated by traversing the dark field frame sequence, and the dark field frame with the smallest absolute difference that does not exceed the preset threshold is selected as the matching frame; if multiple dark field frames meet the threshold condition, the frame with the closest timestamp is selected, and if the threshold is exceeded, it is marked as a matching failure and an exception is triggered; after the matching is completed, the paired independent frames and dark field frames are bound together for subsequent image correction, noise suppression and other processing steps, and the matching results and time difference data are recorded for system accuracy verification.

[0070] Step S300: Establish the geometric / radiative relationship between near-field illumination and imaging based on the tactile camera calibration, and normalize the intensity and geometry of each observation point.

[0071] Furthermore, based on the device calibration, the geometric / radiative relationship between near-field illumination and imaging is established, and the intensity and geometry of each observation are normalized to establish the geometric relationship: the camera intrinsic parameters and the relative position, pointing and reference plane relationship between the camera and the light source are obtained; Establishing Radiation Relationships: Determining the relative response relationships as a function of distance and angle to support normalization. In some embodiments, joint calibration is performed on the imaging module and near-field illumination unit of the haptic camera. Geometrically, a high-precision micro / nano target adapted to the haptic imaging characteristics is used to calibrate the intrinsic and extrinsic parameters of the haptic camera, constructing a geometric model of near-field illumination light propagation and a spatial projection transformation matrix of the haptic sensitive array pixels, clarifying the correspondence between the true spatial coordinates of each observation point on the haptic sensitive surface and the imaging pixels. Radiationally, an integrating sphere combined with a standard radiometric calibration plate is used to calibrate the grayscale response characteristics of the haptic camera under different illumination powers and contact pressures, establishing a radiation transfer function between near-field illumination intensity and the grayscale value of the haptic imaging pixels, quantifying the influence of contact deformation and light source fluctuations on the radiation response. Subsequently, based on the calibration results... The geometric / radiative relationship is normalized for each observation point. Geometric normalization maps observation points under different observation angles and contact postures to a unified standard tactile sensitive surface coordinate system through a projection transformation matrix, eliminating geometric distortion and spatial position errors of observation points caused by device assembly deviations and contact position offsets. Intensity normalization, based on the radiation transfer function, corrects the gray values ​​of each observation point to the equivalent intensity values ​​under standard lighting power and standard contact pressure conditions, eliminating radiation intensity errors caused by light source power drift and inconsistent tactile sensor responses. Finally, through dual normalization of geometry and intensity, geometric projection deviations and radiation response deviations at the system level are effectively eliminated, achieving consistency and accuracy of near-field imaging observation data from the tactile camera.

[0072] Step S400: For independent frames and dark-field frames at the same observation point, match the light source direction according to the timestamp, and use a robust algorithm to aggregate the diffuse reflection intensity in different directions to obtain a baseline map dominated by diffuse reflection.

[0073] Furthermore, using observations from different directions (TD) or different channels (CD), a robust aggregation is used to obtain the baseline response dominated by diffuse reflection; the mirror-dominated observations are constructed with the positive residuals of the baseline. Robust aggregation represents the statistical aggregation (such as minimum, truncated average or equivalent methods) of multi-direction / multi-channel observations to resist anomalies. As an optional implementation, for independent frames and dark-field frames corresponding to the same observation point, the acquisition timestamps and associated light source direction parameters of each frame are extracted. Following the rule of minimizing the timestamp difference and ensuring it does not exceed a preset threshold, precise pairing of independent frames and corresponding dark-field frames is achieved, while ensuring that the light source direction parameters of the paired frames correspond one-to-one. Then, dark-field correction is performed on each pair of paired frames, i.e., the dark current noise and background noise of the corresponding dark-field frame are subtracted from the pixel grayscale values ​​of the independent frames to obtain the noise-removed effective signal frames. Next, pixel regions that conform to diffuse reflection characteristics are selected from the effective signal frames, and interfering pixels such as specular reflection and highlight overflow are removed. Then, based on a robust algorithm, the corrected diffuse reflection intensity values ​​under different light source directions are aggregated. By assigning higher weights to high-confidence data and suppressing outlier interference, the average diffuse reflection intensity of the observation point under multi-directional light sources is obtained. Finally, the aggregated diffuse reflection intensity of all observation points is integrated and normalized to eliminate intensity deviations caused by light source power fluctuations and angle differences, ultimately generating a baseline map dominated by diffuse reflection.

[0074] In step S500, the baseline map dominated by diffuse reflection is input into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

[0075] Furthermore, the observations, after removing the effects of mirrors, are input into a near-field photometric stereo or equivalent algorithm to obtain the surface normal or morphology. Mirror-dominated observations can assist in the determination of boundaries and contact areas. In some embodiments, the baseline map and multiple sets of target images acquired under near-field illumination from different directions constitute the input dataset. The near-field photometric stereo algorithm, based on Lambert's law of reflection and combined with the geometric / radiative relationships obtained from previous device calibration, establishes a system of linear equations between the grayscale values ​​of each pixel under different illumination directions and the surface normal vector. The system of equations is solved using optimization methods such as the least squares method to obtain the surface normal vector corresponding to each pixel. If it is necessary to further generate a 3D morphology map, based on the obtained surface normal data, Poisson reconstruction, integral methods, and other equivalent algorithms are used, combined with calibration parameters such as camera intrinsic and extrinsic parameters, to perform integral calculations on the normal vector field, recovering the three-dimensional height information of the target surface, and finally outputting a high-precision surface normal distribution map or a complete 3D morphology map.

[0076] Furthermore, step S100 also includes step S110.

[0077] Step S110: Illuminate the co-located light source and at least two other light sources in different time slices, or illuminate at least two other light sources with distinguishable wavelengths in the same time slice.

[0078] For example, time-division acquisition involves illuminating L0 and at least two other light sources in different time slices; the minimum organization may include an independent frame of L0, independent frames of the two other light sources, and a dark frame. Color-division acquisition involves simultaneously illuminating at least two other light sources with distinguishable wavelengths in the same time slice, with channel separation achieved by the camera's multi-channel response or external filters; L0 may be acquired as a single frame as an intensity baseline and geometric reference, and then the response mapping of different wavelengths on the camera channels is obtained to achieve channel separability during the processing stage.

[0079] Figure 4 This is a schematic diagram of the time-division and color-division acquisition organization of the present invention, combined with Figure 4 The invention will be described in detail with reference to specific tactile image acquisition scenarios: Fingertip visual-haptic module (TD, minimum frame): Structure: L0 is at an angle of approximately 3° to the line of sight; L1 / L2 are each at an angle of approximately 50° to the line of sight, with azimuths of ±90°; working distance is approximately 45 mm.

[0080] Reflective layer sample: FWHM≈12° , .

[0081] Data acquisition organization: L0, L1, L2, dark field (a third direction can be added for redundancy); Key processing points: Normalize multi-directional observations; obtain diffuse reflection baselines using robust aggregation, construct mirror observations using positive residuals; and implement near-field photometric stereo solution algorithms.

[0082] Surface mount array unit (CD, two-frame high speed): Structure: L0 is at an angle of approximately 4.5° to the line of sight; L1 / L2 are each at an angle of approximately 40° to the line of sight, with azimuths of ±60° between them; working distance is approximately 60 mm.

[0083] Color separation settings: L1 = approximately 525 nm, L2 = approximately 625 nm; the camera uses RGB channels or an external dual bandpass filter for channel separation.

[0084] Reflective layer sample: FWHM≈9° , .

[0085] Acquisition organization: C0=L0; C1=L1+L2. Synchronously illuminate and separate in the camera channel.

[0086] Key processing points: After channel separation, construct and reconstruct diffuse / mirror components according to the process in 7.2.

[0087] Miniature end effector integration (compact optical path): Structure: L0 is quasi-coaxial with the line of sight (angle between origin and prism ≤ 5°) through a hollow lens / beam splitter; L1 and L2 are guided to the sides of the lens by light guides, with an elevation angle of about 55° and an azimuth of ±120°; D≈35–40 mm, for use in confined spaces.

[0088] Reflective layer: Thin reflective layer (thickness 5–10 μm), measured FWHM≈10° .

[0089] Acquisition: TD mode; to improve the rate, duty-phase shifting can be used between L1 and L2.

[0090] Effect: Maintains sparse high-brightness peaks within a narrow field of view, with clear contact boundary positioning.

[0091] Large-area attached tactile pad (extended field of view): Structure: L0 is arranged within a 100×100 mm sensing area (quasi-coaxial emission is achieved through a central optical guide plate); L1 / L2 are installed at the midpoint of the long sides on both sides, with an elevation angle of 45–50° and a near-conjugate azimuth; D≈70–80 mm.

[0092] Reflective layer: A high reflectivity scheme for the window (e.g., metal / dielectric stack), typically FWHM≈8–10° .

[0093] Acquisition: CD mode is preferred (two frames) to meet the high-speed detection requirements of the conveyor line.

[0094] Effect: It can maintain the separation of mirror and diffuse components even at a large working distance, and is suitable for online pressure and appearance inspection.

[0095] High-texture material scenes (enhanced diffuse baseline stability): Structure: L0 is quasi-coaxial; L1 / L2 have an elevation angle of approximately 60° and an azimuth angle of ±150° to increase the difference in incident direction; D≈50 mm.

[0096] Reflective layer: FWHM≈13° The material contains a moderate amount of diffuse reflection but meets the requirements. .

[0097] Acquisition: TD mode; during processing, "truncated averaging" is used for multi-directional observations instead of minimum values ​​to suppress the influence of local textures on the baseline.

[0098] Effect: Achieves smoother normal reconstruction and stable specular positioning on contact surfaces with micro-textures.

[0099] This invention also provides a tactile image acquisition system that can implement the above-described tactile image acquisition method. The tactile image acquisition system includes: The image acquisition module is used to acquire independent frames of co-located light sources, independent frames of other light sources, and dark field frames using time-division or color-division acquisition methods. The image matching module is used to match individual frames and dark frames according to their timestamps; The data calibration module is used to establish the geometric / radiative relationship between near-field illumination and imaging based on the calibration of the tactile camera, and to perform intensity and geometric normalization for each observation point; The robust aggregation module is used to match the light source direction by timestamp for independent frames and dark field frames at the same observation point, and aggregate the diffuse reflection intensity in different directions using a robust algorithm to obtain a baseline map dominated by diffuse reflection. The near-field photometric analysis module is used to input the baseline map dominated by diffuse reflection into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

[0100] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0101] This invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer program code, the computer program code including computer instructions, and the electronic device executes a tactile image acquisition method.

[0102] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned tactile image acquisition method.

[0103] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0104] Another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements a tactile image acquisition method.

[0105] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A tactile camera, characterized in that, include: Camera, transparent elastomer, light source, and reflective layer; The reflective layer covers the near-surface of the transparent elastomer and is used to control the specular peak width and amplitude, separate specular reflection and diffuse reflection, and make the specular / diffuse ratio no greater than 0.

5. The camera, located behind the transparent elastomer, is used to capture optical images reflected by the reflective layer and transmitted through the transparent elastomer; The transparent elastomer is located in front of the light source and is used to generate corresponding elastic deformation when it comes into contact with an object, based on the object's surface texture, shape, and the force applied during contact. The light source is a multi-light source structure including a co-located light source, used to continuously emit light onto the transparent elastomer.

2. A tactile camera according to claim 1, characterized in that, The thickness of the reflective layer is 5. 125μm; The roughness window is 0.

08. 0.15; With an incident angle of 45°, the mirror peak width is 2°. 14°; The normal reflectivity is not less than 0.

6.

3. A tactile camera according to claim 1 as described in claim 2, characterized in that, The angle between the emission direction of the co-position light source and the camera's line of sight at the calibration origin is no greater than 5°. The maximum included angle at any point within the field of view is no greater than 8°.

4. A tactile camera according to claim 1 as described in claim 2, characterized in that, The number of other light sources besides the co-located light source is greater than or equal to 2; The angle between the other light source and the visual axis is 35° to 60°; The angle between the directions of any two light sources is greater than or equal to 60°.

5. A tactile image acquisition method, applied to a tactile camera according to any one of claims 1-4, characterized in that, This includes acquiring tactile images using a time-division multiplexing method, specifically including: The independent frames of the co-located light source, the independent frames of other light sources, and the dark field frames are obtained by using time-division or color-division acquisition methods. Match individual frames and dark frames according to timestamps; Based on the calibration of the tactile camera, establish the geometric / radiative relationship between near-field illumination and imaging, and perform intensity and geometric normalization for each observation point; For independent frames and dark-field frames at the same observation point, the light source direction is matched according to the timestamp, and the diffuse reflection intensity in different directions is aggregated using a robust algorithm to obtain a baseline map dominated by diffuse reflection. Input the baseline map dominated by diffuse reflection into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

6. The tactile image acquisition method according to claim 5, characterized in that, The method of acquiring independent frames of co-located light sources, independent frames of other light sources, and dark field frames using time-division or color-division acquisition includes: In different time slices, a co-located light source and at least two other light sources are illuminated, or in the same time slice, at least two other light sources with distinguishable wavelengths are illuminated simultaneously.

7. A tactile image acquisition system, applied to the tactile image acquisition method according to any one of claims 5-6, characterized in that, include: The image acquisition module is used to acquire independent frames of co-located light sources, independent frames of other light sources, and dark field frames using time-division or color-division acquisition methods. The image matching module is used to match individual frames and dark frames according to their timestamps; The image calibration module is used to establish the geometric / radiative relationship between near-field illumination and imaging based on the calibration of the tactile camera, and to perform intensity and geometric normalization for each observation point; The robust aggregation module is used to match the light source direction by timestamp for independent frames and dark field frames at the same observation point, and aggregate the diffuse reflection intensity in different directions using a robust algorithm to obtain a baseline map dominated by diffuse reflection. The near-field photometric analysis module is used to input the baseline map dominated by diffuse reflection into the near-field photometric stereo or equivalent algorithm to obtain the surface normal or 3D topography map.

8. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein the processor, when executing the computer program, implements a tactile image acquisition method as described in any one of claims 5 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a tactile image acquisition method according to any one of claims 5 to 6.

Images