Noise processing method and device based on tactile sensor, equipment and medium

Abstract

The application discloses a noise processing method and device based on a visual-tactile sensor, equipment and a medium. The method comprises the following steps: determining a noise boundary in at least one target type color channel image of a first image, at least one edge side of an imaging light path of the visual-tactile sensor is respectively configured with a monochromatic light source, each target type color channel is a color channel matched with a light source color of the monochromatic light source configured at each edge side of the imaging light path of the visual-tactile sensor in a plurality of type color channels; performing a mask operation on the target type color channel image of the first image according to the noise boundary in the target type color channel image to obtain a second image; and obtaining a calculation result of the second image by performing image calculation on all color channel images in the second image. The application can realize fast correction of channel noise caused by the asymmetry of the light field space and the signal attenuation effect.

Description

Technical Field

[0001] This invention relates to the field of tactile sensing technology, and in particular to a noise processing method, apparatus, device, and medium based on a visual tactile sensor. Background Technology

[0002] Visor-tactile sensors achieve tactile perception by leveraging the deformation optical response characteristics of soft elastomers. Their core principle involves the soft elastomer deforming upon contact with external stimuli, altering the light propagation state and forming an optical code. This code is collected by a photodetector and converted into a photoelectric signal, which is then decoded by a signal processing unit to extract the deformation information. Due to its combination of flexibility and high sensing accuracy, it has become an important technology in the field of tactile detection. However, the light field propagation stage of vision-tactile sensors suffers from significant technical defects. Low light intensity and small signal amplitude on the weak side of the light field, coupled with poor edge illumination, camera background noise, and excessive attenuation of color channels during video compression, result in an extremely low signal-to-noise ratio on the weak side. To compensate for the high gain applied to the weak side channel, while improving image brightness, readout noise and shot noise are simultaneously amplified. Ultimately, this causes significant noise in the input image on the weak side channel, severely affecting the accurate decoding of deformation information and limiting the sensing accuracy and practical application effectiveness of vision-tactile sensors. Summary of the Invention

[0003] This invention provides a noise processing method, apparatus, device, and medium based on a visual-tactile sensor, to achieve rapid correction of channel noise caused by spatial asymmetry of the light field and signal attenuation effects.

[0004] According to one aspect of the present invention, a noise processing method based on a visual-tactile sensor is provided, the method comprising:

[0005] A noise boundary is determined in at least one target type color channel image of a first image, wherein the first image is an image formed by an imaging optical path of a visual-tactile sensor, and at least one edge side of the imaging optical path of the visual-tactile sensor is respectively configured with a monochromatic light source. Each target type color channel is a color channel among a plurality of type color channels that matches the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-tactile sensor. The noise boundary in each target type color channel image is the boundary of a noise region formed on the target side of each target type color channel image by a target feature. The target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-tactile sensor that matches each target type color channel image. The target side is the opposite side of the edge side of the imaging optical path of the visual-tactile sensor that is configured with a monochromatic light source that matches the target type color channel image.

[0006] Based on the noise boundary in the target type color channel image of the first image, a masking operation is performed on the target type color channel image to obtain the second image;

[0007] The solution result of the second image is obtained by performing image processing on all color channel images in the second image. The solution result of the second image includes the pressure distribution and deformation displacement of the visual-touch sensor due to external contact when the visual-touch sensor acquires the first image.

[0008] According to another aspect of the present invention, a noise processing apparatus based on a visual-tactile sensor is provided, the apparatus comprising:

[0009] A determining module is configured to determine noise boundaries in at least one target type color channel image of a first image, wherein the first image is an image formed by the imaging optical path of a visual-tactile sensor, wherein at least one edge side of the imaging optical path of the visual-tactile sensor is respectively configured with a monochromatic light source, each target type color channel is a color channel among a plurality of type color channels that matches the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-tactile sensor, and the noise boundary in each target type color channel image is the boundary of a noise region formed on the target side of each target type color channel image caused by a target feature, wherein the target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-tactile sensor that matches each target type color channel image, and the target side is the opposite side of the edge side of the imaging optical path of the visual-tactile sensor that matches the target type color channel image;

[0010] The processing module is used to perform a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain a second image;

[0011] The calculation module is used to obtain the calculation result of the second image by performing image calculation on all color channel images in the second image. The calculation result of the second image includes the pressure distribution and deformation displacement of the visual-touch sensor due to external contact when the visual-touch sensor acquires the first image.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the noise processing method based on a visual-tactile sensor as described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the noise processing method based on a visual-touch sensor as described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the noise processing method based on a visual-tactile sensor as described in any embodiment of the present invention.

[0018] In the technical solution of this invention, at least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source. Because the monochromatic light source is arranged on one edge side of the imaging optical path of the visual-touch sensor, during the transmission of light inside the visual-touch sensor, due to the influence of the optical path structure characteristics, asymmetric noise and signal attenuation effects will be concentrated in the region opposite to the light source. That is to say, the target type color channel corresponding to the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-touch sensor will have noise due to the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor. Moreover, the noise region is located on the opposite side of the edge side of the imaging optical path of the visual-touch sensor configured with the monochromatic light source that matches the target type color channel image. Thus, the boundary of the noise region caused by asymmetric noise and signal attenuation effects can be accurately anchored. Based on the noise boundary in the target type color channel image of the first image, the target... Performing a masking operation on the target color channel image allows for masking only the noise region on the opposite side of the monochromatic light source in the target color channel image, rather than masking all color channels of the first image. This eliminates interference from asymmetric noise and signal attenuation effects in the noise region of the target color channel image, while preserving the effective pixel data carrying tactile deformation information in the non-noise regions of the target color channel image to the greatest extent. Image processing of all color channels in the masked second image focuses on the effective pixel data, avoiding interference from invalid data in noise regions. This effectively calculates the pressure distribution and deformation displacement information of the visual-tactile sensor caused by external contact, significantly improving the accuracy and spatial resolution of the calculation results. The output pressure distribution and deformation displacement parameters can truly reflect the actual state of the sensor's external contact, enhancing the practical application performance of the visual-tactile sensor.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of a noise processing method based on a visual-tactile sensor provided in an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating another noise processing method based on a visual-tactile sensor provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of a noise boundary in a target type color channel image provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a noise boundary in a color channel image of another target type provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of a noise processing device based on a visual-tactile sensor provided in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device that implements a noise processing method based on a visual-touch sensor, according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] In one embodiment, Figure 1 This is a schematic flowchart of a noise processing method based on a visual-tactile sensor provided in an embodiment of the present invention. This embodiment is applicable to the case of obtaining distributed force and surface displacement by image processing of images acquired using a visual-tactile sensor. The method can be executed by a noise processing device based on a visual-tactile sensor, which can be implemented in hardware and / or software. The noise processing device based on a visual-tactile sensor can be configured in an electronic device, which can be a visual-tactile sensor loaded with the noise processing method based on a visual-tactile sensor or a robot or intelligent device used in conjunction with a visual-tactile sensor.

[0030] like Figure 1 As shown, the noise processing method based on a visual-tactile sensor may include the following steps:

[0031] S110. Determine the noise boundary in at least one target type color channel image of the first image, wherein the first image is an image formed by the imaging optical path of the visual-touch sensor, and at least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source; wherein, each target type color channel is a color channel among a plurality of type color channels that matches the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-touch sensor, and the noise boundary in each target type color channel image is the boundary of the noise region formed on the target side of each target type color channel image caused by the target feature, the target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor that matches each target type color channel image, and the target side is the opposite side of the edge side configured with the monochromatic light source on the imaging optical path of the visual-touch sensor that matches the target type color channel image.

[0032] The principle of a visual-tactile sensor is that its soft, elastomer sensing layer deforms upon external contact, affecting light propagation and forming optical codes. A photodetector then records this encoded optical information as a photoelectric signal. The signal processing unit then decodes these signals to extract the deformation information. The soft, elastomer sensing layer of the visual-tactile sensor can be a functional component of the sensor's tactile sensing layer, serving as a key medium connecting external physical contact with internal optical signal changes. It can be fabricated using flexible polymer materials with high elasticity and high light transmittance, such as silicone and PDMS (polydimethylsiloxane).

[0033] Because the light intensity is weaker on one edge of the imaging optical path of the visual-tactile sensor, the corresponding optical signal amplitude is lower. Simultaneously, insufficient edge illumination uniformity in the imaging optical path, interference from the camera's inherent background noise, and excessive signal attenuation of the color channel image by the video compression algorithm all contribute to an extremely low signal-to-noise ratio on the weak-light side of the visual-tactile sensor's imaging optical path. To improve the blurring caused by the weak signal on the weak-light side, the camera system typically applies a higher signal gain to the color channel on that side. However, while this gain compensation method increases image brightness and enhances the recognition of weak signals, it simultaneously amplifies the camera's readout noise and shot noise, ultimately causing the color channel on the weak-light side to exhibit significant noise characteristics in the input image.

[0034] During the initialization of the visual-haptic sensor, a reference image I_init is recorded when the sensor is in an unloaded state without tactile perception. When the sensor is running, the real-time image I_realtime and the reference image I_init are differentially analyzed pixel by pixel to obtain I_diff, which is then input into the model to output tactile information. Ideally, if the imaging system of the visual-haptic sensor is completely stable, the real-time image I_empty in the unloaded state should be consistent with the reference image I_init, with all differences being zero, and the tactile output also being zero.

[0035] However, in practice, due to the non-uniformity of the optical and imaging systems of visual haptic sensors, this ideal situation is difficult to achieve. In other words, placing a monochromatic light source on any edge of the imaging optical path of a visual haptic sensor will weaken the light intensity in the opposite edge region of the imaging optical path, ultimately causing the color channel on the weaker side to exhibit significant noise characteristics in the input image. For example, if the red LED light source in the visual haptic sensor is installed on the left side of the imaging optical path, the right side of the real-time image acquired by the sensor will have insufficient red light illumination, resulting in a weak signal. Simultaneously, the illumination attenuation in the edge region of the imaging optical path itself, inherent camera noise, and the strong suppression of color channels by video compression will significantly reduce the signal-to-noise ratio of the right R channel (red channel). Meanwhile, if the blue LED light source in the visual haptic sensor is positioned on the right side of the imaging optical path, the blue light on the left side of the imaging optical path will also be weak. To maintain image brightness, the camera often applies a higher gain to the B channel (blue channel). While this increases signal strength, it also amplifies readout noise and photon shot noise, causing the left B channel to exhibit significant noise in the difference image. These spatially asymmetrical noise and signal attenuation effects result in a systematic deviation between the real-time image I_empty and the reference image I_init, even in a non-contact state where the visual haptic sensor is in an empty state without tactile perception. This causes the difference image I_diff to be non-zero, leading to false tactile responses from the visual haptic sensor. Therefore, it is necessary to identify the noise regions in the color channel image of the first image acquired by the visual haptic sensor.

[0036] The first image can refer to the original multi-color channel image (such as an RGB image) acquired by the visual-tactile sensor during the tactile perception stage, or the optical signal after the soft elastomer sensing layer carrying the visual-tactile sensor deforms due to external contact. The imaging optical path of the visual-tactile sensor can refer to the light propagation path within the sensor, with one end being the light source emitter and the other end being the image acquisition unit (such as a camera), passing through the soft elastomer sensing layer and serving as the channel for light signal transmission and modulation. The monochromatic light source configured in the visual-tactile sensor is not located in the central area of ​​the imaging optical path, but rather installed at at least one edge position (e.g., the left, right, or both sides of the imaging optical path). This edge illumination configuration of the monochromatic light source within the sensor utilizes the scattering and propagation characteristics of light within the soft elastomer to convert the deformation of the elastomer into a detectable change in light signal. The monochromatic light source can refer to a light source emitting a single wavelength of light (such as a red LED or blue LED), rather than a composite white light source. A single wavelength of light corresponds to a fixed color channel in the RGB color space, avoiding interference between different wavelengths and improving the accuracy of signal detection. The target type color channel refers to a specific color channel among multiple color channels that matches the color of the monochromatic light source at the edge of the imaging optical path of the visual-touch sensor. The image acquired by the visual-touch sensor is a multi-color channel image (such as the common RGB three-channel image). The portion selected as the target type color channel has a one-to-one correspondence with the color of the monochromatic light source positioned at the edge of the imaging optical path of the visual-touch sensor. Because the monochromatic light source is located at the edge of the imaging optical path of the visual-touch sensor, light attenuates and scatters unevenly during propagation, forming a fixed noise region on the opposite side of the monochromatic light source. This noise region is only reflected in the color channel that matches the color of the monochromatic light source, thus achieving color channel matching for the noise region.

[0037] For example, if a red LED light source is configured on the left edge of the imaging optical path of the visual-touch sensor, the R channel in the RGB image acquired by the visual-touch sensor is the target type color channel that matches the red LED light source configured on the left edge of the imaging optical path of the visual-touch sensor; if a blue LED light source is configured on the right edge of the imaging optical path of the visual-touch sensor, the B channel in the RGB image is the target type color channel that matches the blue LED light source configured on the left edge of the imaging optical path of the visual-touch sensor. When monochromatic light sources are configured on multiple edge sides of the imaging optical path of the visual-touch sensor (e.g., one monochromatic light source is arranged on each of the left and right sides), multiple target type color channels will be formed, and each target type color channel is independent of the others.

[0038] In each target type color channel image, the noise boundary can be the pixel-level dividing line between the noise region and the effective signal region. The noise region in the target type color channel image is the noise region formed on the target side of each target type color channel image caused by the target features. It is the basis for distinguishing the noise-masking region and the effective tactile signal region in the target type color channel image. The target feature can refer to the monochromatic light source matched with the target type color channel. The asymmetric noise and signal attenuation effect caused by the edge illumination characteristics is the direct physical cause of the noise region. It originates from the energy attenuation and uneven scattering of monochromatic light as it propagates within the imaging optical path, and this attenuation and scattering have obvious directional asymmetry. The target side can refer to the region on the opposite side of the monochromatic light source arrangement edge in the target type color channel image. As the asymmetric noise and signal attenuation effect continuously accumulates during the propagation of light from the light source side to the opposite side, the noise region is concentrated on this side. The above scheme directly relates to the physical causes of noise (such as asymmetric noise and signal attenuation effects), the spatial distribution characteristics of the monochromatic light source on the edge side of the visual-tactile sensor and the noise region generated on the color channel, and fundamentally clarifies the location of the noise boundary. This allows for precise differentiation between inherent hardware-related systematic noise and the pixel region corresponding to the effective tactile signal, significantly improving the accuracy and stability of noise boundary localization. Utilizing the noise boundary ensures that only the high-noise region on the target side is shielded, rather than processing the entire image, maximizing the retention of effective pixel data carrying tactile deformation information within non-noise areas.

[0039] S120. Based on the noise boundary in the target type color channel image of the first image, perform a masking operation on the target type color channel image to obtain the second image.

[0040] The noise boundary in the target type color channel image can refer to the pixel-level boundary between the noise region and the effective signal region within the target type color channel image. The noise region in the target type color channel image can be determined by the asymmetric noise and signal attenuation effect caused by the monochromatic light source configured on the edge side of the imaging optical path of the visual tactile sensor, and is fixed on the opposite side of the monochromatic light source configured on the edge side of the imaging optical path of the visual tactile sensor.

[0041] Masking operations on a target type color channel image can refer to performing image processing operations on a defined noise region based on the noise boundary, setting the pixel values ​​to zero, thus only masking the noise region in the target type color channel image, while keeping the pixel values ​​of the non-noise region in the target type color channel image unchanged.

[0042] The second image can refer to the full-color-channel image obtained after locating the noise boundary of the target color channel in the multiple color channels of the first image and masking the noise region within the noise boundary location of the target color channel. The second image is characterized by retaining the effective pixel data (carrying the optical signal of the soft elastomer sensing layer deformation) of the non-noise regions within the target color channel, and removing pixel values ​​in high-noise regions caused by asymmetric noise and signal attenuation effects. It serves as the input data for image processing by fusing full-channel information.

[0043] The above method only masks the noise-concentrated areas in the target type color channel image, which can eliminate the interference of asymmetric noise and signal attenuation effects on the solution, while preserving the effective pixel data carrying the deformation information of the soft elastic body in the non-noise area to the greatest extent. At the same time, the masking operation is performed independently for different target type color channels (e.g., masking the right noise of the R channel and the left noise of the B channel), and only performing the zeroing operation on the local noise area. Compared with the global filtering algorithm, it significantly reduces the amount of data processing computation, improves the real-time response speed of the sensor, and can make the effective signal areas of each color channel spatially complementary.

[0044] S130. The solution result of the second image is obtained by performing image processing on all color channel images in the second image. The solution result of the second image includes the pressure distribution of the visual-touch sensor due to external contact and the deformation displacement caused by external contact when the visual-touch sensor acquires the first image.

[0045] Image processing can refer to analyzing and calculating the effective pixel data of all color channels in the second image based on a pre-defined algorithm model (such as a neural network model or an optical-mechanical mapping model), integrating the effective tactile signals of the target type color channel with the auxiliary information of the non-target type color channel, and establishing the correspondence between optical pixel features and mechanical parameters, rather than processing a single channel independently. All color channels in the second image can be the full set of color channels contained in the second image, including both the target type color channel matched with the monochromatic light source configured at the edge of the imaging optical path of the visual-tactile sensor, and the non-target type color channels not matched with the monochromatic light source configured at the edge of the imaging optical path of the visual-tactile sensor.

[0046] The solution result of the second image can refer to the pressure distribution and deformation displacement reflecting the physical state of the visual-touch sensor under external contact, output by image processing of all color channels in the second image. The pressure distribution can refer to the magnitude and distribution of pressure at different spatial locations on the surface of the soft elastomer sensing layer of the visual-touch sensor when subjected to external contact; the deformation displacement can refer to the deformation, direction, and displacement value of the soft elastomer sensing layer of the visual-touch sensor due to external contact. Because the first image contains noise signals unrelated to tactile sensation, directly using it for calculation would lead to inaccurate results. By preprocessing the noise areas in the target type color channel image of the first image to remove interference, the signal during image calculation retains only the effective information related to contact deformation, ultimately improving the accuracy of the force distribution and surface displacement calculations.

[0047] The above-mentioned solution introduces full color channel data for collaborative analysis, which can effectively offset the random errors of a single channel (such as ambient light fluctuations and minor sensor noise). Simultaneously, the target type color channel after masking has eliminated inherent hardware-related systematic noise, resulting in a significant improvement in input data quality through this dual optimization. Compared to multi-target channel solution methods, the pressure distribution and deformation displacement results output by this solution have higher spatial resolution, lower numerical errors, and significantly enhanced stability and repeatability under complex working conditions. Moreover, the solution results explicitly include two core parameters: pressure distribution and deformation displacement. These can be directly applied to scenarios such as robot grasping force control, flexible tactile interaction feedback, and precision pressure detection without secondary conversion, significantly improving the efficiency of downstream execution systems and maximizing the practical application of visual-tactile sensors.

[0048] In the technical solution of this invention, at least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source. Because the monochromatic light source is arranged on one edge side of the imaging optical path of the visual-touch sensor, during the transmission of light inside the visual-touch sensor, due to the influence of the optical path structure characteristics, asymmetric noise and signal attenuation effects will be concentrated in the region opposite to the light source. That is to say, the target type color channel corresponding to the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-touch sensor will have noise due to the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor. Moreover, the noise region is located on the opposite side of the edge side of the imaging optical path of the visual-touch sensor configured with the monochromatic light source that matches the target type color channel image. Thus, the boundary of the noise region caused by asymmetric noise and signal attenuation effects can be accurately anchored. Based on the noise boundary in the target type color channel image of the first image, the target... Performing a masking operation on the target color channel image allows for masking only the noise region on the opposite side of the monochromatic light source in the target color channel image, rather than masking all color channels of the first image. This eliminates interference from asymmetric noise and signal attenuation effects in the noise region of the target color channel image, while preserving the effective pixel data carrying tactile deformation information in the non-noise regions of the target color channel image to the greatest extent. Image processing of all color channels in the masked second image focuses on the effective pixel data, avoiding interference from invalid data in noise regions. This effectively calculates the pressure distribution and deformation displacement information of the visual-tactile sensor caused by external contact, significantly improving the accuracy and spatial resolution of the calculation results. The output pressure distribution and deformation displacement parameters can truly reflect the actual state of the sensor's external contact, enhancing the practical application performance of the visual-tactile sensor.

[0049] In one embodiment, Figure 2 This is a flowchart illustrating another noise processing method based on a visual-tactile sensor provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the noise boundary in at least one target type color channel image of the first image in the foregoing embodiments based on the technical solutions of the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0050] like Figure 2 As shown, the noise processing method based on a visual-tactile sensor may include the following steps:

[0051] S210. Obtain multiple reference noise boundaries associated with the visual-touch sensor. The multiple reference noise boundaries are noise boundaries in the different target type color channel images of the images acquired by the visual-touch sensor, which are determined by statistically analyzing the noise boundaries in the different target type color channel images of the multiple third images. The multiple third images are obtained by acquiring images in different types of acquisition scenarios when the visual-touch sensor does not perform tactile perception, and multiple images are acquired in each acquisition scenario.

[0052] Each target type color channel is matched with a monochromatic light source configured on at least one edge side of the imaging optical path of the visual-touch sensor; the noise boundary in each target type color channel image is the boundary of the noise region formed on the target side of each target type color channel image caused by the target features, and the target side is the opposite side of the edge side configured with a monochromatic light source on the imaging optical path of the visual-touch sensor that matches the target type color channel image.

[0053] Multiple third images can refer to multiple images acquired under different acquisition scenarios when the visual-tactile sensor is in a non-tactile sensing state (i.e., the soft elastomer sensing layer of the visual-tactile sensor is not subjected to external contact and has no external force deformation). When the visual-tactile sensor is in a non-tactile sensing state, the soft elastomer sensing layer of the visual-tactile sensor maintains its initial shape, and the images acquired by the visual-tactile sensor do not contain optical signals corresponding to tactile deformation, but only contain inherent hardware noise and scene interference noise.

[0054] Different acquisition scenarios refer to combinations of environmental and operating condition variables that affect image acquisition by visual-tactile sensors. These scenarios are distinguished based on varying ambient light intensities, color temperatures, and sensor operating temperatures. Multiple frames are acquired for each scenario to ensure the comprehensiveness and statistical validity of the data source. Lighting scenarios include low light, medium light, strong light, cool light, and warm light; temperature scenarios include low-temperature, normal-temperature, and high-temperature sensor operation; and environmental interference scenarios include no additional interference, slight electromagnetic interference, and weak vibration interference. The different types of scenarios affect visual-tactile sensor imaging in varying ways and with varying intensities, leading to slight shifts in noise boundaries.

[0055] The target type color channel refers to a color channel type that forms a fixed matching relationship with a monochromatic light source configured on the edge side of the imaging optical path of the visual-touch sensor. The color channel type corresponding to the target type color channel can be uniquely determined by the light source color of the monochromatic light source configured on the edge side of the imaging optical path of the visual-touch sensor (e.g., red light source corresponds to red channel type, blue light source corresponds to blue channel type), and the noise boundary characteristics of the same type color channel are consistent in different acquisition scenarios, so the noise area can be uniformly statistically analyzed based on the type.

[0056] The target feature can refer to a monochromatic light source that matches the target type's color channel. Due to the asymmetric noise and signal attenuation effect caused by the edge illumination characteristics, the energy of the light propagating from the edge of the monochromatic light source to the opposite side decreases nonlinearly with the propagation distance, accompanied by non-uniform scattering of particles within the light path, ultimately forming a stable noise region on the opposite side of the light source. The target side can refer to the region on the opposite side of the target type's color channel image, opposite to the edge of the monochromatic light source. The target side can be determined by the installation position of the single-sided light source on the visual-tactile sensor (e.g., if the monochromatic light source is placed on the left edge of the imaging light path of the visual-tactile sensor, the target side is the right side of the image formed by the imaging light path of the visual-tactile sensor; if the monochromatic light source is placed on the upper edge of the imaging light path of the visual-tactile sensor, the target side is the lower side of the image formed by the imaging light path of the visual-tactile sensor). In the same scene, the target side position of the same target type's color channel has relative stability.

[0057] Reference noise boundaries refer to the set of noise boundaries in different target type color channel images of images acquired by the visual-touch sensor in a non-tactile perception state. This set of noise boundaries is statistically analyzed and forms a baseline template, serving as a reference standard for locating noise boundaries in different target type color channel images of the images acquired by the visual-touch sensor during actual tactile perception. Multiple reference noise boundaries are obtained by identifying noise boundaries in the same target type color channel images of different third images, statistically processing these noise boundaries, and finally obtaining the noise boundaries in the same target type color channel images of the images acquired by the visual-touch sensor. This allows us to obtain the noise boundaries in different target type color channel images of the images acquired by the visual-touch sensor.

[0058] In one embodiment, obtaining multiple reference noise boundaries associated with the visual-touch sensor includes: controlling the visual-touch sensor to continuously acquire multiple frames of images while maintaining a non-tactile perception state under different acquisition scenarios to form a third image under different acquisition scenarios; for the third images under different acquisition scenarios, extracting the noise boundaries caused by asymmetric noise and signal attenuation effects in each target type color channel image according to the target type color channel classification; performing statistical analysis on the noise boundaries of multiple frames under the same target type color channel of the third images in different scenarios (such as calculating the mean and variance of noise boundary coordinates), eliminating random interference in single-frame images, generating noise boundaries in the same target type color channel image of the images acquired by the visual-touch sensor, and so on to obtain the noise boundaries in different target type color channel images of the images acquired by the visual-touch sensor.

[0059] The above solution pre-constructs a reference noise boundary library by statistically analyzing multiple scenes and frames of images from a visual-tactile sensor in a non-tactile sensing state. The third image is acquired based on the sensor in a non-tactile sensing state, and does not contain any effective signals caused by external force deformation, but only hardware inherent noise and scene interference noise. The reference noise boundary statistically derived from this completely corresponds to the noise characteristics of non-tactile signals, enabling precise separation of noise signals from tactile deformation signals, avoiding interference from effective signals on noise boundary statistics, ensuring the accuracy of the reference noise boundary, and the statistical process of the reference noise boundary can be pre-configured without occupying the computing resources of the visual-tactile sensor during real-time operation. Even in embedded devices with limited hardware computing power, noise boundary matching and masking operations can be efficiently achieved.

[0060] S220. Determine at least one noise boundary in the target type color channel image of the first image based on multiple reference noise boundaries.

[0061] The first image is an image formed by the imaging optical path of the visual-touch sensor. At least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source. Each target type color channel is a color channel among multiple type color channels that matches the light source color of the monochromatic light source configured at each edge side of the imaging optical path of the visual-touch sensor. The noise boundary in each target type color channel image is the boundary of the noise region formed on the target side of each target type color channel image caused by the target feature. The target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor that matches each target type color channel image. The target side is the opposite side of the edge side configured with the monochromatic light source on the imaging optical path of the visual-touch sensor that matches the target type color channel image.

[0062] In one embodiment, the multiple reference noise boundaries are noise boundaries in the different target type color channel images of the images acquired by the visual tactile sensor, which are formed by merging the noise boundaries in the different target type color channel images of multiple third images. In this way, the same target type color channel image of the images acquired by the visual tactile sensor shares a noise boundary in different acquisition scenarios. Therefore, the noise boundary in each target type color channel image of the first image can be directly extracted from the multiple reference noise boundaries.

[0063] In one embodiment, the multiple reference noise boundaries are formed by dividing the noise boundaries in the color channel images of different target types of multiple third images according to different acquisition scenarios, and then merging the noise boundaries in the same acquisition scenario. In this way, the same target type color channel image of the image acquired by the visual touch sensor is associated with its corresponding noise boundary in different acquisition scenarios. Therefore, the acquisition scenario type of the visual touch sensor when acquiring the first image is first determined, and then the noise boundary that matches the acquisition scenario type of the first image is extracted from the multiple reference noise boundaries according to the acquisition scenario type of the first image, and used as the noise boundary in each target type color channel image of the first image.

[0064] The above solution can directly match the corresponding reference noise boundary based on the current acquisition scene, eliminating the need for complex boundary detection calculations frame by frame. This significantly reduces the real-time computation load of the algorithm, improves the sensor's response speed, and binds the reference noise boundary to different types of acquisition scenes. This adapts to changes in ambient light and temperature, solving the problem of misjudgment of noise boundaries when the scene changes. When the visual-touch sensor is working under complex conditions, it can quickly match the reference noise boundary of the corresponding acquisition scene, ensuring the accuracy of noise area shielding and avoiding the mis-shielding of effective tactile signals or noise residue due to scene changes. This improves the adaptability and operational stability of the visual-touch sensor in multiple scenarios.

[0065] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein the multiple reference noise boundaries are generated in the following manner:

[0066] Step 1a: Determine multiple reference image groups based on multiple third images, and determine multiple fourth images and multiple fifth images in the reference image groups. Each reference image group corresponds to a different acquisition scene involved in the multiple third images. Each image in the same reference image set is an image acquired by the visual-touch sensor in the same type of acquisition scene. The fifth image is the remaining image in the reference image group excluding the multiple fourth images. The number of multiple fourth images is less than the number of multiple fifth images.

[0067] Step 2a: Perform pixel-by-pixel difference between the multiple fifth images in the reference image group and the sixth image to obtain the difference images corresponding to the multiple fifth images in the reference image group. The sixth image is determined by adding the pixels of the multiple fourth images in the reference image group pixel by pixel and calculating the average value.

[0068] Step 3a: By analyzing the difference images corresponding to multiple fifth images in different reference image groups, determine multiple reference noise boundaries associated with the visual-touch sensor.

[0069] Multiple reference image groups can refer to image sets formed by classifying multiple third images according to their respective acquisition scenarios. One acquisition scenario corresponds to one reference image group, and all third images within the same reference image group are third images acquired by the visual-tactile sensor under the same type of acquisition scenario. The fourth image in each reference image group can refer to a small subset of images selected from each reference image group, used to calculate the baseline image for that acquisition scenario, denoted as the sixth image. Its number is far less than the number of fifth images in the same reference image group. The fifth image in each reference image group can refer to a larger subset of images remaining after removing the fourth image from each reference image group.

[0070] The sixth image can be obtained by averaging multiple fourth images within the reference image group, pixel by pixel. Dividing the sixth image pixel by pixel from multiple fifth images within the reference image group involves subtracting the pixel values ​​at corresponding positions in the fifth and sixth images. This image processing operation removes stable background signals and extracts difference signals. The difference image corresponding to the fifth image can be obtained by pixel-by-pixel difference between the fifth and sixth images; its pixel values ​​reflect the deviation of the fifth image relative to the sixth image, i.e., the noise characteristics of the acquisition scene.

[0071] In one embodiment, performing pixel-by-pixel subtraction between multiple fifth images in a reference image group and a sixth image to obtain difference images corresponding to the multiple fifth images in the reference image group may include: for each reference image group, adding the pixel values ​​at the same position of all fourth images in each reference image group, and then dividing by the number of fourth images to obtain the sixth image associated with each reference image group; and subtracting each fifth image in each reference group from the sixth image pixel-by-pixel to generate difference images corresponding to each fifth image.

[0072] In one embodiment, determining multiple reference noise boundaries associated with the visual-touch sensor by parsing the difference images corresponding to multiple fifth images in different reference image groups may include: identifying the boundary positions of noise regions in each difference image for all difference images in each reference image group; performing statistical analysis (such as calculating the mean and median) on the noise boundary positions of all difference images within the same reference image group, removing outliers, and obtaining reference noise boundaries in the color channel images of different target types of the images acquired by the visual-touch sensor in the acquisition scenario; and integrating the statistical results of all reference image groups to form reference noise boundaries covering the color channel images of different target types of the images acquired by the visual-touch sensor in different acquisition scenarios.

[0073] For example, a visual-tactile sensor is used to acquire N unloaded third images in M ​​different acquisition scenarios, where I_m_n represents the nth third image acquired in the mth scenario. The average of the five fourth images (I_m_1-I_m_5) in each reference image group is calculated pixel-by-pixel and summed to obtain the sixth image I_m_init. The n-5 fifth images (I_m_6-I_m_n) are then subtracted pixel-by-pixel from the sixth image to construct the sixth image within each reference image group. Zhang's difference image I_diff. Analysis of the noise boundary between the red and blue channels caused by asymmetric noise and signal attenuation effects using difference images from different acquisition scenarios and at different durations. Within each pixel, the left blue channel boundary is labeled lb, and the right red channel boundary is labeled rr.

[0074] In the above scheme, the sixth image, serving as a scene benchmark, eliminates stable background signals in the acquisition scene. Pixel-by-pixel differencing cancels out stable signals in the fifth image, retaining only noise-induced differences, thus achieving accurate noise feature extraction. The sixth image is calculated from the average of multiple fourth images, effectively smoothing random noise in a single frame and making the benchmark more stable. Subsequent differencing operations are based on this stable benchmark, resulting in extracted noise features that better match the inherent noise properties of the hardware in the acquisition scene, avoiding misjudgments of noise boundaries caused by single-frame deviations. Each reference image group independently calculates the sixth image and the differencing image, distinguishing and storing noise features from different acquisition scenes, providing a precise basis for subsequent scene-adaptive reference noise boundary generation.

[0075] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments. Determining multiple reference noise boundaries associated with the visual-touch sensor by analyzing the difference images corresponding to multiple fifth images in different reference image groups may include the following steps:

[0076] Step 1b: Determine the first difference image associated with the reference image group from the difference images corresponding to the multiple fifth images in the reference image group, and extract the pixel value of the target type color channel from the first difference image associated with the reference image group as the reference pixel value. The first difference image associated with the reference image group is at least one difference image selected from the difference images corresponding to the multiple fifth images in the reference image group.

[0077] Step 2b: Determine multiple second difference images associated with the reference image group from the difference images corresponding to the multiple fifth images in the reference image group, and divide the pixel values ​​of the multiple second difference images associated with the reference image group in the target type color channel by the reference pixel value, and then calculate the average value of the division result pixel by pixel to obtain the difference processed image in the target type color channel associated with the reference image group. The multiple second difference images are the remaining difference images other than the first difference image in the difference images corresponding to the multiple fifth images in the reference image group.

[0078] The first difference image associated with the reference image group can be at least one representative difference image selected from multiple difference images in the reference image group. It serves as a reference sample for extracting baseline pixel values. The first difference image can be determined based on at least one indicator, such as pixel value stability and noise distribution typicality, in the difference images associated with the reference image group. The baseline pixel value can be a reference value extracted from the target type color channel of the first difference image, representing the typical noise pixel level in the target type color channel image.

[0079] The above scheme uses pixel value division to offset the fluctuations in pixel values ​​between different second difference images, ensuring that the noise features of each image are on the same contrast dimension, thus avoiding interference from accidental fluctuations in a single frame image with the overall statistical results. The process of averaging pixel by pixel can offset random interference noise (such as instantaneous ambient light fluctuations) in the difference images, highlighting the systematic and regular noise features caused by asymmetric noise and signal attenuation effects, making the distribution contours of noise regions clearer. The difference-processed image under the target type color channel associated with the reference image group can superimpose the noise features of multiple second difference images into one image, obtaining the typical noise distribution in different target type color channel images of the images acquired by the visual-tactile sensor in this acquisition scenario. This eliminates the need to process massive amounts of difference images one by one, significantly improving the efficiency of noise boundary extraction.

[0080] Step 3b: Based on the differentially processed images of the target type color channel associated with each reference image group, determine the pixel positions corresponding to the noise boundaries in the target type color channel image of the image acquired by the visual haptic sensor, so as to obtain multiple reference noise boundaries associated with the visual haptic sensor; the differentially processed image indicators of the target type color channel associated with the reference image group are used to indicate the noise areas in the target type color channel image of the image acquired by the visual haptic sensor where the colors exhibit irregular and uneven random changes.

[0081] The differentially processed image of the target type color channel associated with each reference image group can indicate the noise region in the target type color channel image of the visual-tactile sensor under the corresponding acquisition scene type of each reference image group, where the color exhibits irregular and uneven random variations. The irregular and uneven random variations in the noise region can refer to areas in the differentially processed image where pixel values ​​fluctuate randomly and lack fixed texture features. This region is caused by asymmetric noise and signal attenuation effects induced by edge illumination from a monochromatic light source, which is significantly different from the regular pixel changes produced by the tactile deformation of the visual-tactile sensor. Multiple reference noise boundaries can be obtained by integrating the differentially processed image analysis results of all reference image groups, covering the reference noise boundaries in different target type color channel images of the visual-tactile sensor under different acquisition scenes. For example, the analysis... The difference images corresponding to the fifth images respectively, The difference images corresponding to the fifth images are used as the first difference image, and the pixel values ​​in the R channel of the first difference image are extracted and marked as R_i. R_1 is used as the reference pixel value, and the remaining... The difference image is used as the second difference image, and the pixels in the R channel of the second difference image are analyzed. Divide each pixel by the baseline pixel value. Since the difference map mostly consists of regions with a pixel value of 0, we need to take max(x, 1e-8) for each pixel. Then, The mean of the pixels after phase division is calculated and labeled as the difference-processed image R_diff_mean of the target type color channel associated with the reference image group. For example... Figure 3 As shown, the right side of the image is marked with the colors corresponding to different values. Except for the right side, the image is mostly filled with blue, which is the value of 1. The R channel of the empty image remains largely consistent within this region. However, a region with severe color jitter can be clearly observed on the right side of the image. This region is caused by spatially asymmetrical noise and signal attenuation effects. The right boundary (pixel position) of this region is marked as rr. Similarly, Figure 4 express The perturbation of the B channel of an empty image is used to mark the left boundary (pixel position) of that region as lb.

[0082] In the above scheme, the images after differential processing under the target type color channel associated with each reference image group can accurately identify the noise region caused by asymmetric noise and signal attenuation effect, avoiding misjudging the effective signal region as noise. Furthermore, the reference noise boundary is determined in the form of pixel position, which can be directly connected to the algorithm logic of mask operation without additional feature conversion steps, thus improving the engineering practicality of the technical solution and the efficiency of algorithm integration.

[0083] S230. Based on the noise boundary in the target type color channel image of the first image, perform a masking operation on the target type color channel image to obtain the second image.

[0084] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments. The process of obtaining a second image by performing a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image may include the following steps:

[0085] For each target type color channel image in the first image, based on the noise boundary in the target type color channel image of the first image, a first image region and a second image region are determined from the target type color channel image. The first image region is the noise region formed on the target side of the target type color channel image caused by the target features, and the second image region is the remaining image region in the target type color channel image excluding the first image region. The pixel values ​​of the first image region are set to zero and the pixel values ​​of the second image region are kept unchanged to obtain the second image.

[0086] For the RGB color channels of the first image, the red (R) and blue (B) channels, which match the edge monochromatic light source of the imaging optical path of the visual-tactile sensor, are masked. The green channel (G channel), being a non-target color channel, is not additionally masked. In the R channel, rr is pre-determined as the pixel coordinate of the noise boundary. This noise boundary is determined by the reference noise boundary obtained from previous statistics, corresponding to the noise region formed by the asymmetric noise and signal attenuation effect caused by the red light source. Subsequently, the pixel values ​​of the entire region in the R channel image, from rr to img_width (i.e., the total image width), are set to zero to completely mask all noise interference signals within this region.

[0087] Similarly, in the B channel, lb is used as the preset noise boundary pixel coordinate, which corresponds to the noise region distribution range caused by the blue light source. The pixel values ​​of all regions in the B channel image with a width from 0 to lb are set to zero, achieving precise removal of noise signals within this region. Through the above masking operation that distinguishes different color channels, the invalid interference regions in the RGB three-channel image are masked, eliminating pixel interference caused by spatial asymmetry noise and signal attenuation effects, ultimately obtaining dual-color channel data that retains only the effective masked regions (i.e., the effective R and B channel data after noise masking).

[0088] S240. The solution result of the second image is obtained by performing image processing on all color channel images in the second image. The solution result of the second image includes the pressure distribution of the visual-touch sensor due to external contact and the deformation displacement caused by external contact when the visual-touch sensor acquires the first image.

[0089] In one embodiment, obtaining the solution result of the second image by performing image calculation on all color channel images in the second image includes: performing image calculation on all color channel images in the second image using a pre-trained image calculation model to obtain the solution result of the second image. The pre-trained image calculation model is generated by training the image calculation model to be trained based on multiple seventh images. The seventh images are images formed by masking the eighth image used for training by the visual tactile sensor based on the noise boundaries in the color channel images of different target types of images acquired by the visual tactile sensor.

[0090] During the training of the image processing model for a visual-tactile sensor, light field noise severely interferes with the model's recognition and learning of effective tactile signals, leading to decreased accuracy and insufficient robustness in the image processing model, and an inability to accurately reproduce the pressure distribution and deformation displacement caused by external contact with the sensor. To minimize the negative impact of light field noise and improve the training effect and final calculation performance of the image processing model, a targeted masking operation can be performed on the eighth image used for training all input image processing models.

[0091] Performing a masking operation on the eighth image used for training can eliminate pixel interference caused by spatial asymmetry noise in the light field and signal attenuation effects, purifying the effective tactile signals in the eighth image used for training, and providing a data foundation for the subsequent feature learning and calculation functions of the image calculation model. If the masking operation is not performed, the light field noise will seriously interfere with the image calculation model's recognition and learning of the effective tactile signals in the first image, resulting in decreased calculation accuracy and insufficient robustness of the image calculation model. It will be unable to accurately reproduce the pressure distribution and deformation displacement of the visual tactile sensor caused by external contact, and may even lead to problems such as the image calculation model mislearning noise features and distorted calculation results.

[0092] For the RGB color channels of the eighth image used in training, the red (R) and blue (B) channels, which match the monochromatic light source configured at the edge of the imaging optical path of the visual-touch sensor, are masked. The green channel (G channel), being a non-target color channel, is not additionally masked. In the R channel, rr is pre-determined as the noise boundary pixel coordinates. This boundary is determined by the reference noise boundary statistically analyzed in the previous stage, corresponding to the noise region formed by the asymmetric noise and signal attenuation effect caused by the red light source. Subsequently, the pixel values ​​of all regions in the R channel image with a width from rr to img_width (i.e., the total image width) are zeroed out to completely mask all noise interference signals within this region. Similarly, in the B channel, lb is used as the preset noise boundary pixel coordinates. This boundary corresponds to the noise region distribution range caused by the blue light source. The pixel values ​​of all regions in the B channel image with a width from 0 to lb are zeroed out to achieve precise removal of noise signals within this region. Through the above channel-specific masking operation, we successfully masked the invalid interference areas in the RGB three-channel image, eliminated pixel interference caused by light field spatial asymmetry noise and signal attenuation effect, and finally obtained two-color channel data that retains only the effective area of ​​the mask.

[0093] In subsequent model training, only the seventh image, which contains a portion of valid data from both dual-color channels and a portion of valid data from the full color channels after masking, is used as the training input, instead of the original eighth image's full RGB three-channel data. This approach guides the model to focus on learning the features of effective tactile signals, avoiding the model's mislearning of noise signal features and ensuring the targeted and effective training of the image processing model. Simultaneously, by combining the input of a portion of valid dual-color channel data with a portion of valid data from the full color channels, the advantages of multi-channel complementary processing can be fully utilized, enabling the model to efficiently complete dual-color processing within the effective masked area. This accurately establishes the mapping relationship between optical pixel signals and tactile mechanical parameters (pressure distribution, deformation displacement), ultimately improving the model's processing accuracy, stability, and environmental robustness.

[0094] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments. Before performing a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain the second image, the following steps are further included:

[0095] Additive Gaussian noise is applied to the target type color channel image of the first image to simulate the light field noise around the noise boundary in the target type color channel image.

[0096] The noise boundaries (B channel boundary lb, R channel boundary rr) in the target type color channels (i.e., R channels and B channels matched with the monochromatic light source at the edge of the imaging optical path) of the image acquired by the visual-tactile sensor are determined by... This is obtained through statistical analysis of the differential images. Due to the limitations of the statistical samples and the randomness of the light field noise, this statistical method is prone to situations where some actual noise exceeds the preset boundaries lb and rr, resulting in residual light field noise near the boundaries that has not been completely shielded, thus interfering with the accuracy of subsequent masking operations and the model solution effect.

[0097] To correct residual noise near noise boundaries and improve the model's adaptability to noise near boundaries, additive Gaussian noise data augmentation can be applied to the target type color channel image of the first image to simulate the light field noise around the noise boundaries in the target type color channel image. This achieves accurate simulation of the real light field noise around the noise boundaries (such as R channel rr, B channel lb) in the target type color channel image, especially the residual noise that exceeds the statistical boundaries lb and rr, thereby achieving the correction and simulation of noise near the boundaries.

[0098] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein data enhancement of the target type color channel image of the first image by additive Gaussian noise may include the following steps:

[0099] Additive Gaussian noise data enhancement is performed within a reference region in the target type color channel image of the first image. The reference region is a region formed by extending a reference width on both sides of the noise boundary in the target type color channel image, with the noise boundary as the center. The reference width is determined based on the width of the noise region in the target type color channel image, and the reference width is smaller than the width of the noise region in the target type color channel image.

[0100] The reference region can be a specific area centered on the noise boundary of the target type color channel image, extending a reference width to both sides of the boundary. This represents the directional range for additive Gaussian noise addition. The definition of the reference region focuses on the highly interfering area near the noise boundary while limiting the effective signal area through the reference width. The reference width can be the width extending to both sides from the noise boundary in the target type color channel image. Its value is determined based on the actual width of the noise region in the target type color channel image, and it must satisfy the condition that the reference width is smaller than the width of the noise region in the target type color channel image. This ensures that the reference region falls entirely within the noise region, preventing noise enhancement operations from covering the effective signal area carrying tactile deformation information.

[0101] Additive Gaussian noise (AGG) data augmentation involves superimposing normally distributed random noise onto pixel values ​​within a reference region of a target-type color channel image. The mean and variance of AGA match the characteristics of light field noise in real-world conditions, accurately simulating random interference near noise boundaries caused by light field spatial asymmetry and signal attenuation. The mean and variance of AGA are determined based on the noise regions in the differentially processed images of each target-type color channel associated with the reference image group. For example, in the noise regions of each target-type color channel associated with the reference image group, the arithmetic mean and variance of all pixel values ​​within the noise regions are calculated. The calculated mean is directly used as the mean m of the AGA, and the calculated variance is directly used as the variance var of the AGA.

[0102] For example, taking at least one target type color channel including B channel and R channel as an example, in the delineation of the noise enhancement region, directional additive Gaussian noise addition operation is performed with the noise boundary lb of B channel and the noise boundary rr of R channel as the central axis. For R channel, a reference width for noise enhancement on R channel can be randomly generated by the formula noise_width=random.uniform(0,(img_width-rr) / 4), where img_width represents the image width of the first image and rr represents the noise boundary in R channel image. Then, additive Gaussian noise is superimposed in the reference region R[:,rr-noise_width:rr+noise_width:0] of R channel image. Similarly, for B channel, a reference width for noise enhancement on B channel can be randomly generated by the formula noise_width=random.uniform(0,lb / 4), where lb represents the noise boundary in B channel image. Additive Gaussian noise with the same characteristics is superimposed in the reference region B[:,lb-noise_width:lb+noise_width:2] of B channel image. The data processing method of directional noise enhancement described above can accurately simulate residual light field noise near the boundary that cannot be completely eliminated by traditional masking operations, thereby realizing online correction of light field noise and effectively improving the adaptability of subsequent models to complex noise under real working conditions.

[0103] In the above scheme, to address the issue that noise boundaries can easily extend beyond the statistical boundaries when statistically analyzing multi-frame difference maps, Gaussian noise is directionally added to a reference region near the boundary. This accurately simulates residual light field noise exceeding the statistical boundaries in real-world conditions, compensating for the bias in the statistical boundaries and making the training data more closely match the noise distribution characteristics of the actual acquisition scenario. The reference width being smaller than the noise region width ensures that the reference region is completely within the noise region. The addition of additive Gaussian noise does not cover the effective signal region carrying tactile deformation information, thus achieving noise enhancement while avoiding damage to the effective signal.

[0104] In the technical solution of this invention, at least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source. Because the monochromatic light source is arranged on one edge side of the imaging optical path of the visual-touch sensor, during the transmission of light inside the visual-touch sensor, due to the influence of the optical path structure characteristics, asymmetric noise and signal attenuation effects will be concentrated in the region opposite to the light source. That is to say, the target type color channel corresponding to the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-touch sensor will have noise due to the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor. Moreover, the noise region is located on the opposite side of the edge side of the imaging optical path of the visual-touch sensor configured with the monochromatic light source that matches the target type color channel image. Thus, the boundary of the noise region caused by asymmetric noise and signal attenuation effects can be accurately anchored. Based on the noise boundary in the target type color channel image of the first image, the target... Performing a masking operation on the target color channel image allows for masking only the noise region on the opposite side of the monochromatic light source in the target color channel image, rather than masking all color channels of the first image. This eliminates interference from asymmetric noise and signal attenuation effects in the noise region of the target color channel image, while preserving the effective pixel data carrying tactile deformation information in the non-noise regions of the target color channel image to the greatest extent. Image processing of all color channels in the masked second image focuses on the effective pixel data, avoiding interference from invalid data in noise regions. This effectively calculates the pressure distribution and deformation displacement information of the visual-tactile sensor caused by external contact, significantly improving the accuracy and spatial resolution of the calculation results. The output pressure distribution and deformation displacement parameters can truly reflect the actual state of the sensor's external contact, enhancing the practical application performance of the visual-tactile sensor.

[0105] In one embodiment, Figure 5This is a schematic diagram of a noise processing device based on a visual-tactile sensor provided in an embodiment of the present invention. This embodiment is applicable to situations where image processing is performed on images acquired using a visual-tactile sensor to obtain distributed force and surface displacement. This method can be executed by a noise processing device based on a visual-tactile sensor, which can be implemented in hardware and / or software. This noise processing device based on a visual-tactile sensor can be configured in an electronic device, which may be a visual-tactile sensor loaded with a noise processing method based on a visual-tactile sensor, or a robot or intelligent device used in conjunction with a visual-tactile sensor.

[0106] like Figure 5 As shown, the noise processing device based on a visual-tactile sensor includes the following:

[0107] The determining module 510 is used to determine noise boundaries in at least one target type color channel image of a first image, wherein the first image is an image formed by the imaging optical path of a visual-tactile sensor, and at least one edge side of the imaging optical path of the visual-tactile sensor is respectively configured with a monochromatic light source, each target type color channel is a color channel among a plurality of type color channels that matches the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-tactile sensor, and the noise boundary in each target type color channel image is the boundary of a noise region formed on the target side of each target type color channel image caused by a target feature, wherein the target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-tactile sensor that matches each target type color channel image, and the target side is the opposite side of the edge side of the imaging optical path of the visual-tactile sensor that matches the target type color channel image;

[0108] Processing module 520 is used to perform a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain a second image;

[0109] The calculation module 530 is used to obtain the calculation result of the second image by performing image calculation on all color channel images in the second image. The calculation result of the second image includes the pressure distribution of the visual-touch sensor due to external contact and the deformation displacement caused by external contact when the visual-touch sensor acquires the first image.

[0110] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein determining the noise boundary in at least one target type color channel image of the first image includes:

[0111] Acquire multiple reference noise boundaries associated with the visual-touch sensor, and determine noise boundaries in at least one target type color channel image of the first image based on the multiple reference noise boundaries;

[0112] The multiple reference noise boundaries are noise boundaries in the color channel images of different target types of the images acquired by the visual-touch sensor, which are determined by statistically analyzing the noise boundaries in the color channel images of different target types of multiple third images. The multiple third images are images acquired in different types of acquisition scenarios when the visual-touch sensor does not perform tactile perception, and multiple images are acquired in each acquisition scenario.

[0113] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein the plurality of reference noise boundaries are generated in the following manner:

[0114] Multiple reference image groups are determined based on the multiple third images, and multiple fourth images and multiple fifth images are determined in the reference image groups. Each reference image group corresponds to a different acquisition scene involved in the multiple third images. Each image in the same reference image set is an image acquired by a visual-touch sensor in the same type of acquisition scene. The fifth image is the remaining image in the reference image group excluding the multiple fourth images. The number of the multiple fourth images is less than the number of the multiple fifth images.

[0115] The difference images corresponding to the fifth images in the reference image group are obtained by performing pixel-by-pixel difference between the sixth image and the multiple fifth images in the reference image group. The sixth image is determined by adding the pixels of the multiple fourth images in the reference image group pixel by pixel and taking the average value.

[0116] By analyzing the difference images corresponding to multiple fifth images in different reference image groups, multiple reference noise boundaries associated with the visual-tactile sensor are determined.

[0117] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein determining multiple reference noise boundaries associated with the visual-touch sensor by parsing the difference images corresponding to multiple fifth images in different reference image groups includes:

[0118] The first difference image associated with the reference image group is determined from the difference images corresponding to the multiple fifth images in the reference image group, and the pixel value of the target type color channel is extracted from the first difference image associated with the reference image group as the reference pixel value. The first difference image associated with the reference image group is at least one difference image selected from the difference images corresponding to the multiple fifth images in the reference image group.

[0119] From the difference images corresponding to the multiple fifth images in the reference image group, we determine the multiple second difference images associated with the reference image group, and divide the pixel values ​​of the multiple second difference images associated with the reference image group in the target type color channel by the reference pixel value, and then calculate the average value of the division result pixel by pixel to obtain the difference-processed image in the target type color channel associated with the reference image group; the multiple second difference images are the remaining difference images other than the first difference image in the difference images corresponding to the multiple fifth images in the reference image group.

[0120] Based on the differentially processed images of the target type color channels associated with each of the reference image groups, the pixel positions corresponding to the noise boundaries in the target type color channel images of the images acquired by the visual-touch sensor are determined to obtain multiple reference noise boundaries associated with the visual-touch sensor; the differentially processed image indicators of the target type color channels associated with the reference image groups are used to indicate noise areas in the target type color channel images of the images acquired by the visual-touch sensor where the colors exhibit irregular and uneven random changes.

[0121] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein, based on the noise boundary in the target type color channel image of the first image, a masking operation is performed on the target type color channel image to obtain a second image, including:

[0122] For each target type color channel image in the first image, based on the noise boundary in the target type color channel image of the first image, a first image region and a second image region are determined from the target type color channel image. The first image region is a noise region formed on the target side of the target type color channel image caused by the target feature, and the second image region is the remaining image region in the target type color channel image other than the first image region.

[0123] The pixel values ​​of the first image region are set to zero, and the pixel values ​​of the second image region are kept unchanged to obtain the second image.

[0124] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein, before performing a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain the second image, the method further includes:

[0125] Additive Gaussian noise is applied to the target type color channel image of the first image to simulate the light field noise around the noise boundary in the target type color channel image.

[0126] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments, wherein performing additive Gaussian noise data enhancement on the target type color channel image of the first image includes:

[0127] Additive Gaussian noise data enhancement is performed in a reference region within the target type color channel image of the first image. The reference region is a region formed by extending a reference width on both sides of the noise boundary in the target type color channel image, with the noise boundary as the center. The reference width is determined based on the width of the noise region in the target type color channel image, and the reference width is smaller than the width of the noise region in the target type color channel image.

[0128] In the technical solution of this invention, at least one edge side of the imaging optical path of the visual-touch sensor is respectively configured with a monochromatic light source. Because the monochromatic light source is arranged on one edge side of the imaging optical path of the visual-touch sensor, during the transmission of light inside the visual-touch sensor, due to the influence of the optical path structure characteristics, asymmetric noise and signal attenuation effects will be concentrated in the region opposite to the light source. That is to say, the target type color channel corresponding to the light source color of the monochromatic light source configured on each edge side of the imaging optical path of the visual-touch sensor will have noise due to the monochromatic light source configured on one edge side of the imaging optical path of the visual-touch sensor. Moreover, the noise region is located on the opposite side of the edge side of the imaging optical path of the visual-touch sensor configured with the monochromatic light source that matches the target type color channel image. Thus, the boundary of the noise region caused by asymmetric noise and signal attenuation effects can be accurately anchored. Based on the noise boundary in the target type color channel image of the first image, the target... Performing a masking operation on the target color channel image allows for masking only the noise region on the opposite side of the monochromatic light source in the target color channel image, rather than masking all color channels of the first image. This eliminates interference from asymmetric noise and signal attenuation effects in the noise region of the target color channel image, while preserving the effective pixel data carrying tactile deformation information in the non-noise regions of the target color channel image to the greatest extent. Image processing of all color channels in the masked second image focuses on the effective pixel data, avoiding interference from invalid data in noise regions. This effectively calculates the pressure distribution and deformation displacement information of the visual-tactile sensor caused by external contact, significantly improving the accuracy and spatial resolution of the calculation results. The output pressure distribution and deformation displacement parameters can truly reflect the actual state of the sensor's external contact, enhancing the practical application performance of the visual-tactile sensor.

[0129] The noise processing device based on the visual-tactile sensor provided in the embodiments of the present invention can execute the noise processing method based on the visual-tactile sensor provided in any of the embodiments of the present invention, and has the corresponding functions and beneficial effects of executing the noise processing method based on the visual-tactile sensor. For details, please refer to the relevant operations of the noise processing method based on the visual-tactile sensor in the foregoing embodiments.

[0130] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0131] In one embodiment, Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 6 The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0132] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to execute the noise processing method based on the visual tactile sensor provided by the present invention.

[0133] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the noise processing method based on a visual-tactile sensor provided by this invention.

[0136] In some embodiments, the noise processing method based on a visual-tactile sensor provided herein can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the noise processing method based on a visual-tactile sensor by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs for implementing the noise processing method based on visual-tactile sensors of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium stores computer instructions that, when executed by a processor, implement the noise processing method based on a visual-touch sensor provided by this invention.

[0140] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the noise processing method based on a visual-tactile sensor provided according to embodiments of the present invention. A computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0144] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the noise processing method based on a visual-touch sensor as provided in any embodiment of this application.

[0145] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A noise processing method based on a visual-tactile sensor, characterized in that, The method includes: Noise boundaries are determined in at least one target type color channel image of a first image, which is an image formed by the imaging optical path of a visual-tactile sensor. At least one edge side of the imaging optical path of the visual-tactile sensor is respectively configured with a monochromatic light source. Each target type color channel is a color channel among multiple type color channels that matches the light source color of the monochromatic light source configured at each edge side of the imaging optical path of the visual-tactile sensor. The noise boundary in each target type color channel image is the boundary of a noise region formed on the target side of each target type color channel image by a target feature. The target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured on one edge side of the imaging optical path of the visual-tactile sensor that matches each target type color channel image. The image path of the visual-tactile sensor, which is matched with the target type color channel image, is configured with a monochromatic light source on the opposite side of the edge. Determining noise boundaries in at least one target type color channel image of the first image includes: acquiring multiple reference noise boundaries associated with the visual-tactile sensor, and determining noise boundaries in at least one target type color channel image of the first image based on the multiple reference noise boundaries. The multiple reference noise boundaries are noise boundaries in different target type color channel images of the images acquired by the visual-tactile sensor, determined by statistically analyzing noise boundaries in different target type color channel images of multiple third images. The multiple third images are obtained by acquiring images in different acquisition scenarios when the visual-tactile sensor is not performing tactile perception, and multiple images are acquired in each acquisition scenario. Based on the noise boundary in the target type color channel image of the first image, a masking operation is performed on the target type color channel image to obtain the second image; The solution result of the second image is obtained by performing image processing on all color channel images in the second image. The solution result of the second image includes the pressure distribution and deformation displacement of the visual-touch sensor due to external contact when the visual-touch sensor acquires the first image.

2. The method according to claim 1, characterized in that, The multiple reference noise boundaries are generated in the following manner: Multiple reference image groups are determined based on the multiple third images, and multiple fourth images and multiple fifth images are determined in the reference image groups. Each reference image group corresponds to a different acquisition scene involved in the multiple third images. Each image in the same reference image set is an image acquired by a visual-touch sensor in the same type of acquisition scene. The fifth image is the remaining image in the reference image group excluding the multiple fourth images. The number of the multiple fourth images is less than the number of the multiple fifth images. The difference images corresponding to the fifth images in the reference image group are obtained by performing pixel-by-pixel difference between the sixth image and the multiple fifth images in the reference image group. The sixth image is determined by adding the pixels of the multiple fourth images in the reference image group pixel by pixel and taking the average value. By analyzing the difference images corresponding to multiple fifth images in different reference image groups, multiple reference noise boundaries associated with the visual-tactile sensor are determined.

3. The method according to claim 2, characterized in that, By analyzing the difference images corresponding to multiple fifth images in different reference image groups, multiple reference noise boundaries associated with the visual-touch sensor are determined, including: The first difference image associated with the reference image group is determined from the difference images corresponding to the multiple fifth images in the reference image group, and the pixel value of the target type color channel is extracted from the first difference image associated with the reference image group as the reference pixel value. The first difference image associated with the reference image group is at least one difference image selected from the difference images corresponding to the multiple fifth images in the reference image group. From the difference images corresponding to the multiple fifth images in the reference image group, we determine the multiple second difference images associated with the reference image group, and divide the pixel values ​​of the multiple second difference images associated with the reference image group in the target type color channel by the reference pixel value, and then calculate the average value of the division result pixel by pixel to obtain the difference-processed image in the target type color channel associated with the reference image group; the multiple second difference images are the remaining difference images other than the first difference image in the difference images corresponding to the multiple fifth images in the reference image group. Based on the differentially processed images of the target type color channels associated with each of the reference image groups, the pixel positions corresponding to the noise boundaries in the target type color channel images of the images acquired by the visual-touch sensor are determined to obtain multiple reference noise boundaries associated with the visual-touch sensor; the differentially processed image indicators of the target type color channels associated with the reference image groups are used to indicate noise areas in the target type color channel images of the images acquired by the visual-touch sensor where the colors exhibit irregular and uneven random changes.

4. The method according to claim 1, characterized in that, Based on the noise boundaries in the target type color channel image of the first image, a masking operation is performed on the target type color channel image to obtain a second image, including: For each target type color channel image in the first image, based on the noise boundary in the target type color channel image of the first image, a first image region and a second image region are determined from the target type color channel image. The first image region is a noise region formed on the target side of the target type color channel image caused by the target feature, and the second image region is the remaining image region in the target type color channel image other than the first image region. The pixel values ​​of the first image region are set to zero, and the pixel values ​​of the second image region are kept unchanged to obtain the second image.

5. The method according to any one of claims 1 to 4, characterized in that, Before performing a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain the second image, the method further includes: Additive Gaussian noise is applied to the target type color channel image of the first image to simulate the light field noise around the noise boundary in the target type color channel image.

6. The method according to claim 5, characterized in that, Additive Gaussian noise data augmentation is performed on the target type color channel image of the first image, including: Additive Gaussian noise data enhancement is performed in a reference region within the target type color channel image of the first image. The reference region is a region formed by extending a reference width on both sides of the noise boundary in the target type color channel image, with the noise boundary as the center. The reference width is determined based on the width of the noise region in the target type color channel image, and the reference width is smaller than the width of the noise region in the target type color channel image.

7. A noise processing device based on a visual-tactile sensor, characterized in that, The device includes: A determination module is used to determine noise boundaries in at least one target type color channel image of a first image, wherein the first image is an image formed by the imaging optical path of a visual-tactile sensor, and at least one edge side of the imaging optical path of the visual-tactile sensor is respectively configured with a monochromatic light source. Each target type color channel is a color channel among a plurality of type color channels that matches the light source color of the monochromatic light source configured at each edge side of the imaging optical path of the visual-tactile sensor. The noise boundary in each target type color channel image is the boundary of a noise region formed on the target side of each target type color channel image by a target feature. The target feature is used to indicate the asymmetric noise and signal attenuation effect generated by the monochromatic light source configured at one edge side of the imaging optical path of the visual-tactile sensor that matches each target type color channel image. The target side is the opposite side of the edge of the imaging optical path of the visual-tactile sensor that matches the target type color channel image; wherein, determining the noise boundary in at least one target type color channel image of the first image includes: acquiring multiple reference noise boundaries associated with the visual-tactile sensor, and determining the noise boundary in at least one target type color channel image of the first image based on the multiple reference noise boundaries; wherein, the multiple reference noise boundaries are respectively the noise boundaries in different target type color channel images of the images acquired by the visual-tactile sensor determined by statistically analyzing the noise boundaries in different target type color channel images of multiple third images, the multiple third images are obtained by image acquisition in different types of acquisition scenarios when the visual-tactile sensor does not perform tactile perception, and multiple images are acquired in each acquisition scenario; The processing module is used to perform a masking operation on the target type color channel image based on the noise boundary in the target type color channel image of the first image to obtain a second image; The calculation module is used to obtain the calculation result of the second image by performing image calculation on all color channel images in the second image. The calculation result of the second image includes the pressure distribution and deformation displacement of the visual-touch sensor due to external contact when the visual-touch sensor acquires the first image.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the noise processing method based on a visual-tactile sensor as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the noise processing method based on a visual-tactile sensor as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the noise processing method based on a visual-tactile sensor according to any one of claims 1-6.

Images