A visual-tactile sensor, image processing method, device and medium

Abstract

The application discloses a visual tactile sensor, an image processing method, equipment and a medium, and relates to the technical field of visual tactile sensors, and comprises a mechanical component and an optical component. The mechanical component comprises a transparent elastomer, a transparent support layer supporting the transparent elastomer, and a reflective layer arranged on the transparent elastomer. The reflective layer comprises a mark point pattern and an initial reflection pattern. The optical component comprises a light source and a camera. The light source is configured to provide illumination for the reflective layer. The camera is configured to collect a color image of the reflective layer. In the color image, the color of the mark point pattern is different from the color of the initial reflection pattern. In a gray-scale image obtained by performing gray-scale processing on the color image, the gray-scale value difference between the mark point pattern and the initial reflection pattern is less than a preset gray-scale value threshold. According to the application, dual information of mechanics and texture can be synchronously, real-timely and non-destructively obtained through single imaging.

Description

Technical Field

[0001] This invention relates to the field of visual-tactile sensor technology, and more particularly to a visual-tactile sensor, an image processing method, an apparatus, and a medium. Background Technology

[0002] Touch is a crucial way for humans to perceive and interact with the physical world. Imparting tactile sensing capabilities to robots can significantly improve their adaptability and operational level in complex tasks. In recent years, visual-tactile sensors have attracted widespread attention due to their ability to simultaneously provide high-resolution visual information and multimodal tactile signals, and are gradually being applied in fields such as industrial inspection, robot operation, and human-robot interaction. These sensors typically extract surface texture information by arranging reflective layer patterns at the contact interface, while simultaneously setting a marker layer to capture deformation under force, thereby inferring multidimensional mechanical information.

[0003] However, this method has a significant drawback: the marker pattern of the marker layer will cause optical interference to the reflection pattern of the reflection layer, resulting in the acquisition of marker features in the reflection pattern, thus making it impossible to restore the complete and lossless original texture information.

[0004] Therefore, how to acquire high-quality texture information and accurate multi-dimensional force information simultaneously and in real time without loss has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is how to acquire high-quality texture information and accurate multi-dimensional force information simultaneously and in real time without loss.

[0006] To address the aforementioned problems, in a first aspect, embodiments of the present invention provide a visual-tactile sensor, characterized in that it comprises:

[0007] A mechanical component includes a transparent elastomer, a transparent support layer supporting the transparent elastomer, and a reflective layer disposed on the transparent elastomer, wherein the reflective layer includes a marker pattern and an initial reflection pattern;

[0008] Optical components include a light source and a camera, the light source being configured to provide illumination to the reflective layer; the camera being configured to acquire a color image of the reflective layer;

[0009] In the color image, the color of the marker pattern is different from the color of the initial reflection pattern; in the grayscale image obtained by grayscale processing the color image, the difference in grayscale value between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold.

[0010] Optionally, it also includes a reflective device configured to reflect the image of the mechanical component onto the camera.

[0011] Optionally, the material of the marker pattern and the material of the initial reflection pattern are different color variations of the same material.

[0012] Optionally, the contact angle between the marked dot pattern and the surface of the transparent elastomer is an acute angle.

[0013] Optionally, the mechanical component further includes a protective layer disposed on the outside of the reflective layer.

[0014] In a second aspect, embodiments of the present invention provide an image processing method applied to the visual-tactile sensor as described in the first aspect, the method comprising:

[0015] Acquire a color image of the reflective layer captured by the camera;

[0016] The color image is converted into a grayscale image, wherein the difference in grayscale value between the marker pattern and the initial reflection pattern in the grayscale image is less than a preset grayscale value threshold.

[0017] Optionally, converting the color image to a grayscale image includes:

[0018] Determine the pixel difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three color channels of the RGB color channel in the color image;

[0019] The weighting coefficients of each color channel in the three color channels are determined based on the pixel difference. The weighting coefficients are used to convert the color image into a grayscale image. The weighting coefficients satisfy the constraint that after the pixel values ​​of the color of the marker pattern and the color of the initial reflection pattern are weighted using the weighting coefficients, the difference in grayscale values ​​between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold.

[0020] The color image is converted into a grayscale image based on the weighting coefficients.

[0021] Optionally, determining the pixel difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three RGB color channels in the color image includes:

[0022] When the difference in grayscale value between the marker pattern and the initial reflection pattern in the initial grayscale image is greater than the preset grayscale value threshold, the pixel value difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three color channels of the RGB color channel in the color image is determined; the initial grayscale image is obtained by converting the color image to grayscale based on preset initial weight parameters.

[0023] Thirdly, embodiments of the present invention provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the second aspect.

[0024] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the method described in the second aspect.

[0025] Compared with the prior art, the technical effects achieved by the embodiments of the present invention include:

[0026] This invention provides a visual-tactile sensor, comprising mechanical and optical components. The mechanical components include a transparent elastomer, a transparent support layer supporting the transparent elastomer, and a reflective layer disposed on the transparent elastomer. The reflective layer includes a marker dot pattern and an initial reflective pattern. The optical components include a light source and a camera. The light source is configured to provide illumination to the reflective layer. The camera is configured to acquire a color image of the reflective layer. In the color image, the color of the marker dot pattern is different from the color of the initial reflective pattern. In the grayscale image obtained by grayscale processing the color image, the difference in grayscale values ​​between the marker dot pattern and the initial reflective pattern is less than a preset grayscale value threshold. In this embodiment of the invention, the reflective layer has a marker pattern that differs in color from the initial reflective pattern in a color image. By tracking the deformation of the marker pattern, multidimensional force information can be calculated. In the grayscale image obtained after grayscale processing, the difference in grayscale values ​​between the marker pattern and the initial reflective pattern is less than a preset grayscale value threshold. That is, in the grayscale image, the grayscale values ​​of the marker pattern and the initial reflective pattern are nearly identical. When performing texture recognition based on the grayscale image, the marker pattern will not interfere with texture recognition. By analyzing the grayscale image, non-destructive, high-resolution texture information of the contact object surface can be obtained. This scheme requires only a single imaging operation to simultaneously, in real-time, and non-destructively acquire both mechanical and textural information. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0030] Figure 1 This is a schematic diagram of the structure of a visual-tactile sensor proposed in an embodiment of the present invention;

[0031] Figure 2 This is an exploded view of a visual-tactile sensor proposed in an embodiment of the present invention;

[0032] Figure 3 An exploded view of a visual-tactile sensor according to another embodiment of the present invention;

[0033] Figure 4 This is a schematic flowchart of an image processing method proposed in an embodiment of the present invention;

[0034] Figure 5 This is a structural block diagram of a computer device proposed in an embodiment of the present invention.

[0035] Figure Labels

[0036] Protective layer 10, reflective layer 20, transparent elastomer 30, transparent support layer 40, light source 50, mounting base 60, camera 70, housing 80, reflective device 90. Detailed Implementation

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Similar component reference numerals in the drawings represent similar components. Obviously, the embodiments described below are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0038] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terminology used in this specification of embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of the invention. As used in this specification of embodiments of the invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0040] See Figures 1-3 This invention provides a visual-tactile sensor, comprising mechanical components and optical components, the specific structure of which is described below:

[0041] The mechanical component includes a transparent elastomer 30, a transparent support layer 40 supporting the transparent elastomer 30, and a reflective layer 20 disposed on the transparent elastomer 30. The reflective layer 20 includes a marker pattern and an initial reflection pattern.

[0042] The optical components include a light source 50 and a camera 70. The light source 50 is located on one side of the mechanical components and is configured to provide illumination to the reflective layer 20. The camera 70 is located on one side of the mechanical components and is configured to acquire color images of the reflective layer 20.

[0043] Specifically, the transparent elastomer 30 can convert externally applied forces into its own deformation, providing a physical basis for subsequent acquisition of mechanical information. The light source 50 provides illumination to the entire system, ensuring that the reflective layer 20 can be clearly imaged. The camera 70, located on one side of the mechanical components, is used to acquire color images of the reflective layer 20.

[0044] In the color image, the color of the marker pattern is different from the color of the initial reflection pattern; in the grayscale image obtained by grayscale processing the color image, the difference in grayscale values ​​between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold. The grayscale value threshold can be set by those skilled in the art; the smaller the grayscale value threshold, the less interference the marker pattern causes to texture recognition. The preset grayscale processing can specifically be an image processing method provided in the following embodiments of the present invention.

[0045] The core of this invention lies in the fact that the reflective layer 20 is constructed as a composite structure comprising a marker pattern and an initial reflective pattern. In color images, because the marker pattern and the initial reflective pattern are designed to have different color characteristics, sufficient color contrast is formed between them, enabling the complete shape and spatial distribution of the marker pattern to be captured clearly and accurately. When subjected to contact and pressure from an external object, the transparent elastomer 30 undergoes corresponding deformation, causing the reflective layer 20 attached to it to shift or change shape. At this time, in the acquired color image, the changes in the position, shape, or spacing of the marker patterns with different colors can be tracked and analyzed with high precision. Based on these specific deformation data, the multidimensional force information applied to the sensor surface, including but not limited to the magnitude and direction of normal force and shear force, can be accurately calculated through a pre-established mechanical model or calibration algorithm.

[0046] Meanwhile, the design essence of the reflective layer 20 is further reflected in the grayscale image processing. By carefully selecting the spectral reflectance characteristics of the materials constituting the marker pattern and the initial reflective pattern, and controlling their response after grayscale conversion, it is ensured that although the colors are different in the color space, their grayscale values ​​are highly consistent when converted to grayscale images (the difference in grayscale values ​​is less than a preset grayscale value threshold). This characteristic allows the marker pattern, which was clearly distinguishable in the color image, to merge into a reflective pattern because its grayscale value is similar to that of the surrounding initial reflective pattern. By analyzing the reflective pattern, high-resolution, non-destructive texture information of the contact object surface can be directly obtained.

[0047] As can be seen, this invention utilizes only a single image acquisition and, through subsequent different image processing paths, generates two independent high-quality information streams in parallel: from the color image, accurate multidimensional force information can be reliably and stably calculated; while from the grayscale image, high-fidelity, lossless texture information without marker interference can be obtained. These two types of information are completely synchronized in time, originating from the same physical contact state at the same moment, thus enabling the simultaneous acquisition of texture information and multidimensional force information.

[0048] In some embodiments, see Figure 3 The system also includes a reflector 90, located on one side of the mechanical component and configured to reflect the image of the mechanical component to the camera 70. The reflector 90 can be specifically a mirror. The reflector 90 is used to reflect the image of the mechanical component to the camera 70. The addition of this component brings significant technical benefits in terms of mechanical layout optimization and improved space utilization efficiency. In traditional direct-light optical path designs, the camera 70 must be positioned directly opposite the transparent elastomer 30, which often results in a large sensor volume in the direction perpendicular to the contact plane (Z-axis), making integration in space-constrained applications (such as robot fingertips) impractical. By introducing the reflector 90, the light propagation path is altered, allowing the camera 70 to acquire images from the side or even parallel to the mechanical component. This folded-back optical path design allows the camera 70 to be positioned on the side of the sensor body rather than the top, significantly reducing the overall profile height of the sensor and achieving a flatter and more compact sensor structure.

[0049] In other embodiments, see Figure 2 Alternatively, the reflector 90 can be omitted to reduce costs, and a traditional direct optical path design can be used, with the camera 70 located inside the mounting base 60, facing the transparent elastomer 30.

[0050] In some embodiments, the materials of the marker pattern and the initial reflection pattern are different color variations of the same material. Different color variations of the same material ensure consistency in key physical and mechanical properties between the marker pattern and the initial reflection pattern, such as elastic modulus, hardness, adhesion, and coefficient of thermal expansion, thereby improving accuracy. Using the same material ensures that the marker pattern and the initial reflection pattern have the same material structure and surface state, thus giving them the same basic optical properties, such as refractive index and reflection mode. Using different color variations of the same material for the marker pattern and the initial reflection pattern ensures that they have different colors in color images, while their grayscale features are highly consistent in grayscale images. This ensures clear distinguishability between the marker pattern and the initial reflection image during multi-dimensional force recognition based on color images, and prevents interference with the initial reflection pattern during texture recognition based on grayscale images. In some embodiments, the marker pattern uses silver metallic powder, and the initial reflection pattern uses bronze metallic powder; however, this is not specifically limited in the embodiments of the present invention.

[0051] In some embodiments, both the initial reflection pattern and the marker pattern are disposed on the surface of the transparent elastomer 30, and the initial reflection pattern and the marker pattern themselves form a reflective layer 20. The contact angle between the marker pattern and the surface of the transparent elastomer 30 is an acute angle. It should be understood that in this embodiment, the marker pattern is formed by coating the surface of the elastomer with a selected material. Taking a dot-shaped marker pattern as an example, the marker will form a protrusion on the surface of the elastomer. The smaller the contact angle, the lower the boundary discrimination after the marker pattern is imaged. After grayscale processing, the overall grayscale values ​​of the grayscale image are closer, and the texture recognition effect is better. When the contact angle between the marker and the surface of the transparent elastomer 30 is close to a right angle, the reflection characteristics of the incident light at the contact position will change significantly, resulting in a significant difference in the RGB values ​​on both sides of the boundary, and bright spots or dark spots appear in the corresponding image. In subsequent image processing, these abnormal grayscale values ​​generated by geometric boundaries rather than the texture itself will form strong edge noise, interfering with the extraction of texture information.

[0052] In other embodiments, the reflective layer 20 includes at least one substrate, with a marking pattern and an initial reflective pattern disposed on the substrate, and the contact angle between the marking pattern material and the substrate is an acute angle. In this embodiment, the reflective layer 20 includes at least one substrate, with a marking pattern and an initial reflective pattern disposed on the substrate, and the contact angle between the marking pattern material and the substrate is an acute angle. The substrate can be a transparent or opaque material. The substrate is an elastic material to ensure that when the mechanical component is pressed, the marking pattern disposed thereon deforms, so as to obtain mechanical information based on the deformation of the marking pattern. The reflective layer 20 may include only one substrate, with both the marking pattern and the initial reflective pattern disposed on the surface of the substrate. For example, the initial reflective pattern covers the substrate, and the marking pattern is disposed on top of the initial reflective pattern. Alternatively, the marking pattern and the initial reflective pattern are alternately disposed on the same substrate surface. Optionally, the reflective layer 20 may include two substrates, with the marking pattern and the initial reflective pattern disposed on the two substrates respectively, and the two substrates are stacked to form the reflective layer 20. By controlling the contact angle to an acute angle, a smooth transition slope interface is formed between the pattern material and the substrate. When light shines on this slope, the change in its incident angle is gradual, so the change in reflected light intensity is also continuous, avoiding drastic grayscale jumps at the boundary pixels and thus improving accuracy.

[0053] In some preferred embodiments, the mechanical component further includes a protective layer 10 disposed on the outside of the reflective layer 20.

[0054] In practice, the reflective layer 20 is typically composed of a fine coating or pattern. Direct exposure to the external environment makes it highly susceptible to damage from friction, scratches, contamination, or chemical corrosion, leading to decreased optical performance or functional failure. In this embodiment, by adding a protective layer 10 (usually made of a transparent, wear-resistant, and scratch-resistant flexible material such as polyurethane or silicone) as the outermost barrier, mechanical shocks and wear from external contact can be effectively absorbed and dispersed, protecting the integrity of the marking patterns and initial reflection patterns on the underlying reflective layer 20. This significantly extends the sensor's lifespan, maintains the stability of its long-term measurement accuracy (whether force or texture information), and enables the sensor to adapt to high-frequency, high-intensity interactive applications, enhancing the product's practical value and reliability.

[0055] In some embodiments, a transparent support layer 40 is disposed on one side of the transparent elastomer 30 and fixedly disposed on the housing 80 of the visual-touch sensor.

[0056] In practice, the transparent elastomer 30 and its functional thin films (reflective layer 20, protective layer 10) form a flexible whole that undergoes complex deformation under stress. Without a rigid or relatively stable supporting substrate, the entire flexible body may experience uncontrolled overall displacement or twisting, which would introduce additional, non-contact force-induced displacement of the marker points, severely interfering with the accuracy of multidimensional force information calculation. The transparent support layer 40 (such as a transparent acrylic or glass plate) is fixed to the sensor housing 80, providing a firm and flat attachment surface for the flexible elastomer, ensuring that the sensor has a clear zero reference plane in the non-contact state. When external force is applied to the protective layer 10, the deformation mainly occurs in the transparent elastomer 30, while its connection surface with the support layer remains relatively fixed. This allows the displacement and deformation of the marker points to more accurately reflect the applied force, rather than the shaking of the entire sensor. This is crucial for improving the accuracy, repeatability, and signal-to-noise ratio of force sensing, and is the fundamental guarantee for achieving high-precision tactile perception. Understandably, the housing 80 has a window for mounting the transparent support layer 40, through which the camera 70 takes images.

[0057] This invention provides a visual-tactile sensor, comprising mechanical and optical components. The mechanical components include a transparent elastomer, a transparent support layer supporting the elastomer, and a reflective layer disposed on the elastomer. The reflective layer includes a marker pattern and an initial reflective pattern. The optical components include a light source and a camera. The light source is configured to provide illumination to the reflective layer, and the camera is configured to acquire a color image of the reflective layer. In the color image, the color of the marker pattern differs from the color of the initial reflective pattern. In the grayscale image obtained by processing the color image to a preset grayscale value, the difference in grayscale values ​​between the marker pattern and the initial reflective pattern is less than a preset grayscale value threshold, and the marker pattern and the initial reflective pattern together constitute the reflective pattern. In this embodiment, the marker pattern and the initial reflective pattern differ in color in the color image to accurately track deformation and calculate multidimensional force information. In the grayscale image, the difference in grayscale values ​​between the two is less than a preset grayscale value threshold, causing them to merge and jointly constitute the reflective pattern. By analyzing the reflective pattern, high-resolution, non-destructive texture information of the contact object surface can be directly obtained. This method requires only a single image to acquire both mechanical and textural information synchronously, in real time, and without loss.

[0058] See Figure 4 This invention provides an image processing method applied to the visual-tactile sensor proposed in any of the above embodiments. The method includes the following steps:

[0059] S1, acquire a color image of the reflective layer captured by the camera.

[0060] S2, converting the color image into a grayscale image, wherein the difference in grayscale value between the marker pattern and the initial reflection pattern in the grayscale image is less than a preset grayscale value threshold. The grayscale value threshold can be set by those skilled in the art, and this invention does not specifically limit it.

[0061] In practice, a camera is first used to acquire color images of the reflective layer. Under the color image, the marked point pattern and the initial reflection pattern are clearly distinguishable due to the color difference, providing a data source for subsequent mechanical calculations.

[0062] Furthermore, the color image is converted to a grayscale image. During the conversion, the difference between the color of the marker pattern and the grayscale value of the initial reflection pattern is less than a preset grayscale value threshold, and the marker pattern and the initial reflection pattern together constitute the reflection pattern. This characteristic allows the marker pattern, which was originally clearly distinguishable in the color image, to merge into a single reflection pattern because its grayscale value is the same as the grayscale value of the surrounding initial reflection pattern. By analyzing the reflection pattern, high-resolution, non-destructive texture information of the contacting object surface can be directly obtained.

[0063] In some preferred embodiments, the above step of "converting a color image to a grayscale image" specifically includes the following steps: determining the pixel difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three color channels of the color image; determining the weight coefficient of each color channel in the three color channels based on the pixel difference, the weight coefficient being used to convert the color image to a grayscale image; the weight coefficient satisfying the constraint condition: after weighting the pixel values ​​of the color of the marker pattern and the color of the initial reflection pattern using the weight coefficient, the difference in grayscale values ​​between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold; and converting the color image to a grayscale image based on the weight coefficient.

[0064] Specifically, firstly, the difference between the marked point pattern and the initial reflection pattern in each of the three RGB color channels is quantitatively analyzed. These difference data are the basis for solving the grayscale conversion weights. Subsequently, this embodiment of the invention sets a key constraint: the sought weighting coefficients must ensure that the difference in grayscale values ​​obtained after weighted calculation of the two colors is less than a preset grayscale value threshold. In this embodiment, a preferred scheme is adopted, and the constraint is specifically set as follows: the sought weighting coefficients must ensure that the two colors obtain the same grayscale value after weighted calculation, that is, the difference in their grayscale values ​​is equal to 0.

[0065] The RGB values ​​corresponding to the marked point patterns are respectively , , The RGB values ​​corresponding to the initial reflection pattern are respectively , , Taking this as an example, we will explain the process of determining the weighting coefficients.

[0066] In this embodiment of the invention, the conversion of a color image to a grayscale image is based on a weighted algorithm. Existing weight allocation allows grayscale values ​​to restore the brightness and darkness levels of a color image. However, in this embodiment of the invention, the purpose of converting a color image to grayscale is to weaken the brightness and darkness levels of the color image, so that the grayscale values ​​of each pixel in the resulting grayscale image are approximately the same.

[0067] For the two sets of RGB values ​​mentioned above, their grayscale values ​​are respectively ,in:

[0068]

[0069] in, , , The weights of the RGB channels when converting a color image to a grayscale image. When the grayscale values ​​are the same, there are... = ,Right now:

[0070]

[0071] For RGB to grayscale image conversion, set weight constraints:

[0072]

[0073] make , , The above system of equations can be expressed as:

[0074]

[0075] In the solution process, it is necessary to use one variable as a free variable and solve for the other two variables. Taking free variables as an example ( ):

[0076]

[0077] This is a system of two equations.

[0078] Solve using the substitution method, assuming... The second equation in the above system of equations Substituting into the first equation, we get , .

[0079] For example: In order to satisfy A suitable t-value needs to be set. Using (R, G, B) values ​​of (100, 150, 200) and (50, 200, 100) as the first and second materials respectively, the weights required to ensure consistent grayscale values ​​after converting the RGB images of the two materials to grayscale images are calculated. =50, =-50, =100. Substituting into the above calculation process, we get... ,

[0080] Considering the non-negativity constraint of the weights, ,have to Calculations yielded Take the median value. = Substituting into the above formula, we get , 0.5833.

[0081] It should be noted that in the embodiments of the present invention, it is not limited to using only two colors. That is, the total number of colors of the marker pattern and the initial reflection pattern can also exceed two types, such as three or more types. Appropriate weighting parameters can be used to ensure that the grayscale values ​​of the three or more colors are consistent after being converted to grayscale, that is, the colors are consistent.

[0082] For example, suppose the marker pattern and the initial reflection pattern are composed of three colors of material, and their RGB values ​​are:

[0083] Color 1: (R1, G1, B1); Color 2: (R2, G2, B2); Color 3: (R3, G3, B3).

[0084] The weights W of the three RGB channels R W G W B .

[0085] The goal is to make their grayscale values ​​equal, let this common grayscale value be K. Based on the grayscale formula, we can obtain the following three equations:

[0086] 1.R1 * W R + G1 * W G + B1 * W B =K

[0087] 2.R2 * W R + G2 * W G + B2 *W B =K

[0088] 3.R3 * W R + G3 * W G+ B3 * W B =K

[0089] And there are weight constraints:

[0090] 4. W R + W G + W B =1

[0091] The above four equations contain four unknowns (W). R W G W B This is a system of linear equations, K, which theoretically has a solution.

[0092] Solution method:

[0093] The solution can be found using the elimination method. By subtracting the third equation from the first two equations, the unknown K can be eliminated.

[0094] Subtract equation 2 from equation 1:

[0095] (R1 - R2) * W R + (G1 - G2) * W G + (B1 - B2) * W B =0

[0096] Subtract equation 3 from equation 1:

[0097] (R1 - R3) * W R + (G1 - G3) * W G +(B1 - B3) *W B =0

[0098] We obtain two new equations, plus the weight constraint equation: W R + W G + W B =1

[0099] We obtain a system of three linear equations in three variables:

[0100] 1.(R1 - R2) * W R + (G1 - G2) * W G + (B1 - B2) * W B =0

[0101] 2.(R1 - R3) * W R + (G1 - G3) * W G + (B1 - B3)W B =0

[0102] 3. W R + W G + W B =1

[0103] This system of equations can then be easily solved using the matrix method. The weighting coefficients obtained through the above method ensure that the grayscale image formed by the marker pattern and the initial reflection pattern, regardless of the materials used, is nearly identical, providing a foundation for texture recognition in visual-tactile sensors. Existing weight-based grayscale conversion methods are designed for general image displays and cannot meet the specific requirement of making two particular colors completely identical in grayscale. Furthermore, this algorithm has a certain degree of universality; by inputting different color differences, the optimal weights corresponding to different color combinations can be solved. This allows the method to adapt to possible changes or expansions in the color of the sensor's reflective layer material, enhancing the flexibility and adaptability of the entire technical solution.

[0104] In some embodiments, the color image is first converted into an initial grayscale image based on initial weight parameters. If the difference in grayscale values ​​between the marker pattern and the initial reflection pattern in the initial grayscale image exceeds a preset grayscale threshold, the color image is then converted to grayscale using the grayscale conversion method described in the above embodiments. Specifically, the initial grayscale image is obtained by converting the color image to grayscale based on preset initial weight parameters. The initial weight parameters are conventional weight parameters set according to the human eye's sensitivity to different colors in existing grayscale conversion methods. These weight parameters have weights of 0.299, 0.587, and 0.114 for the R, G, and B channels, respectively. In this way, when it is uncertain whether the grayscale characteristics of the materials used in the marker pattern and the initial reflection pattern are the same, grayscale conversion is first performed using a conventional grayscale conversion method to determine whether the difference in grayscale values ​​between the marker pattern and the initial reflection pattern in the converted grayscale image meets the preset grayscale threshold. If the preset grayscale threshold is met, the weight determination process during grayscale conversion can be omitted, saving computational power.

[0105] Further, please refer to Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0106] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0107] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to perform an image processing method.

[0108] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0109] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can perform an image processing method.

[0110] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0111] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of an image processing method provided in any of the above method embodiments.

[0112] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0113] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0114] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of an image processing method provided in any of the above-described method embodiments.

[0115] Storage media are physical, non-transitory storage media, such as USB flash drives, external hard drives, read-only memory (ROM), magnetic disks, or optical disks—various physical storage media capable of storing program code. Computer-readable storage media can be non-volatile or volatile.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0117] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0118] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0119] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0120] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0121] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0122] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0123] The above description describes specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual-tactile sensor, characterized in that, include: A mechanical component includes a transparent elastomer, a transparent support layer supporting the transparent elastomer, and a reflective layer disposed on the transparent elastomer, wherein the reflective layer includes a marker pattern and an initial reflection pattern; Optical components include a light source and a camera, the light source being configured to provide illumination to the reflective layer; the camera being configured to acquire a color image of the reflective layer; In the color image, the color of the marker pattern is different from the color of the initial reflection pattern; in the grayscale image obtained by grayscale processing the color image, the difference in grayscale value between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold.

2. The visual-tactile sensor according to claim 1, characterized in that, It also includes a reflective device configured to reflect the image of the mechanical component onto the camera.

3. The visual-tactile sensor according to claim 1, characterized in that, The material of the marker pattern and the material of the initial reflection pattern are different color variations of the same material.

4. The visual-tactile sensor according to claim 1, characterized in that, The contact angle between the marked dot pattern and the surface of the transparent elastomer is an acute angle.

5. The visual-tactile sensor according to claim 1, characterized in that, The mechanical component also includes a protective layer disposed on the outside of the reflective layer.

6. An image processing method, characterized in that, The method, applied to the visual-tactile sensor as described in any one of claims 1-5, comprises: Acquire a color image of the reflective layer captured by the camera; The color image is converted into a grayscale image, wherein the difference in grayscale value between the marker pattern and the initial reflection pattern in the grayscale image is less than a preset grayscale value threshold.

7. The image processing method according to claim 6, characterized in that, The step of converting the color image to a grayscale image includes: Determine the pixel difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three color channels of the RGB color channel in the color image; The weighting coefficients of each color channel in the three color channels are determined based on the pixel difference. The weighting coefficients are used to convert the color image into a grayscale image. The weighting coefficients satisfy the constraint that after the pixel values ​​of the color of the marker pattern and the color of the initial reflection pattern are weighted using the weighting coefficients, the difference in grayscale values ​​between the marker pattern and the initial reflection pattern is less than a preset grayscale value threshold. The color image is converted into a grayscale image based on the weighting coefficients.

8. The image processing method according to claim 7, characterized in that, Determining the pixel differences between the color of the marker pattern and the color of the initial reflection pattern in each of the three RGB color channels in the color image includes: When the difference in grayscale value between the marker pattern and the initial reflection pattern in the initial grayscale image is greater than the preset grayscale value threshold, the pixel value difference between the color of the marker pattern and the color of the initial reflection pattern in each of the three color channels of the RGB color channel in the color image is determined; the initial grayscale image is obtained by converting the color image to grayscale based on preset initial weight parameters.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 6-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 6-8.

Images