Method and system for recognizing object surface morphology and hardness based on visual-tactile perception

Abstract

The present application relates to the field of robot tactile perception, in particular to a method and system for identifying object surface morphology and hardness based on visual tactile perception. The method comprises the following steps: S1: pressing the object to be measured by a visual tactile sensor with a fixed displacement, obtaining a pressure map sequence of the object to be measured during the pressing process; S21A: obtaining the force size sequence corresponding to the pressure map sequence through a first classification neural network model; S22A: calculating the maximum value of force and the gradient of force in the force size sequence corresponding to the pressure map sequence; S21B: analyzing the last frame of pressure map in the pressure map sequence through a photometric stereo algorithm, and drawing a three-dimensional point cloud view; S22B: calculating the fitting radius of the three-dimensional point cloud view; S3: splicing the maximum value of force, the gradient value of force and the calculated radius to form a feature matrix; S4: analyzing the feature matrix through a second classification neural network model to obtain the category of the surface morphology and hardness of the object to be measured.

Description

Technical Field

[0001] This invention relates to the field of robot tactile perception, and in particular to a method and system for recognizing the surface shape and hardness of objects based on visual-tactile perception. Background Technology

[0002] With the rapid development of industrial automation and intelligent robot technology, the ability to accurately and without damage grasp objects has become a crucial requirement in the robotics field. When grasping an object, its surface morphology and hardness are the most important physical properties. Surface morphology refers to the visual geometric appearance of an object's surface, primarily its size and local shape. Surface morphology determines the size of the robotic gripper needed and the grasping method. Hardness refers to an object's ability to resist indentation by a harder object. It determines the pressure the robot needs to apply to the object's surface to ensure successful grasping without damage.

[0003] One existing method for identifying objects to be grasped by a robot relies on vision sensors. However, vision sensors only identify objects by capturing their surface morphology, making it difficult to determine the object's hardness. Applying too much force may damage the object, while applying too little force will result in grasping failure. Another existing method relies on tactile sensors. Tactile sensors require complex mechanical movements and repeated contact with the object to determine its surface morphology and hardness, resulting in low detection efficiency. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to overcome the defects or deficiencies of the prior art and provide a method and system for identifying the surface shape and hardness of an object based on visual-tactile perception.

[0005] A method for recognizing the surface morphology and hardness of an object based on visual-tactile perception includes the following steps:

[0006] S1: Press the object under test with a fixed displacement D using a visual tactile sensor to obtain a pressure map sequence of the object under test during the pressing process;

[0007] S21A: Obtain the force magnitude sequence corresponding to the pressure map sequence through a first neural network model;

[0008] S22A: Calculate the maximum force F in the force magnitude sequence corresponding to the pressure diagram. max and the gradient of force G;

[0009] S21B: Obtain a point cloud view by analyzing the last frame of the pressure map sequence using a photometric stereo algorithm, and then draw a three-dimensional point cloud view.

[0010] S22B: Calculate the fitting radius R of the 3D point cloud view;

[0011] S3: The maximum force F max The gradient value of the force G and the computational radius R are concatenated to form a feature matrix {F}. max ,G,R};

[0012] S4: Transform the characteristic matrix {F} max The surface morphology and hardness category P of the object under test are obtained by analyzing G and R through a second classification neural network model.

[0013] By pressing the object under test with a fixed displacement, the variables of the object during pressing are fixed, making it easier to characterize its hardness using the magnitude of the force. Three RGB light sources from the visual-tactile sensor illuminate the flexible contact layer. When the flexible contact layer deforms, the light produces different gradient changes. The camera of the visual-tactile sensor receives the RGB light reflected back from the flexible contact layer in real time and captures images to obtain a pressure map sequence during the pressing process. This image sequence reflects the deformation of the flexible contact layer at corresponding moments during the pressing process. The maximum force F during the pressing process is obtained by analyzing the deformation of the flexible contact layer. max The force gradient G and the calculation radius R can reflect the hardness and surface morphology of an object. This detection method can identify and detect the hardness and surface morphology of an object with only a single pressing contact, and has the advantages of fast detection speed and high efficiency.

[0014] Furthermore, the first classification neural network model in step S2 is trained through the following steps:

[0015] SA: Using visual and tactile sensors and force sensors, multiple different standard spheres are pressed at a fixed displacement, and pressure map sequences and force magnitude sequences are collected simultaneously.

[0016] SB: The pressure map sequence and its corresponding stress magnitude sequence are input into the first classification neural network model for training to obtain the mapping relationship between the pressure map and the magnitude of the force.

[0017] By conducting preliminary experiments, we obtained the force sequence and pressure map sequence when pressing multiple different standard spheres. Then, we established the mapping relationship between pressure map and force through a first-class neural network model, so that the magnitude of the force can be directly obtained from the pressure map captured by the camera.

[0018] Furthermore, the photometric stereo algorithm calculation expression for step S21B is as follows:

[0019]

[0020] Simplified to: I(x,y)=R(x,y)L(x,y)n(x,y)

[0021] In the formula, I(x,y) represents the brightness value of the three RGB light sources at coordinates (x,y) in the image, R(x,y) represents the reflectivity of the flexible contact layer surface, L(x,y) represents the unit vector of the illumination direction of each RGB light source, and n(x,y) represents the normal vector of the 3D surface. I(x,y) is obtained from the captured pressure map; R(x,y) is a fixed value, i.e., the reflectivity of the reflective layer; L(x,y) is a fixed value, determined according to the angle between the camera and the three RGB light sources.

[0022] Further, step S22B includes the following steps:

[0023] S22B1: Compress the three-dimensional point cloud view on a plane parallel to the flexible contact layer to obtain a two-dimensional projection view of the three-dimensional point cloud view.

[0024] S22B2: Use Gaussian filtering to reduce noise in a two-dimensional projected view;

[0025] S22B3: The denoised 2D projection is attempted to be converted into an 8-bit grayscale image, and the grayscale image is subjected to binarization thresholding to obtain a binarized image with a roughly circular boundary.

[0026] S22B4: Performs circle detection on a binarized image with an approximately circular boundary using Hough circle detection and obtains the fitted radius R.

[0027] The above steps are used to calculate the fitting radius of the pressure map captured by the camera through photometric stereo algorithm and image processing steps. The fitting radius can then be used to reflect the surface morphology of the object.

[0028] Furthermore, the second classification neural network model package in step S4 is trained through the following steps:

[0029] SA': Pressing multiple standard spheres of different softness and hardness and different radii with a fixed displacement using a visual-tactile sensor and simultaneously collecting pressure map sequences;

[0030] SB': Input the pressure map sequence into the trained first classification neural network model to obtain the magnitude of the force corresponding to each frame of the pressure map in the pressure map sequence;

[0031] SC': The maximum force F of each standard sphere when it is pressed is calculated by using steps S21A, S22A, S21B, and S22B to obtain the magnitude of the force corresponding to each frame of pressure image. max The gradient value G of the force and the fitting radius R are concatenated to form a feature matrix {F}. max,G,R};

[0032] SD': For each feature matrix, assign a classification level label based on the different hardness, softness, and radius of the corresponding standard sphere;

[0033] SE': Divide the feature matrix and the corresponding classification level label into a training set and a test set, input them into the second classification neural network model to train it, and obtain the mapping relationship between the feature matrix and the classification level label.

[0034] The feature matrix {F} of different standard spheres was obtained through numerous preliminary experiments. max The dataset {F}, along with their corresponding hardness and surface morphology classification level labels, is used to train a second neural network model, enabling the second neural network model to utilize the feature matrix {F}. max The classification level labels corresponding to the objects are fitted by {G, R}.

[0035] A surface morphology and hardness recognition processor based on visual-tactile perception includes a first neural network model for parsing a force magnitude sequence corresponding to a pressure map sequence; wherein the pressure map sequence is obtained by pressing the object under test with a visual-tactile sensor at a fixed displacement D, and obtaining the pressure map sequence of the object under test during the pressing process.

[0036] Gradient calculator, used to calculate the maximum force F in the pressure map sequence. max and the gradient of force G;

[0037] A photometric stereo calculator is used to analyze the last frame of a pressure map in a pressure map sequence and draw a 3D point cloud view.

[0038] Image processor used to calculate the fitting radius R of a 3D point cloud view;

[0039] Matrix splicer, used to convert the maximum force F max The gradient value of the force G and the computational radius R are concatenated to form a feature matrix {F}. max ,G,R};

[0040] The second classification neural network model is used to analyze the feature matrix {F} max , G, R}, to obtain the category P of the surface morphology and hardness corresponding to the object.

[0041] Furthermore, the first classification neural network model is a ResNet50 network; the second classification neural network model is a multilayer perceptron classifier with a hidden layer size of (75, 25).

[0042] A surface morphology and hardness recognition system based on visual-tactile perception includes a visual-tactile sensor, a robotic arm, and a surface morphology and hardness recognition processor. The visual-tactile sensor is disposed at the end of the robotic arm. When the robotic arm grasps an object to be tested, it presses the visual-tactile sensor against the object with a fixed displacement D and obtains a pressure map sequence of the object during the pressing process. The surface morphology and hardness recognition processor analyzes the pressure map sequence to obtain the surface morphology and hardness category P corresponding to the object.

[0043] Furthermore, the visual-tactile sensor includes a flexible contact layer, three RGB light sources, and a camera; the flexible contact layer can contact an object and deform; the three RGB light sources illuminate one side of the flexible contact layer from three different angles. This allows the three RGB lights illuminating the flexible contact layer to produce different gradient changes when the flexible contact layer deforms.

[0044] Furthermore, the visual-tactile sensor also includes a reflective layer and a light-diffusing film. Light emitted from the three RGB light sources passes through the light-diffusing film and illuminates the flexible contact layer. The flexible contact layer reflects a portion of the light back to the camera, while the remaining light passes through the flexible contact layer and illuminates the reflective layer adjacent to it, and is then reflected back to the camera. The reflective layer increases the reflectivity of the three RGB light sources illuminating the flexible contact layer, improving the signal-to-noise ratio of the pressure map signal acquired by the camera. The light-diffusing film transforms the light emitted from the three RGB light sources from point sources to surface sources, making the illumination more uniform and preventing the formation of bright spots on the flexible contact layer.

[0045] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0046] Figure 1 This is a structural diagram of a visual-touch sensor;

[0047] Figure 2 A schematic diagram of a processor for recognizing the surface morphology and hardness of objects;

[0048] Figure 3 This is a flowchart of a method for recognizing the surface shape and hardness of an object based on visual-tactile perception.

[0049] Figure 4 These are schematic diagrams of different standard spheres in embodiments of the present invention;

[0050] Figure 5 These are schematic diagrams of pressure maps and three-dimensional point cloud views of different standard spheres in embodiments of the present invention. Detailed Implementation

[0051] The object surface morphology and hardness recognition system based on visual-tactile perception of the present invention includes a robotic arm, a visual-tactile sensor 1 disposed at the end of the robotic arm, and an object surface morphology and hardness recognition processor 2. When the robotic arm grasps the object to be tested, the visual-tactile sensor 1 is pressed against the object with a fixed displacement D, and a pressure map sequence of the object during the pressing process is obtained. The object surface morphology and hardness recognition processor analyzes the pressure map sequence to obtain the surface morphology and hardness category P corresponding to the object. Using the visual-tactile sensor 1, the surface morphology and hardness information of the object can be obtained simultaneously through a single detection contact, which has the advantages of fast detection speed and high efficiency.

[0052] For details, please refer to Figure 1 This is a structural diagram of the visual-tactile sensor 1, which includes a housing 11, a flexible contact layer 12, three RGB light sources 13, a reflective layer 14, a support plate 15, a light-diffusing film 16, and a camera 17. The housing 11 is a cylindrical structure with a circular opening on one side and a rectangular box extending from the other side towards the cylindrical sidewall. The flexible contact layer 12 covers the circular opening of the housing 11 and is used to contact objects and generate corresponding deformation. The three RGB light sources 13 are equally spaced and obliquely embedded in the cylindrical sidewall of the housing 11, and can emit red, green, and blue light respectively towards the flexible contact layer 12. The reflective layer 14 covers the side of the flexible contact layer 12 away from the housing 11, enhancing the reflectivity of the flexible contact layer 12, thereby increasing the intensity of the light reflected back after the three RGB light sources 13 illuminate the flexible contact layer 12, and improving the signal-to-noise ratio. The support plate 15, made of transparent acrylic material, is embedded in the outer shell 11 near the opening and supports the flexible contact layer 12. The light-diffusing film 16 is disposed at the light-emitting ends of the three RGB light sources 13 to convert the point light sources of the RGB light sources 13 into surface light sources, expanding the illumination area and preventing bright spots from appearing on the flexible contact layer 12. The light emitted from the three RGB light sources passes through the light-diffusing film and illuminates the flexible contact layer 12. The flexible contact layer 12 reflects a portion of the light back to the camera, while the remaining light passes through the flexible contact layer 12 and illuminates the reflective layer 14 adjacent to it, and is then reflected back to the camera. The camera 17 can capture a pressure map formed by the light reflected back from the flexible contact layer 12 by the three RGB light sources 13.

[0053] Please see Figure 2 This is a schematic diagram of the structure of the object surface morphology and hardness recognition processor 2. Figure 3This is a flowchart of an object surface shape and hardness recognition method implemented based on the aforementioned visual-tactile sensor and object surface shape and hardness recognition processor. The object surface shape and hardness recognition processor 2 is disposed inside a rectangular box extending from the side wall of the housing 11 of the visual-tactile sensor 1 towards the cylinder, and includes a first classification neural network model 21, a gradient calculator 22, a photometric stereo calculator 23, an image processor 24, a matrix stitcher 25, and a second classification neural network model 26.

[0054] The robotic arm and the visual-tactile sensor 1 are used to perform step S1: the robotic arm drives the visual-tactile sensor 1 to press the object to be tested with a fixed displacement D, thereby obtaining a pressure map sequence of the object to be tested during the pressing process.

[0055] Specifically, by setting the motion parameters of the robotic arm, the robotic arm drives the end effector visual-tactile sensor 1 to press the object under test with a fixed displacement D. At this time, the flexible contact layer 12 of the visual-tactile sensor 1 comes into contact with the object under test and deforms. The three RGB light sources 13 of the visual-tactile sensor 1 illuminate the flexible contact layer 12. When the flexible contact layer 12 deforms, the light produces different gradient changes. The camera of the visual-tactile sensor 1 receives the RGB light reflected back from the flexible contact layer 12 in real time and takes images to obtain a pressure map sequence during the pressing process. This image sequence reflects the deformation of the flexible contact layer 12 at the corresponding moment during the pressing process. By pressing the object under test with a fixed displacement, the variables of the object under test when pressed are fixed, which makes it convenient to use the magnitude of the force to reflect its hardness later.

[0056] The object surface morphology and hardness recognition processor 2 is used to execute the relevant steps S2-S5, specifically;

[0057] The first classification neural network model 21 is used to execute step S21A: parse each frame of pressure map sequence obtained in step S1, obtain the magnitude of the force corresponding to each frame of pressure map, and the magnitude sequence of the force corresponding to the pressure map sequence.

[0058] Specifically, the first classification neural network model 21 is first trained through steps SA and SB to obtain the mapping relationship between the pressure map and the magnitude of the force:

[0059] SA: Using a visual-tactile sensor 1 and a force sensor, multiple different standard spheres are pressed at a fixed displacement, and pressure map sequences and force magnitude sequences are collected simultaneously.

[0060] Specifically, the standard sphere is a sphere that is approximately a perfect circle in its free state. The different standard spheres used in step SA have different materials and radii; please refer to [link to relevant documentation]. Figure 4In this embodiment, six different standard spheres were used, with radii ranging from 4mm to 17mm and hardness ranging from 38HA to 100HA, to expand the dataset and improve the robustness of the first classification neural network model 22. During pressing, the flexible contact layer 12 of the visual-tactile sensor 1 is placed upwards on a flat surface, and the standard sphere is gently placed on the flexible contact layer 12. The force sensor is mounted on a vertically movable displacement platform, which presses the standard sphere with a fixed displacement D, causing deformation of the flexible contact layer 12 of the visual-tactile sensor 1. The camera of the visual-tactile sensor 1 captures a pressure map of the flexible contact layer 12 at each frame during the pressing process, and the pressure sensor records the magnitude of the force corresponding to each frame of the pressure map.

[0061] SB: The pressure map sequence and its corresponding stress magnitude sequence are input as a dataset into the first classification neural network model 22 for training to obtain the mapping relationship between the pressure map and the force magnitude. Specifically, the pressure map sequence and its corresponding force magnitude sequence are first divided into a training set and a validation set. Then, the training set and validation set of the pressure map sequence and its corresponding force magnitude sequence are input into the first classification neural network model for training. In this embodiment, the architecture of the first classification neural network model is a ResNet50 network, and finally the mapping relationship between the pressure map and the force magnitude is obtained.

[0062] In step S21A, the pressure map sequence obtained in step S1 is input into the neural network model 22 after training, so that the magnitude of the force corresponding to each frame of pressure map can be obtained, that is, the magnitude sequence of the force when the object under test is pressed can be obtained.

[0063] The gradient calculator 22 is used to perform step S22A: calculate the maximum force F from the force magnitude sequence corresponding to the pressure diagram sequence. max And the gradient of force G.

[0064] Specifically, when the visual-tactile sensor 1 presses the object to be tested, the maximum force occurs at the deepest point when the visual-tactile sensor 1 presses the object to be tested, which is the last frame image captured by the visual-tactile sensor 1 during the pressing process. The maximum force can reflect the hardness of the object to be tested.

[0065] The specific expression for calculating the force gradient is as follows:

[0066]

[0067] In the formula G(F) i F represents the magnitude of the force gradient in the i-th frame of the pressure map. iThis represents the magnitude of the force in the i-th frame of the pressure map. The force gradient represents the rate of change of the force between each frame throughout the entire pressing process, which reflects how quickly the force changes when the object is pressed, and thus reflects the elasticity of the object.

[0068] Please see Figure 5 The photometric stereo calculator 23 is used to perform step S21B: to parse the last frame of the pressure map in the pressure map sequence using the photometric stereo algorithm and draw its three-dimensional point cloud view.

[0069] Specifically, the photometric stereo algorithm processes multiple images of the same object, using different light source angles and intensities to acquire varying photometric information. It then utilizes pixel brightness information from the images to reconstruct the normal vectors and geometric features of the 3D surface, thereby detecting changes in the object's surface shape. Its calculation expression is as follows:

[0070]

[0071] Simplified to: I(x,y)=R(x,y)L(x,y)n(x,y)

[0072] In the formula, I(x,y) represents the brightness value of the three RGB light sources 13 at coordinates (x,y) in the image, R(x,y) represents the reflectivity of the flexible contact layer 12 surface, L(x,y) represents the unit vector of the illumination direction of each RGB light source 13, and n(x,y) represents the normal vector of the 3D surface. I(x,y) is obtained from the captured pressure map; R(x,y) is a fixed value, i.e., the reflectivity of the reflective layer 14; L(x,y) is a fixed value, determined according to the angle between the camera and the three RGB light sources 13. Based on the above formula, n(x,y) can be solved, which is the normal vector of each pixel point that causes deformation of the flexible contact layer 12, and a 3D point cloud view can be drawn based on this normal vector.

[0073] The image processor 24 is used to perform step S22B: calculate the fitting radius R of the three-dimensional point cloud view.

[0074] Specifically, step S22B includes the following steps:

[0075] S22B1: The three-dimensional point cloud view is compressed on a plane parallel to the flexible contact layer 12 to obtain a two-dimensional projection view of the three-dimensional point cloud view. The height information of the three-dimensional point cloud view exists in grayscale form in the two-dimensional projection view.

[0076] S22B2: Noise reduction processing of the two-dimensional projection view using Gaussian filtering. Due to the dark and uneven brightness of the field of view during shooting, and the excessively high operating temperature of the camera during long-term operation, noise is generated in the captured pressure map. Therefore, Gaussian filtering is used to reduce noise in the two-dimensional projection view to further improve the signal-to-noise ratio of the pressure map.

[0077] S22B3: Convert the denoised 2D projection view into an 8-bit grayscale image, and perform binarization thresholding on the grayscale image to obtain a binarized image with approximately circular boundaries.

[0078] Specifically, its calculation expression is:

[0079]

[0080] In the formula, I(x,y) represents the gray value of each pixel in the input grayscale image, and B(x,y) represents the gray value of each pixel in the binarized image after binarization thresholding. Binarizing the original two-dimensional projected view makes its edge point features more prominent, making it easier to calculate the edge shape features at the fitting point.

[0081] S22B4: Perform circle detection on a binarized image with an approximately circular boundary using Hough circle detection to obtain the fitted radius R.

[0082] Specifically, when using Hough circle detection, the first step is to obtain the possible edge points (x, y, y) of the image using the Canny edge detection algorithm. i ,y i Simultaneously, the gradient direction of each edge point is calculated. For each edge point, other edge points are searched within a certain range along the opposite direction of its gradient. If multiple collinear edge points are found, these points are considered to potentially belong to the same circle, and the center of the line connecting these points is taken as a candidate point for the circle's center. For each candidate center point, its distance to all other edge points is calculated, and the distribution of these distance values ​​is statistically analyzed. If a certain distance value appears more than a certain threshold number of times, it is considered to be the radius of the circle. Finally, using the conversion relationship between optical images and actual length units, pixels are converted to millimeters to obtain the actual fitted radius R.

[0083] The matrix splicer 25 is used to perform step S3: to convert the maximum force F max The gradient value of the force G and the computational radius R are concatenated to form a feature matrix {F}. max The feature matrix {F} is defined as follows: {F}, {G}, {R}. maxThe feature matrix {G, R} contains the hardness information and surface morphology information of the object to be tested. Different feature matrices correspond to different hardness information and different surface morphology information of the object to be tested. The hardness information and surface morphology information of the object to be tested can be determined by parsing the parameters of the feature matrix.

[0084] The second classification neural network model 26 is used to perform step S4: to classify the feature matrix {F} max The surface morphology and hardness category P of the object under test are obtained by analyzing G and R through a second classification neural network model.

[0085] Specifically, the second classification neural network model 26 is first trained through the following steps:

[0086] SA': Multiple standard spheres of varying hardness and radii are pressed with a fixed displacement using a visual-tactile sensor 1, and pressure map sequences are simultaneously acquired. Specifically, for each standard sphere, the acquired pressure map sequence is Q. i (T j ), where i represents the i-th standard sphere, j represents the j-th frame, and T j This represents the pressure map captured in frame j.

[0087] SB': After inputting the pressure map sequence into the first classification neural network model 21 trained by steps SA and SB, the magnitude of the force corresponding to each frame of the pressure map sequence is obtained. Specifically, the collected pressure map sequence Q... i (T j Each frame of the ground pressure map T j Inputting the first classification neural network model 21 yields a corresponding force magnitude value F. j Then the sequence of force magnitudes is obtained as Q. i (F j ), where i represents the i-th standard sphere, j represents the j-th frame, F j This represents the magnitude of the force corresponding to the pressure map captured in the j-th frame. Finally, the pressure map sequence is Q. i (T j The sequence of the magnitudes of the forces is Q. i (F j ) splicing together to obtain Q i (T j ,F j ).

[0088] SC': The maximum force F of each standard sphere when it is pressed is calculated by taking each frame of pressure map and its corresponding force through steps S21A, S22A, S21B, and S22B. maxThe gradient value G of the force and the fitting radius R are concatenated in step S3 to form a feature matrix {F}. max For each standard sphere, we obtain a corresponding characteristic matrix Q. i {F max , G, R}, and standardize them to have zero mean and variance.

[0089] SD': For each feature matrix, assign a classification level label P based on the different hardness, softness, and radius of the corresponding standard sphere. i Category rating label P i It reflects the differences in hardness and surface morphology of different objects.

[0090] SE': The feature matrix and its corresponding classification level labels are divided into training and testing sets, which are then input into the second classification neural network model 26 for training to obtain the mapping relationship between the feature matrix and the classification level labels. Specifically, the second classification neural network model 26 is a multilayer perceptron classifier with a hidden layer size of (75, 25). The neurons in the hidden layer can learn complex nonlinear feature combinations of the input data and accurately identify the classification level label of the object to be tested by learning the features in the feature matrix. During training, the learning rate is set to 0.001 to avoid the model failing to converge during training.

[0091] Compared to existing methods for robots to perceive the hardness and surface morphology of objects, this invention designs and uses a visual-tactile sensor. When the flexible contact layer on the sensor comes into contact with an object, a camera captures a pressure map of the layer under pressure. Image processing of this pressure map simultaneously yields information about the object's surface morphology and hardness. Finally, a neural network is used to classify and identify the object. This detection method can identify the hardness and surface morphology of an object with only a single press contact, offering advantages such as high detection speed and high efficiency.

[0092] Based on the same inventive concept, this application also provides an electronic device, which can be a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the object surface morphology and hardness recognition method based on visual-tactile perception, as described in this embodiment of the invention; the memory is used to store a computer program executable by the processor.

[0093] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the aforementioned embodiment of a method for identifying the surface morphology and hardness of an object based on visual-tactile perception. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method for identifying the surface morphology and hardness of an object based on visual-tactile perception as described in any of the above embodiments.

[0094] This application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0095] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A method for recognizing the surface morphology and hardness of an object based on visual-tactile perception, characterized in that: Includes the following steps: S1: Press the object under test with a fixed displacement D using a visual tactile sensor to obtain a pressure map sequence of the object under test during the pressing process; S21A: Obtain the force magnitude sequence corresponding to the pressure map sequence through a first-class neural network model; the first-class neural network model is a ResNet50 network. S22A: Calculate the magnitude of the force corresponding to the pressure map sequence, specifically the maximum force in the sequence. and the gradient of force G; S21B: The last frame of the pressure map in the pressure map sequence is analyzed using a photometric stereo algorithm, and a three-dimensional point cloud view is drawn. S22B: Calculate the fitting radius R of the 3D point cloud view; S3: Maximum force The gradient value of the force G and the computational radius R are concatenated to form a feature matrix { ,G,R}; S4: Transform the characteristic matrix { The surface morphology and hardness category P of the object under test are obtained by parsing through a second classification neural network model, where the second classification neural network model is a multilayer perceptron classifier.

2. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to claim 1, characterized in that: The first classification neural network model in step S21A is trained through the following steps: SA: Using visual and tactile sensors and force sensors, multiple different standard spheres are pressed with a fixed displacement, and pressure map sequences and force magnitude sequences are collected simultaneously. SB: The pressure map sequence and its corresponding stress magnitude sequence are input into the first classification neural network model for training to obtain the mapping relationship between the pressure map and the magnitude of the force.

3. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to claim 2, characterized in that: The photometric stereo algorithm calculation expression in step S21B is as follows: Simplified to: In the formula, Represents the coordinates of the three RGB light sources in the image. The brightness value at that location, Represents the reflectivity of the flexible contact layer surface. A unit vector representing the direction of illumination from each RGB light source. The normal vector representing a 3D surface; Obtained from the captured pressure map; It is a fixed value, namely the reflectivity of the reflective layer; It is a fixed value, determined by the angle between the camera and the three RGB light sources.

4. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to claim 3, characterized in that: Step S22B includes the following steps: S22B1: Compress the three-dimensional point cloud view on a plane parallel to the flexible contact layer to obtain a two-dimensional projection view of the three-dimensional point cloud view. S22B2: Use Gaussian filtering to reduce noise in a two-dimensional projected view; S22B3: Convert the denoised two-dimensional projection view into a grayscale image, and perform binarization thresholding on the grayscale image to obtain a binarized image with approximately circular boundaries; S22B4: Perform circle detection on a binarized image with an approximately circular boundary using Hough circle detection to obtain the fitted radius R.

5. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to any one of claims 2-4, characterized in that: The second classification neural network model in step S4 is trained through the following steps: SA': Pressing multiple standard spheres of different softness and hardness and different radii with a fixed displacement using a visual-tactile sensor and simultaneously collecting pressure map sequences; SB': Input the pressure map sequence into the trained first classification neural network model to obtain the magnitude of the force corresponding to each frame of the pressure map in the pressure map sequence; SC': The maximum force of each standard sphere when it is pressed is calculated by combining the pressure image of each frame with the magnitude of the corresponding force through steps S21A, S22A, S21B, and S22B. The gradient value of the force G and the fitting radius R are concatenated to form a feature matrix { ,G,R}; SD': For each feature matrix, assign a classification level label based on the different hardness, softness, and radius of the corresponding standard sphere; SE': Divide the feature matrix and the corresponding classification level label into a training set and a test set, input them into the second classification neural network model to train it, and obtain the mapping relationship between the feature matrix and the classification level label.

6. A processor for recognizing the surface shape and hardness of an object based on visual-tactile perception, characterized in that: include: The first classification neural network model is used to analyze the force magnitude sequence corresponding to a pressure map sequence; wherein, the pressure map sequence is obtained by pressing the object under test with a visual tactile sensor at a fixed displacement D, and the pressure map sequence of the object under test during the pressing process; the first classification neural network model is a ResNet50 network; Gradient calculator, used to calculate the maximum force in the pressure map sequence. and the gradient of force G; A photometric stereo calculator is used to analyze the last frame of a pressure map in a pressure map sequence using a photometric stereo algorithm and to draw a 3D point cloud view. Image processor used to calculate the fitting radius R of a 3D point cloud view; Matrix splicer, used to combine maximum forces The gradient value of the force G and the computational radius R are concatenated to form a feature matrix { ,G,R}; The second classification neural network model is used to analyze the feature matrix { , G, R}, to obtain the category P of the surface morphology and hardness corresponding to the object, and the second classification neural network model is a multilayer perceptron classifier.

7. The object surface morphology and hardness recognition processor based on visual-tactile perception according to claim 6, characterized in that: The hidden layer size of the second classification neural network model is (75, 25).

8. A system for recognizing the surface shape and hardness of an object based on visual-tactile perception, characterized in that: The device includes a visual-tactile sensor, a robotic arm, and a surface morphology and hardness recognition processor as described in claim 6. The visual-tactile sensor is disposed at the end of the robotic arm. When the robotic arm grasps the object to be tested, it presses the visual-tactile sensor against the object with a fixed displacement D and obtains a pressure map sequence of the object during the pressing process. The surface morphology and hardness recognition processor analyzes the pressure map sequence to obtain the category P of the surface morphology and hardness corresponding to the object to be tested.

9. The object surface morphology and hardness recognition system based on visual-tactile perception according to claim 8, characterized in that: The visual-tactile sensor includes a flexible contact layer, three RGB light sources, and a camera; the flexible contact layer can come into contact with an object and deform; the three RGB light sources illuminate one side of the flexible contact layer from three different angles, and the light is received by the camera after being reflected by the flexible contact layer.

10. The object surface morphology and hardness recognition system based on visual-tactile perception according to claim 9, characterized in that: The visual-tactile sensor also includes a reflective layer and a light-diffusing film. The light emitted from the three RGB light sources shines on the flexible contact layer through the light-diffusing film. The flexible contact layer reflects a portion of the light back to the camera, while the other portion of the light passes through the flexible contact layer and shines on the reflective layer that is in close contact with the flexible contact layer, and is then reflected back to the camera.

Images