Object surface morphology and hardness identification method and system based on visual tactile perception

Abstract

The invention relates to the field of robot tactile perception, in particular to an object surface morphology and hardness identification method and system based on visual tactile perception. Comprising the following steps: S1, pressing a to-be-measured object at a fixed displacement through a visual touch sensor, and obtaining a pressure diagram sequence of the to-be-measured object in the pressing process; s21A: obtaining a magnitude sequence of force corresponding to the pressure graph sequence through a first classification neural network model; s22A: calculating the maximum value of the force and the gradient of the force in the magnitude sequence of the force corresponding to the pressure diagram sequence; s21B, analyzing the last frame of pressure diagram in the pressure diagram sequence through a photometric stereo algorithm, and drawing a three-dimensional point cloud view; s22B, the fitting radius of the three-dimensional point cloud view is calculated; s3, splicing the maximum value of the force, the gradient value of the force and the calculation radius to form a feature matrix; and S4, analyzing the feature matrix through a second classification neural network model to obtain the types of the surface morphology and hardness corresponding to the to-be-detected object.

Description

Technical Field

[0001] The present invention relates to the field of robot tactile perception, and particularly to a method and system for identifying the surface morphology and hardness of an object based on visual-tactile perception. Background Art

[0002] With the rapid development of industrial automation and intelligent robot technology, how to accurately and non-destructively grasp an object has become an important requirement in the field of robotics. When grasping an object, its surface morphology and hardness are the most important physical properties. The surface morphology refers to the visual geometric appearance characteristics of the object surface, mainly referring to the size and local shape of the object. The surface morphology determines how large a mechanical claw the robot needs to use to grasp the object and the grasping method. Hardness refers to the ability of a local part of an object to resist the penetration of a hard object into its surface, which determines how much pressure the robot needs to apply to the object surface to ensure that the object can be smoothly grasped without being damaged.

[0003] An existing method for identifying an object to be grasped by a robot relies on a visual sensor. However, the visual sensor only identifies the object by capturing the surface morphology of the object, and it is difficult to judge the hardness of the object. If the applied force is too large, the object may be damaged, and if the applied force is too small, the clamping may fail. Another existing method for identifying an object to be grasped by a robot relies on a tactile sensor. When detecting an object, the tactile sensor needs complex mechanical movements and repeated contacts with the object to be measured to determine the surface morphology and hardness of the object. The detection efficiency of this identification method is low. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to overcome the defects or deficiencies of the prior art, and provide a method and system for identifying the surface morphology and hardness of an object based on visual-tactile perception.

[0005] A method for identifying the surface morphology and hardness of an object based on visual-tactile perception includes the following steps:

[0006] S1: Press the object to be measured with a fixed displacement D through a visual-tactile sensor to obtain a sequence of pressure maps during the pressing process of the object to be measured;

[0007] S21A: Obtain a sequence of force magnitudes corresponding to the sequence of pressure maps through a first neural network model;

[0008] S22A: Calculate the maximum value F of the force magnitudes in the sequence of force magnitudes corresponding to the pressure map max and the force gradient G;

[0009] S21B: Analyze the last frame in the sequence of pressure maps through a photometric stereo algorithm to obtain a point cloud view, and draw a three-dimensional point cloud view;

[0010] S22B: Calculate the fitting radius R of the three-dimensional point cloud view;

[0011] S3: Combine the maximum value F of the force max , the gradient value G of the force, and the calculated radius R to form a feature matrix {F max , G, R};

[0012] S4: Analyze the feature matrix {F max , G, R} through a second classification neural network model to obtain the category P of the surface morphology and hardness corresponding to the object to be measured.

[0013] Press the object to be measured with a fixed displacement, fixing the variables when the object to be measured is pressed, which is convenient for using the magnitude of the force to characterize its hardness later. The three RGB light sources of the visual tactile sensor irradiate on the flexible contact layer. When the flexible contact layer deforms, the light generates different gradient changes. The camera of the visual tactile sensor receives the RGB light reflected by the flexible contact layer in real time and takes images to obtain a sequence of pressure maps during the pressing process. This image sequence reflects the deformation generated by the flexible contact layer at the corresponding moment during the pressing process. Analyze the maximum value F max , the gradient G of the force, and the calculated radius R of the force during the process of the object being pressed through the deformation of the flexible contact layer, which can reflect the hardness and surface morphology of the object. This detection method can complete the recognition and detection of the hardness and surface morphology of the object only through a single pressing contact, and has the advantages of fast detection speed and high efficiency.

[0014] Further, the first classification neural network model in step S2 is trained through the following steps:

[0015] SA: Press multiple different standard spheres with a fixed displacement through the visual tactile sensor and the force sensor respectively, and synchronously collect a sequence of pressure maps and a sequence of force magnitudes;

[0016] SB: Use the sequence of pressure maps and the corresponding sequence of force magnitudes as a data set to input into the first classification neural network model for training to obtain the mapping relationship between the pressure map and the force magnitude.

[0017] Obtain the sequence of forces and the sequence of pressure maps when pressing multiple different standard spheres through pre-experiments, and then establish the mapping relationship between the pressure map and the force through the first classification neural network model, so that the magnitude value of the force can be directly obtained from the pressure map captured by the camera.

[0018] Further, the calculation expression of the photometric stereo algorithm in step S21B is as follows:

[0019]

[0020] Simplified to: I(x,y)=R(x,y)L(x,y)n(x,y)

[0021] In the formula, I(x,y) represents the brightness value of the three RGB light sources at the coordinate (x,y) in the image, R(x,y) represents the reflectivity of the flexible contact layer surface, L(x,y) represents the unit vector of the irradiation direction of each RGB light source, and n(x,y) represents the normal vector of the 3D surface. I(x,y) is obtained from the captured pressure map; R(x,y) is a fixed value, that is, the reflectivity of the reflective layer; L(x,y) is a fixed value determined by the angle between the camera and the three RGB light sources.

[0022] Further, the step S22B includes the following steps:

[0023] S22B1: compressing the three-dimensional point cloud view on a plane parallel to the flexible contact layer to obtain a two-dimensional projection view of the three-dimensional point cloud view;

[0024] S22B2: Use Gaussian filtering to reduce noise in the 2D projection view;

[0025] S22B3: convert the denoised two-dimensional projection to an 8-bit grayscale image, and perform a binary threshold process on the grayscale image to obtain a binary image with a roughly circular boundary;

[0026] S22B4: Perform circle detection on the binary image with a roughly circular boundary through Hough circle detection, and obtain the fitting radius R.

[0027] Through the above steps, the pressure map captured by the camera is used to calculate the fitting radius through the photometric stereo algorithm and image processing steps, so that the surface morphology of the object can be reflected by the fitting radius.

[0028] Furthermore, the second classification neural network model package in step S4 is trained by the following steps:

[0029] SA': Use a visual-tactile sensor to press multiple standard spheres of different soft and hard materials and different radii with a fixed displacement and simultaneously collect a sequence of pressure images;

[0030] SB': input the pressure map sequence into the trained first classification neural network model to obtain the magnitude of the force corresponding to each frame of the pressure map in the pressure map sequence;

[0031] SC': Calculate the magnitude of the force corresponding to each frame of the pressure graph through the steps S21A, S22A, S21B, S22B to obtain the maximum force F of each standard sphere when it is pressed. max , the force gradient value G and the fitting radius R, and concatenate them to form a feature matrix {F max, G, R};

[0032] SD’: For each feature matrix, assign a classification level label according to the hardness and radius of the corresponding standard sphere.

[0033] SE’: Divide the feature matrix and the corresponding classification level label into a training set and a test set, and input them into the second classification neural network model for training to obtain the mapping relationship between the feature matrix and the classification level label.

[0034] Through a large number of preliminary experiments, the feature matrix {F max , G, R} of different standard spheres when pressed and their corresponding classification level labels of hardness and surface morphology are obtained, and used as a data set to train the second neural network model, so that the second neural network model can fit the corresponding classification level label of the object through the feature matrix {F max , G, R}.

[0035] An object surface morphology and hardness recognition processor based on visual and tactile perception, including a first neural network model for parsing the sequence of force magnitudes corresponding to the pressure map sequence; wherein, the pressure map sequence is obtained by pressing the object to be measured with a fixed displacement D by a visual and tactile sensor, and the pressure map sequence of the object to be measured during the pressing process;

[0036] Gradient calculator for calculating the maximum value F of the force in the pressure map sequence max and the gradient G of the force;

[0037] Photometric stereo calculator for parsing the last frame of the pressure map sequence in the pressure map sequence and drawing a three-dimensional point cloud view;

[0038] Image processor for calculating the fitting radius R of the three-dimensional point cloud view;

[0039] Matrix splicer for splicing the maximum value F of the force max , the gradient value G of the force and the calculated radius R to form a feature matrix {F max , G, R};

[0040] Second classification neural network model for parsing the feature matrix {F max , G, R} to obtain the category P of the surface morphology and hardness of the object.

[0041] Furthermore, the first classification neural network model is a ResNet50 network; the second classification neural network model is a multi-layer perceptron classifier, and its hidden layer size is (75, 25).

[0042] An object surface morphology and hardness recognition system based on visual-tactile perception, comprising a visual-tactile sensor, a manipulator, and the above-mentioned object surface morphology and hardness recognition processor; the visual-tactile sensor is arranged at the end of the manipulator. When the manipulator grasps the object to be measured, the visual-tactile sensor presses the object to be measured with a fixed displacement D and obtains a sequence of pressure maps during the pressing process of the object to be measured; the object surface morphology and hardness recognition processor analyzes the sequence of pressure maps to obtain the category P of the surface morphology and hardness corresponding to the object to be measured.

[0043] Further, the visual-tactile sensor includes a flexible contact layer, three RGB light sources, and a camera; the flexible contact layer can contact the object and deform; the three RGB light sources irradiate light on one surface of the flexible contact layer from three different angles respectively. Thus, different gradient changes can be generated when the three RGB lights irradiated on the flexible contact layer deform the flexible contact layer.

[0044] Further, the visual-tactile sensor further includes a reflective layer and a light homogenizing film. The light emitted by the three RGB light sources irradiates on the flexible contact layer through the light homogenizing film. The flexible contact layer reflects a part of the light back to the camera, and the other part of the light passes through the flexible contact layer and irradiates on the reflective layer closely attached to the flexible contact layer and is reflected back to the camera. The reflective layer can increase the reflectivity of the three RGB light sources irradiated on the flexible contact layer and improve the signal-to-noise ratio of the pressure map signal collected by the camera; the light homogenizing film can change the light emitted by the three RGB light sources from point light sources to surface light sources, making the irradiation more uniform and preventing bright spots from being formed on the flexible contact layer.

[0045] For better understanding and implementation, the present invention will be described in detail below with reference to the accompanying drawings. Description of the Drawings

[0046] Figure 1 It is a structural diagram of the visual-tactile sensor;

[0047] Figure 2 It is a schematic structural diagram of the object surface morphology and hardness recognition processor;

[0048] Figure 3 It is a flowchart of a method for recognizing the surface morphology and hardness of an object based on visual-tactile perception;

[0049] Figure 4 It is a schematic diagram of different standard spheres in the embodiment of the present invention;

[0050] Figure 5 It is a schematic diagram of the pressure map and three-dimensional point cloud view of different standard spheres in the embodiment of the present invention. Detailed Embodiments

[0051] The object surface morphology and hardness recognition system based on visual and tactile perception of the present invention includes a manipulator, a visual and tactile sensor 1 arranged at the end of the manipulator, and an object surface morphology and hardness recognition processor 2. When the manipulator grasps the object to be measured, the visual and tactile sensor 1 is pressed against the object to be measured with a fixed displacement D, and a sequence of pressure maps during the pressing process of the object to be measured is obtained. The object surface morphology and hardness recognition processor analyzes the sequence of pressure maps to obtain the category P of the surface morphology and hardness corresponding to the object to be measured. By using the visual and tactile sensor 1, the surface morphology information and hardness information of the object can be obtained simultaneously through one detection contact, which has the advantages of fast detection speed and high efficiency.

[0052] Specifically, please refer to Figure 1 , which is the structural diagram of the visual and tactile sensor 1. The visual and tactile sensor 1 includes a housing 11, a flexible contact layer 12, three RGB light sources 13, a reflective layer 14, a support plate 15, a light homogenizing film 16, and a camera 17. The housing 11 is a cylindrical structure with a circular opening on one side and a rectangular box extending from the side wall of the cylinder on the other side. The flexible contact layer 12 is covered on the circular opening of the housing 11 and is used to contact the object and generate corresponding deformations. The three RGB light sources 13 are inclined and embedded in the side wall of the cylinder of the housing 11 at equal intervals and can emit red, green, and blue light to the flexible contact layer 12 respectively. The reflective layer 14 is covered on the surface of the flexible contact layer 12 away from the housing 11 and is used to enhance the reflectivity of the flexible contact layer 12, thereby increasing the intensity of the light reflected back after the three RGB light sources 13 irradiate on the flexible contact layer 12 and improving the signal-to-noise ratio. The support plate 15 is embedded at one end of the housing 11 near the opening and is made of transparent acrylic material and is used to support the flexible contact layer 12. The light homogenizing film 16 is arranged at the light exit end of the three RGB light sources 13 to convert the point light sources of the RGB light sources 13 into surface light sources, expand the irradiation area, and prevent bright spots from being generated when the light irradiates on the flexible contact layer 12. The light emitted by the three RGB light sources irradiates on the flexible contact layer 12 through the light homogenizing film. The flexible contact layer 12 reflects a part of the light back to the camera, and the other part of the light passes through the flexible contact layer 12 and irradiates on the reflective layer 14 closely attached to the flexible contact layer 12 and is reflected back to the camera. The camera 17 can capture the pressure map formed by the light reflected back after the three RGB light sources 13 irradiate on the flexible contact layer 12.

[0053] Please refer to Figure 2 , which is the structural schematic diagram of the object surface morphology and hardness recognition processor 2, Figure 3It is a flowchart of an object surface shape and hardness recognition method implemented based on the visual-tactile sensor and the object surface shape and hardness recognition processor. The object surface shape and hardness recognition processor 2 is arranged inside a rectangular box body extending from the side wall of a cylindrical body on one side of the housing 11 of the visual-tactile sensor 1, and it includes a first classification neural network model 21, a gradient calculator 22, a photometric stereo calculator 23, an image processor 24, a matrix splicer 25, and a second classification neural network model 26.

[0054] The manipulator and the visual-tactile sensor 1 are used to execute step S1: driving the visual-tactile sensor 1 by the manipulator to press the object to be measured with a fixed displacement D, and obtaining a sequence of pressure maps during the pressing of the object to be measured.

[0055] Specifically, by setting the motion parameters of the manipulator, the manipulator drives the visual-tactile sensor 1 at the end to press the object to be measured with a fixed displacement D. At this time, the flexible contact layer 12 of the visual-tactile sensor 1 comes into contact with the object to be measured and deforms. The three RGB light sources 13 of the visual-tactile sensor 1 irradiate on the flexible contact layer 12. When the flexible contact layer 12 deforms, different gradient changes occur in the light. The camera of the visual-tactile sensor 1 receives the RGB light reflected by the flexible contact layer 12 in real time and takes images to obtain a sequence of pressure maps during the pressing process. This image sequence reflects the deformation generated by the flexible contact layer 12 at the corresponding moments during the pressing process. Pressing the object to be measured with a fixed displacement fixes the variables when the object to be measured is pressed, which is convenient for using the magnitude of the force to reflect its hardness later.

[0056] The object surface shape and hardness recognition processor 2 is used to execute steps S2 - S5, specifically;

[0057] The first classification neural network model 21 is used to execute step S21A: analyzing each frame of the pressure map sequence obtained in step S1, obtaining the magnitude of the force corresponding to each frame of the pressure map, and the sequence of the magnitudes of the forces corresponding to the pressure map sequence.

[0058] Specifically, the first classification neural network model 21 is first trained through steps SA and SB to obtain the mapping relationship between the pressure map and the magnitude of the force:

[0059] SA: Pressing multiple different standard spheres with a fixed displacement by the visual-tactile sensor 1 and the force sensor respectively, and synchronously collecting the sequence of pressure maps and the sequence of the magnitudes of the forces.

[0060] Specifically, the standard sphere is a sphere that is approximately a perfect circle in the free state. The hard and soft materials and radii of the different standard spheres used in step SA are different. Please refer to Figure 4, in this embodiment, 6 different standard spheres are used respectively, with a radius range between 4 mm and 17 mm and a hardness range between 38 HA and 100 HA, to expand the dataset and improve the robustness of the first classification neural network model 22. When pressing, the flexible contact layer 12 of the visual tactile sensor 1 is placed face up on a plane, and the standard sphere is gently placed on the flexible contact layer 12; the force sensor is installed on a vertically movable displacement platform, and the displacement platform presses the standard sphere with a fixed displacement D against the force sensor, causing the flexible contact layer 12 of the visual tactile sensor 1 to deform; the camera of the visual tactile sensor 1 captures the pressure map generated by each frame of the flexible contact layer 12 during the pressing process, and the pressure sensor records the magnitude of the force corresponding to each frame of the pressure map.

[0061] SB: The pressure map sequence and the sequence of the magnitudes of the corresponding forces are used as a dataset to be input into the first classification neural network model 22 for training to obtain the mapping relationship between the pressure map and the magnitude of the force. Specifically, first, the pressure map sequence and the sequence of the magnitudes of the corresponding forces are divided into a training set and a validation set, and then the training set and the validation set of the pressure map sequence and the sequence of the magnitudes of the corresponding forces are input into the first classification neural network model for training. In this embodiment, the architecture of the first classification neural network model is the ResNet50 network, and finally the mapping relationship between the pressure map and the magnitude of the force is obtained.

[0062] In the step S21A, after the pressure map sequence obtained in the step S1 is input into the trained neural network model 22, the magnitude of the force corresponding to each frame of the pressure map can be obtained, that is, the sequence of the magnitudes of the forces when the object to be measured is pressed can be obtained.

[0063] The gradient calculator 22 is used to execute the step S22A: calculate the maximum value F of the force from the sequence of the magnitudes of the forces corresponding to the pressure map sequence max and the gradient G of the force.

[0064] Specifically, when the object to be measured is pressed through the visual tactile sensor 1, the maximum value of the force appears at the deepest position when the visual tactile sensor 1 presses the object to be measured, that is, the last frame of the picture collected by the visual tactile sensor 1 during the pressing process. The maximum value of the force can reflect the hardness of the object to be measured.

[0065] The specific calculation expression of the gradient of the force is as follows:

[0066]

[0067] In the formula, G(F i ) represents the magnitude of the force gradient at the i-th frame of the pressure map, F irepresents the magnitude value of the force at the pressure map of the i-th frame. The force gradient represents the rate of change of the force magnitude between each frame during the entire pressing process, that is, it can reflect how fast the force changes when the object to be measured is pressed, and thus can reflect the elasticity of the object.

[0068] Please refer to Figure 5 , the photometric stereo calculator 23 is used to execute step S21B: Parse the last pressure map in the pressure map sequence through the photometric stereo algorithm and draw its three-dimensional point cloud view.

[0069] Specifically, the photometric stereo algorithm processes multiple images of the same object, uses different light source angles and intensities to obtain different photometric information, and then uses the pixel brightness information in the images to reconstruct the normal vectors and geometric features of the three-dimensional surface, thereby detecting the change state of the object surface shape. Its calculation expression is as follows:

[0070]

[0071] Simplified to: I(x,y) = R(x,y)L(x,y)n(x,y)

[0072] In the formula, I(x,y) represents the brightness value at the coordinate (x,y) in the image of the three RGB light sources 13, R(x,y) represents the reflectivity of the surface of the flexible contact layer 12, L(x,y) represents the unit vector of the irradiation direction of each RGB light source 13, and n(x,y) represents the normal vector of the three-dimensional surface. I(x,y) is obtained from the captured pressure map; R(x,y) is a fixed value, that is, the reflectivity of the reflective layer 14; L(x,y) is a fixed value, which is determined according to the angles between the camera and the three RGB light sources 13. According to the above formula, n(x,y) can be solved, which is the normal vector of each pixel point where the flexible contact layer 12 deforms, and the three-dimensional point cloud view can be drawn based on this normal vector.

[0073] The image processor 24 is used to execute step S22B: Calculate the fitting radius R of the three-dimensional point cloud view.

[0074] Specifically, step S22B includes the following steps:

[0075] S22B1: Compress the three-dimensional point cloud view on a plane parallel to the flexible contact layer 12 to obtain a two-dimensional projection view of the three-dimensional point cloud view. The height information of the three-dimensional point cloud view exists in the form of grayscale in the two-dimensional projection view.

[0076] S22B2: Denoise the two-dimensional projection view using Gaussian filtering. Since the field of view is dark and the brightness is uneven during camera shooting, and the working temperature of the camera is too high for a long time, noise will be generated on the captured pressure map. Therefore, Gaussian filtering is used to denoise the two-dimensional projection view to further improve the signal-to-noise ratio of the pressure map.

[0077] S22B3: Convert the denoised two-dimensional projection view into an 8-bit grayscale image, and perform binary thresholding on the grayscale image to obtain a binary image with a roughly circular boundary.

[0078] Specifically, its calculation expression is:

[0079]

[0080] In the formula, I(x, y) represents the grayscale value of each pixel point in the input grayscale image, and B(x, y) represents the grayscale value of each pixel point in the binary image after binary thresholding. After binarizing the original two-dimensional projection view, the edge point features become more prominent, making it easier to calculate and fit its edge shape features.

[0081] S22B4: Perform circular detection on the binary image with a roughly circular boundary through Hough circle detection to obtain the fitted radius R.

[0082] Specifically, when using Hough circle detection, first obtain the possible edge points (x i , y i ) of the image through the canny edge detection algorithm, and calculate the gradient direction of each edge point at the same time. For each edge point, search for other edge points within a certain range along the opposite direction of its gradient. If multiple collinear edge points are found, it is considered that these points may belong to the same circle, and the center of the line connecting these points is used as a candidate for the center of the circle. For each candidate center of the circle, calculate its distance to all other edge points, and statistically analyze the distribution of the distance values. If the number of occurrences of a certain distance value exceeds a certain threshold, then this distance value is considered as the radius of the circle. Finally, using the conversion relationship between the optical image and the actual length unit, convert between pixels and millimeters to obtain the actual fitted radius R.

[0083] The matrix splicer 25 is used to execute step S3: splice the maximum force F max , the force gradient value G, and the calculated radius R to form a feature matrix {F max , G, R}. The feature matrix {F max, G, R} contains the hardness information and surface morphology information of the object to be measured. The characteristic matrices corresponding to objects to be measured with different hardness information and different surface morphology information are different. The hardness information and surface morphology information of the object to be measured can be determined by analyzing the parameters of the characteristic matrix.

[0084] The second classification neural network model 26 is used to execute step S4: parsing the characteristic matrix {F max , G, R} through a second classification neural network model to obtain the categories P of the surface morphology and hardness corresponding to the object to be measured.

[0085] Specifically, the second classification neural network model 26 first performs model training through the following steps:

[0086] SA’: Press a plurality of standard spheres with different hardnesses and different radii at a fixed displacement by the visual-tactile sensor 1 and synchronously collect a sequence of pressure maps. Specifically, for each standard sphere, the collected sequence of pressure maps is Q i (T j ), where i represents the i-th standard sphere, j represents the j-th frame, and T j represents the pressure map captured in the j-th frame.

[0087] SB’: After inputting the sequence of pressure maps into the first classification neural network model 21 trained through the above steps SA and SB, the magnitude of the force corresponding to each frame of the pressure map in the sequence of pressure maps is obtained. Specifically, input each frame of the pressure map T i (T j ) in the collected sequence of pressure maps Q j into the first classification neural network model 21 to obtain a corresponding force magnitude value F j , and then obtain a sequence of force magnitudes as Q i (F j ), where i represents the i-th standard sphere, j represents the j-th frame, F j represents the magnitude of the force corresponding to the pressure map captured in the j-th frame. Finally, splice the sequence of pressure maps Q i (T j ) and the sequence of force magnitudes Q i (F j ) to obtain Q i (T j , F j ).

[0088] SC’: Calculate the maximum value F of the force when each standard sphere is pressed through the above steps S21A, S22A, S21B, and S22B for each frame of the pressure map and its corresponding force magnitude max, the gradient value G of the force and the fitting radius R, and splice them through the step S3 to form a feature matrix {F max , G, R}. For each standard sphere, a corresponding feature matrix Q i {F max , G, R} is obtained, and it is normalized to have zero mean and variance.

[0089] SD’: For each feature matrix, according to the different hardness materials and radii of the corresponding standard spheres, a classification level label P i is assigned. The classification level label P i reflects the differences in hardness and surface morphology of different objects.

[0090] SE’: Divide the feature matrix and the corresponding classification level label into a training set and a test set, and input them into the second classification neural network model 26 for training to obtain the mapping relationship between the feature matrix and the classification level label. Specifically, the second classification neural network model 26 is a multi-layer perceptron classifier, and the size of its hidden layer is (75, 25). The neurons in the hidden layer can learn complex non-linear feature combinations of the input data, and accurately identify the classification level label of the object to be measured by learning the features in the feature matrix. During the training process, the learning rate is set to 0.001 to avoid the problem that the model cannot converge during the training process.

[0091] Compared with the existing methods for robots to perceive the hardness and surface morphology of objects, the present invention designs and uses a visual-tactile sensor. When the flexible contact layer on the visual-tactile sensor contacts an object, the camera can capture the pressure map generated after the flexible contact layer is pressed. Image processing is performed on the pressure map to obtain the surface morphology information and hardness information of the object at the same time. Finally, the object is classified and identified through a neural network. This detection method can complete the recognition and detection of the hardness and surface morphology of the object only through a single pressing contact, and has the advantages of fast detection speed and high efficiency.

[0092] Based on the same inventive concept, the present application also provides an electronic device, which can be a terminal device such as a server, a desktop computing device or a mobile computing device (for example, a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the method for identifying the surface morphology and hardness of an object based on visual-tactile perception in the embodiments of the present invention; the memory is used to store a computer program executable by the processor.

[0093] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiments of the foregoing method for identifying the surface morphology and hardness of an object based on visual and tactile perception. The computer-readable storage medium stores a computer program, and when the program is executed by a processor, it implements the steps of the method for identifying the surface morphology and hardness of an object based on visual and tactile perception recorded in any of the foregoing embodiments.

[0094] The present application may be in the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0095] The above-described embodiments merely represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the inventive concept of the present invention, several modifications and improvements can still be made, and the present invention also intends to include these modifications and improvements.

Claims

1. A method for identifying the surface shape and hardness of an object based on visual and tactile perception, characterized in that: Including the following steps: S1: Press the object to be measured with a fixed displacement D through a visual-tactile sensor to obtain a sequence of pressure maps during the pressing of the object to be measured; S21A: Obtain a sequence of force magnitudes corresponding to the sequence of pressure maps through a first classification neural network model; S22A: Calculate the maximum value F of the forces in the sequence of force magnitudes corresponding to the pressure map sequence max and the force gradient G; S21B: Analyze the last frame of the pressure map sequence through a photometric stereo algorithm and draw a three-dimensional point cloud view; S22B: Calculate the fitting radius R of the three-dimensional point cloud view; S3: Concatenate the maximum value F of the force max , the gradient value G of the force, and the calculation radius R to form a feature matrix {F max , G, R}; S4: Parse the feature matrices {F max , G, R} through a second classification neural network model to obtain the categories P of the surface morphology and hardness corresponding to the object to be measured.

2. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to claim 1, wherein: The first classification neural network model in step S2 is trained through the following steps: SA: Press multiple different standard spheres with a fixed displacement through a visual-tactile sensor and a force sensor respectively, and synchronously collect a sequence of pressure maps and a sequence of force magnitudes; SB: Use the sequence of pressure maps and the corresponding sequence of force magnitudes as a data set to input into the first classification neural network model for training to obtain the mapping relationship between the pressure map and the force magnitude.

3. The method for identifying the surface shape and hardness of an object based on visual-tactile perception according to claim 2, wherein: The calculation expression of the photometric stereo algorithm in step S21B is as follows: Simplified to: I(x,y) = R(x,y)L(x,y)n(x,y) In the formula, I(x,y) represents the brightness value at the coordinate (x,y) in the image of the three RGB light sources, R(x,y) represents the reflectivity of the surface of the flexible contact layer, L(x,y) represents the unit vector of the irradiation direction of each RGB light source, and n(x,y) represents the normal vector of the 3D surface. I(x,y) is obtained from the captured pressure map; R(x,y) is a fixed value, that is, the reflectivity of the reflection layer; L(x,y) is a fixed value determined by the angle between the camera and the three RGB light sources.

4. The method for identifying the surface shape and hardness of an object based on visual and tactile perception according to claim 3, wherein: Step S22B includes the following steps: S22B1: Compress the three-dimensional point cloud view on a plane parallel to the flexible contact layer to obtain a two-dimensional projection view of the three-dimensional point cloud view; S22B2: Denoise the two-dimensional projection view with Gaussian filtering; S22B3: Convert the denoised two-dimensional projection view into a grayscale image and perform binary threshold processing on the grayscale image to obtain a binary image with a roughly circular boundary; S22B4: Perform circular detection on the binary image with a roughly circular boundary through Hough circle detection to obtain the fitting radius R.

5. The method for identifying the surface morphology and hardness of an object based on visual-tactile perception according to any one of claims 2-4, characterized in that: The second classification neural network model in step S4 is trained through the following steps: SA’: Press multiple standard spheres with different soft and hard materials and different radii with a fixed displacement through a visual-tactile sensor respectively and synchronously collect a sequence of pressure maps; SB’: Input the sequence of pressure maps into the trained first classification neural network model to obtain the magnitude of the force corresponding to each frame of the pressure map in the sequence of pressure maps; SC’: Calculate the maximum value F of the force, the gradient value G of the force, and the fitting radius R of each standard sphere when pressed by calculating each frame of the pressure map and the corresponding magnitude of the force through the steps S21A, S22A, S21B, and S22B, and splice them to form a feature matrix {F max , G, R}; max ​ SD’: For each feature matrix, assign a classification level label according to the different soft and hard materials and radii of the corresponding standard sphere; SE’: Divide the feature matrix and the corresponding classification level label into a training set and a test set, input them into the second classification neural network model for training to obtain the mapping relationship between the feature matrix and the classification level label.

6. An object surface morphology and hardness recognition processor based on visual and tactile perception, characterized in that: Including: A first neural network model for parsing a sequence of magnitudes of forces corresponding to a sequence of pressure maps; wherein the sequence of pressure maps is a sequence of pressure maps obtained by a visual-tactile sensor pressing a to-be-tested object with a fixed displacement D during the pressing process of the to-be-tested object. Gradient calculator for calculating the maximum value F of the force in the pressure map sequence max and the gradient G of the force; A photometric stereo calculator for parsing the last frame of the pressure map sequence by a photometric stereo algorithm and drawing a three-dimensional point cloud view. An image processor for calculating a fitting radius R of the three-dimensional point cloud view. Matrix splicer, used to splice the maximum value of force F nax , the gradient value G of force, and the calculation radius R to splice and form a feature matrix {F max , G, R}; The second classification neural network model is used to parse the feature matrices {F max , G, R} to obtain the categories P of the surface morphology and hardness corresponding to the object.

7. The object surface shape and hardness recognition processor based on visual and tactile perception according to claim 6, characterized in that: The first classification neural network model is a ResNet50 network; the second classification neural network model is a multi-layer perceptron classifier with a hidden layer size of (75, 25).

8. An object surface morphology and hardness recognition system based on visual and tactile perception, characterized in that: It includes a visual-tactile sensor, a manipulator, and an object surface morphology and hardness recognition processor as described in claim 6; the visual-tactile sensor is arranged at the end of the manipulator. When the manipulator grabs the to-be-tested object, the visual-tactile sensor presses the to-be-tested object with a fixed displacement D and obtains a sequence of pressure maps during the pressing process of the to-be-tested object; the object surface morphology and hardness recognition processor analyzes the sequence of pressure maps to obtain the category P of the surface morphology and hardness corresponding to the to-be-tested object.

9. The object surface morphology and hardness recognition system based on visual-tactile perception according to claim 8, characterized in that: The visual-tactile sensor includes a flexible contact layer, three RGB light sources, and a camera; the flexible contact layer can contact an object and deform; the three RGB light sources respectively irradiate light on one surface of the flexible contact layer from three different angles, and the light is received by the camera after being reflected by the flexible contact layer.

10. The object surface morphology and hardness recognition system based on visual and tactile perception according to claim 8, characterized in that: The visual-tactile sensor further includes a reflective layer and a light homogenizing film. The light emitted by the three RGB light sources irradiates on the flexible contact layer through the light homogenizing film. The flexible contact layer reflects a part of the light back to the camera, and the other part of the light passes through the flexible contact layer and irradiates on the reflective layer closely attached to the flexible contact layer and is reflected back to the camera.

Images