A skin force deformation measurement system, method and apparatus

Abstract

The application discloses a skin stress deformation measurement system, method and device, wherein the system comprises: an executor end provided with a transparent pressing part, the pressing part is used for pressing the skin and moving according to a preset direction; a micro camera arranged in the executor end and collecting a first image through the pressing part; a stereo vision camera installed outside the executor end and used for collecting a second image on a skin stretching area and / or a skin accumulation area; and a data processing module, which obtains three-dimensional coordinates of skin particles in an effective contact area according to the first image, obtains three-dimensional coordinates of skin particles in the skin stretching area or the skin accumulation area according to the second image, obtains a deformation condition of the skin according to the obtained three-dimensional coordinates of the skin particles, and performs three-dimensional estimation on the skin stress deformation. The application performs three-dimensional estimation on the deformed skin, provides more accurate data for a skin contact force model, and can be applied to the field of skin stress deformation measurement.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of skin deformation measurement, in particular to a skin deformation measurement system, method and device. BACKGROUND

[0002] The instruments or robots that contact human skin during operation are mainly applied in the fields of rehabilitation medicine, old people care, massage health care, beauty care, etc., and they have the common feature of maintaining a certain force contact during movement on the skin surface.

[0003] The skin contains a large amount of nerve tissue, the normal load produces downward compression deformation, and the adhesion friction generated between the skin and the contact surface causes the skin to stretch and accumulate in the tangential direction, which will trigger the tactile perception of compression, extrusion and traction. Large skin deformation will cause different degrees of discomfort in tactile perception, and in severe cases, it will cause pain and even damage the skin. Therefore, studying skin deformation under different operating environments helps to further study the relationship between skin deformation and human tactile perception, and ensures the safety and comfort of the human body during operation.

[0004] However, the skin area directly contacted by massage is often blocked by the end effector of the massage robot, so the actual compression state of the skin cannot be estimated directly through external vision; and in the current field of three-dimensional estimation of skin deformation under force, many studies are limited to the study of two-dimensional stretching deformation of the skin caused by pressing movement, such as the in-situ visualization experiment research and analysis of human skin friction behavior proposed by Guan Ziyin. The skin speckle fixed on one side of the arm has limitations, one is that it can only estimate the two-dimensional stretching of the skin, and two is that as the effector moves, the speckle far from the effector cannot estimate the actual stretching deformation of the skin.

[0005] In order to obtain the real skin deformation, the end effector is required to be in direct contact with the skin, such as the invention patent with the application publication number CN201811005278.2, which performs a labeling operation on the skin surface deformation detection, affecting the contact properties of the skin and cannot accurately detect the accuracy of the skin material.

[0006] The above studies on skin deformation estimation rarely estimate the deformation of the effective contact area, the skin stretching area and the skin accumulation area in three dimensions.

[0007] In addition, the skin deformation is frequent during operation, and the algorithm efficiency is very high. In the past, the region detection algorithm used a fully connected layer after feature convolution, and a large number of output parameters reduced the efficiency of the algorithm. For example, the paper "Image Semantic Segmentation Based on Convolutional Neural Network" proposed by Chen Hongxiang uses a fully connected layer after feature convolution, which reduces the efficiency of region detection.

[0008] Therefore, in order to meet the requirements of skin stress deformation estimation, it is necessary to solve the problems of end effector occlusion of the effective contact area of ​​skin massage, the computational efficiency of the effective contact area detection algorithm, and the three-dimensional estimation of the skin accumulation area and the skin stretching area. Summary of the Invention

[0009] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a measurement system, method and device for skin stress deformation.

[0010] The technical solution adopted in this invention is:

[0011] A system for measuring skin deformation under stress, comprising:

[0012] At the end of the actuator, there is a transparent pressing part, which is used to press the skin and move in a preset direction; wherein the skin is provided with a feature pattern including multiple skin particles;

[0013] A miniature camera is disposed within the end of the actuator and captures a first image through the pressing portion;

[0014] A stereo vision camera, mounted externally at the end of the actuator, is used to acquire a second image of the skin stretching area and / or the skin accumulation area; the skin stretching area is the region behind the end of the actuator in the direction of movement, and the skin accumulation area is the region in front of the end of the actuator in the direction of movement.

[0015] The data processing module obtains the effective contact area between the pressing part and the skin based on the first image, obtains the three-dimensional coordinates of the skin particles in the effective contact area based on the first image, obtains the three-dimensional coordinates of the skin particles in the skin stretching area or skin accumulation area based on the second image, obtains the skin deformation based on the obtained three-dimensional coordinates of the skin particles, and performs three-dimensional estimation of the skin deformation under force.

[0016] Another technical solution adopted in this invention is:

[0017] A method for measuring skin deformation under stress, comprising the following steps:

[0018] Design a feature pattern in the skin treatment area, the feature pattern including multiple skin particles P = {P1, P2…P} i …P num}, where num is the number of skin particles in the skin treatment area;

[0019] Based on the pressure applied, the skin is divided into effective contact area, skin accumulation area, and skin stretching area;

[0020] A first image is acquired using a miniature camera, and the effective contact area is determined based on the first image.

[0021] The three-dimensional coordinates of skin particles in the effective contact area are obtained from the first image.

[0022] A second image is acquired on the skin accumulation area and / or skin stretching area using a stereo vision camera, and the three-dimensional coordinates of skin particles in the skin accumulation area and / or skin stretching area are obtained based on the second image.

[0023] The deformation of the skin is obtained by acquiring the three-dimensional coordinates of the skin particles, and the three-dimensional estimation of the skin's stress deformation is performed.

[0024] Furthermore, the first image is a color image.

[0025] The second image includes a color image. With depth images

[0026] Where ti represents time ti, This represents the color image captured by the stereo vision camera at time ti. This represents the color image captured by the miniature camera at time ti. This represents the depth image captured by the stereo vision camera at time ti.

[0027] Further, obtaining the effective contact area based on the first image includes:

[0028] The first image is input into the trained region recognition model, which outputs a feature map marked with the effective contact area.

[0029] The training set used to train the region recognition model is obtained in the following way:

[0030] The effective contact area is marked by a masking method to obtain a mask image (Mask), which is used as a training sample. The training set is then obtained based on the training sample.

[0031] Furthermore, the region recognition model is trained in the following manner:

[0032] The sample image Pic2 is scaled and normalized to obtain the scaled and normalized image Pic′2.

[0033] The scaled and normalized image Pic′2 is input into the feature extraction network to obtain deep information of the image;

[0034] The deep information layer N is input into the category extraction network, and the deep information layer N is convolved into the category information layer Cls through upsampling, downsampling and convolution.

[0035] Perform bilinear interpolation on the category information layer Cls to obtain the output layer Out. The value of each layer of the output layer Out represents the probability of that category. Output the range of the actual pressing area based on the reference probability λ.

[0036] During network training, positive sample data for training is selected from a large number of negative samples output by the network using a positive sample matching method, and then a loss function is constructed for regression adjustment of the convolution kernel coefficients.

[0037] Furthermore, the loss function Loss consists of the cross-entropy loss between the output layer Out and the labeled mask image Mask;

[0038] The expression for the loss function formula, Loss, is:

[0039]

[0040]

[0041]

[0042] Where N is the number of parameters in the output layer Out, L0 is the total probability of the effective contact area, and L1 is the total probability of the background. This refers to the probability that the i-th parameter of the output layer (Out) is a valid contact region. This refers to the probability that the i-th parameter of the output layer Out is the background.

[0043] Further, obtaining the three-dimensional coordinates of skin particles in the skin accumulation region and / or skin stretching region based on the second image includes:

[0044] Based on the second image, identify skin particles in the skin accumulation area and / or skin stretching area, and obtain the two-dimensional image matching relationship of the skin particles;

[0045] Establish a base coordinate system T0 on the actuator base, and use the depth image acquired by the stereo vision camera at time ti. The depth information of skin particles in the skin accumulation region and / or skin stretching region at time ti is obtained by using a depth-first search algorithm and 3D mapping.

[0046] Based on the calibration relationship between the actuator and the stereo vision camera Obtain the three-dimensional coordinates of skin particles in the skin accumulation region and / or skin stretching region at time ti in the actuator coordinate system.

[0047] The three-dimensional coordinates of each skin particle at time ti are obtained based on the two-dimensional image matching relationship of skin particle P.

[0048] Further, obtaining the three-dimensional coordinates of skin particles in the effective contact area based on the first image includes:

[0049] Based on the first image, identify skin particles in the effective contact area and obtain the two-dimensional image matching relationship of the skin particles;

[0050] Based on the actuator's device dimensions and 3D mapping, the depth information of skin particles in the effective contact area at time ti is obtained.

[0051] Based on the calibration relationship between the actuator and the miniature camera Obtain the three-dimensional coordinates of the skin mass point in the effective contact area at time ti in the actuator coordinate system.

[0052] The three-dimensional coordinates of each skin particle at time ti are obtained based on the two-dimensional image matching relationship of skin particle P.

[0053] Where n i n represents the number of skin particles identified at time ti. i ≤num.

[0054] Further, the step of identifying skin particles in the effective contact area based on the first image and obtaining the two-dimensional image matching relationship of the skin particles includes:

[0055] Color images acquired by a miniature camera at time ti Perform grayscale histogram analysis and binarize the grayscale histogram.

[0056] Filtering color images Contours whose area is greater than the preset pixel threshold S after binarization

[0057] Based on the line segment extraction algorithm, the contour is... The process is performed to obtain contour line segment K2, which is composed of continuous contour line segment points;

[0058] Based on the nine-point scanning method, obtain the mass point Ep within the contour line segment K2. ti and point mass Ep ti In color images Two-dimensional coordinates on;

[0059] The mass Ep will be moved according to the direction of the actuator's movement. ti With point mass Ep t(i-1) Perform matching to obtain the point mass Ep ti A one-to-one correspondence with skin particles P.

[0060] Another technical solution adopted in this invention is:

[0061] A device for measuring skin deformation under stress, comprising:

[0062] At least one processor;

[0063] At least one memory for storing at least one program;

[0064] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0065] The beneficial effects of this invention are: when the actuator is used to press the skin, the effective contact area, skin accumulation area and skin stretching area between the actuator and the skin caused by the deformation of the skin under force are estimated in three dimensions by means of a measuring device and a three-dimensional estimation method, so as to provide more accurate data for the skin contact force model. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of the connection of a skin stress deformation measurement system according to an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the region recognition convolutional neural network device in an embodiment of the present invention;

[0069] Figure 3 This is a flowchart of a method for measuring skin deformation under stress according to an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of the structure of a skin stress deformation measurement system according to an embodiment of the present invention;

[0071] Figure 5 These are the feature patterns and skin pressure maps in the stereoscopic vision camera images in this embodiment of the invention;

[0072] Figure 6 This is a region division diagram in the image of the miniature camera in an embodiment of the present invention;

[0073] Figure 7 This is a time-displacement diagram of skin particles in an embodiment of the present invention;

[0074] Figure 8 This is a three-dimensional estimation diagram of skin deformation under pressure in an embodiment of the present invention.

[0075] Figure label:

[0076] Figure 4 In the diagram: 1-Miniature camera, 2-Stereo vision camera (installation position for measuring the stacked area), 3-Camera adapter (installation position for measuring the stacked area), 4-Actuator end effector, 5-Human skin, 6-Visual range of the miniature camera, 7-Visual range of the stereo vision camera, 8-Skin stretching area, 9-Effective contact area, 10-Skin stacking area, 11-Actuator movement direction, 12-Stereo vision camera (installation position for measuring the stretched area), 13-Camera adapter (installation position for measuring the stretched area).

[0077] Figure 5 In the middle: 6-Vision range of the miniature camera, 14-Feature pattern, 15-Guiding operation trajectory, 16-Skin texture point.

[0078] Figure 6 In the middle: 4-actuator end, 9-effective contact area, 10-skin accumulation area. Detailed Implementation

[0079] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0080] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0081] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0082] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0083] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0084] See Figure 1 and Figure 4 This embodiment provides a system for measuring skin deformation under stress, including:

[0085] The actuator end 4 is provided with a transparent pressing part, which is used to press the skin and move according to a preset direction; wherein the skin is provided with a feature pattern including multiple skin particles;

[0086] A miniature camera 1 is disposed inside the end of the actuator and acquires a first image through the pressing portion;

[0087] A stereo vision camera 2 is mounted on the outside of the actuator end and is used to acquire a second image on the skin stretching area and / or skin accumulation area; the skin stretching area is the area behind the actuator end in the direction of movement, and the skin accumulation area is the area in front of the actuator end in the direction of movement.

[0088] Data processing module (i.e.) Figure 1 The computer in the image obtains the effective contact area between the pressing part and the skin based on the first image, obtains the three-dimensional coordinates of the skin particles in the effective contact area based on the first image, obtains the three-dimensional coordinates of the skin particles in the skin stretching area or skin accumulation area based on the second image, obtains the skin deformation based on the obtained three-dimensional coordinates of the skin particles, and performs three-dimensional estimation of the skin deformation under force.

[0089] In this embodiment, the actuator will configure the stereo vision camera 2 according to two working conditions: detecting skin stretching areas and skin accumulation areas. A pressing operation will be performed on the subject's skin 5, requiring a three-dimensional estimation of the deformation of the subject's skin after pressure. The number of stereo vision cameras can be one or two. If two are used, they are symmetrically positioned on both sides of the actuator's end effector 4, one for acquiring images of the skin stretching area and the other for acquiring images of the skin accumulation area. If only one is used, the position of the stereo vision camera will be changed to acquire images of either the skin stretching or skin accumulation area.

[0090] When measuring the skin accumulation area, the stereo vision camera 2 is fixed in front of the transparent actuator 4 via the camera external connector 3, allowing observation of the skin accumulation area. When measuring the skin stretching area, the stereo vision camera 12 is fixed behind the transparent actuator via the camera external connector 13, allowing observation of the skin stretching area. Here, "in front" refers to the area in front of the actuator's movement direction, and "behind" refers to the area behind the actuator's movement direction. As an optional implementation, the miniature camera 3 is fixed inside the transparent actuator 4 via the camera external connector 3. The method for generating the guided work trajectory 15 is not the focus of this application and will not be described in detail here.

[0091] After the miniature camera 1 acquires the first image and the stereo vision camera 2 acquires the second image, the acquired images are sent to the computer. The computer obtains the effective contact area between the pressing part and the skin based on the first image, obtains the three-dimensional coordinates of the skin particles in the effective contact area based on the first image, obtains the three-dimensional coordinates of the skin particles in the skin stretching area or skin accumulation area based on the second image, obtains the skin deformation based on the obtained three-dimensional coordinates of the skin particles, and performs three-dimensional estimation of the skin deformation under force.

[0092] Based on the system described above, see Figure 3 This embodiment also provides a method for measuring skin deformation under stress, including the following steps:

[0093] S1. Design a feature pattern in the skin treatment area, the feature pattern including multiple skin particles P = {P1, P2…P} i …P num}, where num is the number of skin particles in the skin treatment area.

[0094] Step S1 includes steps S11-S13:

[0095] S11. Design feature patterns based on the grid marking method. See [link / reference] Figure 5 The grid marking method refers to the fact that the shape of the feature pattern should show a periodic pattern, and the feature pattern should show a periodic repetition pattern in areas with rich features such as edges, corners, and points.

[0096] S12. Based on step S11, design the feature pattern and draw the feature pattern in the skin treatment area, dividing the skin treatment area into multiple skin points P = {P1, P2, ..., P}. i …P nim}, where num refers to the number of skin particles in the skin treatment area.

[0097] S13. Mark the effective contact areas using a masking method to obtain a mask image (Mask), which provides a training set for the subsequent training of the region recognition convolutional neural network, such as... Figure 6 As shown in the image. The masking method described here refers to using a computer to select an appropriate marker diameter and creating a marker on the image by clicking on it with the mouse, which is used to delineate the effective contact area.

[0098] S2. Based on the pressure conditions, the skin is divided into effective contact area, skin accumulation area, and skin stretching area.

[0099] See Figure 4 The skin is divided into a skin stretching area, a skin accumulation area, and an effective contact area. To allow the miniature camera to observe the image of the effective contact area, the pressing part of the actuator is designed with a transparent material and undergoes fine grinding and polishing.

[0100] When detecting areas of skin buildup, a stereo vision camera is mounted in the forward direction of the actuator's movement using a camera adapter. Fine-tuning is performed to ensure the stereo vision camera can capture clear images of these areas. When detecting areas of skin stretching, the stereo vision camera is mounted in the opposite direction of the actuator's movement using a camera adapter. Fine-tuning is performed to ensure the stereo vision camera can capture clear images of these stretching areas. The color images captured by the stereo vision camera are then processed. With depth images Send to the computer.

[0101] A miniature camera is mounted inside the actuator's pressing area via a camera adapter. Fine-tuning allows the camera to capture clear images of its effective visual area, which are then used to identify the effective contact area. The color image of the pressing area captured by the miniature camera is then displayed. Send to the computer.

[0102] S3. Acquire a first image using a miniature camera, and obtain the effective contact area based on the first image.

[0103] In this embodiment, a region recognition convolutional neural network is constructed to identify effective contact regions. See [link to documentation]. Figure 2 Step S3 specifically includes steps S31-S35:

[0104] S31. The color image Pic2 acquired by the miniature camera has dimensions (w0×h0). The input size of the region recognition convolutional neural network is (w0×h0). in ×h in The maximum value among w0 and h0, max(w0×h0), is selected as the maximum bound for scaling, and Pic2 is scaled without distortion. The width and height are compared to the network input size (w). in ,j in Fill the empty parts of the image with black with an RGB value of (128, 128, 128), and then normalize the RGB values ​​of the image to 256, resulting in the scaled and normalized image Pic′2.

[0105] Specifically, the acquired color image of the pressed area, Pic2, has dimensions of (480×640). The input size of the region recognition convolutional neural network is (320×320). 640 is chosen as the maximum scaling boundary, and Pic2 is scaled down without distortion. The image is given a width and height of (320,320) = (320×240). Compared with the network input size (320×320), the empty parts of the image are filled with black with RGB values ​​of (128,128,128). Then, the RGB values ​​of the image are normalized with 256 as 1 to obtain the scaled and normalized image Pic′2.

[0106] S32. Input the scaled and normalized image Pic′2 into the feature extraction network to obtain deep information of the image. First, the scaled image is convolved into a middle information layer M by 3 convolutional layers; then, the deep information layer M is convolved into a deep information layer N by a residual network.

[0107] Specifically, the scaled image is first convolved into a mid-level information layer M with a size of (60×60×512) through three convolutional layers; then the deep information layer M is convolved into a deep information layer N with a size of (60×60×2048) through a residual network.

[0108] S33. Input the deep information layer N into the category extraction network, and convolve the deep information layer N into the category information layer Cls through upsampling, downsampling and convolution.

[0109] Specifically, the deep information layer N is input into the category extraction network, and the deep information layer N is convolved into a category information layer Cls of size (60×60×2) through upsampling, downsampling and convolution.

[0110] S34. Perform bilinear interpolation on the category information layer Cls to obtain a value of (w in ×h inThe output layer Out (×2) represents the probability of each category in the output layer Out. The range of the actual pressed area is output based on the reference probability λ. The number 2 represents the number of categories for both the background and the actual pressed area.

[0111] Specifically, bilinear interpolation is performed on the category information layer Cls to obtain an output layer Out of size (480×640×2). The value of each layer of the output layer Out represents the probability of that category. The actual pressing area is output based on a reference probability of 50%.

[0112] S35. During network training, positive sample data suitable for training is selected from a large number of negative samples output by the network using a positive sample matching method. A loss function is then constructed to adjust the convolution kernel coefficients. The loss function Loss consists of the cross-entropy loss between the output layer Out and the labeled mask image Mask; the formula for the loss function Loss is as follows:

[0113]

[0114]

[0115]

[0116] Where N is the number of parameters in the output layer Out, L0 is the total probability of the effective contact area, and L1 is the total probability of the background. This refers to the probability that the i-th parameter of the output layer (Out) is a valid contact region. This refers to the probability that the i-th parameter of the output layer Out is the background.

[0117] The labeled image Mask is manually labeled and is a category image of size (w0×h0×1), with a value of 1 in the effective contact area and 0 in the background area. During training, the number of training iterations, hyperparameters, and training conditions are set for the constructed region recognition convolutional neural network to train the network parameters.

[0118] S4. Acquire a second image of the skin accumulation area and / or skin stretching area using a stereo vision camera, and obtain the three-dimensional coordinates of skin particles in the skin accumulation area and / or skin stretching area based on the second image; obtain the three-dimensional coordinates of skin particles in the effective contact area based on the first image.

[0119] Specifically, step S4 includes steps S41-S45:

[0120] S41. Construct a feature pattern recognition algorithm for the color image acquired by the stereo vision camera at time ti. Color images captured by miniature cameras Perform grayscale histogram analysis and binarize the grayscale histogram.

[0121] S42. Filter color images Contours whose area after binarization is greater than a specified pixel threshold S

[0122] S43. Based on the line segment extraction algorithm, the contour is... The process yields contour segments K1 and K2, which are composed of continuous contour segment points.

[0123] The line segment extraction algorithm involves sequentially detecting contours. Does the pixel neighborhood around the mid-contour point contain only x? c If a contour point meets a certain condition, then delete that contour point to obtain the contour. Increase x after one round of testing c The quantity continues to scan the outline. Delete contour points that meet the conditions; after multiple repetitions, the contour line segments of the contour can be obtained. Outline segment It consists of contour line segments and points.

[0124] For example: the line segment extraction algorithm sequentially detects contours. If the pixel neighborhood surrounding a mid-contour point contains only 3 contour points, then delete that contour point to obtain the contour. Contour detection sequentially If the pixel neighborhood surrounding a mid-contour point contains only four contour points, then delete that contour point to obtain the contour. Contour detection sequentially If the pixel neighborhood surrounding a mid-contour point contains only 5 contour points, then delete that contour point to obtain the contour line segment. Outline segment It consists of contour line segments and points.

[0125] S44. Obtain identifiable skin particles at time ti using the nine-point scanning method. and Ep ti exist and Two-dimensional coordinates on the surface.

[0126] The nine-point scanning method involves scanning eight surrounding points within a 3×3 pixel area to determine if the center point is a contour line segment point. If only one contour line segment point exists in the surrounding area, the center point can be recorded as a skin texture point.

[0127] in These are the identifiable skin particles in the skin accumulation and stretching regions at time ti. The identifiable skin particles in the effective contact area at time ti.

[0128] S45. Adjust Ep according to the actuator's direction of movement. ti With Ep t(i-1) Perform matching to obtain Ep ti A one-to-one correspondence with P.

[0129] S5. Obtain the skin deformation based on the obtained three-dimensional coordinates of the skin particles, and perform three-dimensional estimation of the skin deformation under stress.

[0130] Specifically, step S5 includes steps S51-S56:

[0131] S51. Construct a skin mass point coordinate transformation algorithm, establish a base coordinate system T0 on the actuator base, and use the depth image acquired by the stereo vision camera at time ti. Depth information of skin particles in the skin accumulation and stretching regions at time ti was obtained using a depth-first search algorithm and 3D mapping.

[0132] S52. Based on the calibration relationship between the actuator and the stereo vision camera. It can obtain the three-dimensional coordinates of skin particles in the skin accumulation region and skin stretching region at time ti in the actuator coordinate system.

[0133]

[0134] S53. Obtain the depth information of skin particles in the effective contact area at time ti based on the actuator's device dimensions and three-dimensional mapping.

[0135] S54. Based on the calibration relationship between the actuator and the miniature camera. It can obtain the three-dimensional coordinates of the skin particles in the effective contact area at time ti in the actuator coordinate system.

[0136]

[0137] S55. Obtain the three-dimensional coordinates of each skin particle at time ti based on the two-dimensional image matching relationship of skin particle P.

[0138] Where n i n represents the number of skin particles identified at time ti. i ≤num;

[0139] S56. Construct a real-time skin deformation model, calculate the compression and tension of the skin force based on the displacement difference of skin particles on the guided working trajectory, and perform three-dimensional estimation of the skin's stress deformation. (See also...) Figure 7 Based on the displacement difference of skin particles along the guided work trajectory, the compression and tension of the skin are calculated, and the three-dimensional estimation of skin deformation under stress is performed, such as... Figure 8 As shown, Figure 8 This is a model of real-time skin deformation. Figure 7 (a) is a time-displacement diagram of skin particles in the skin accumulation area. Figure 7 (b) is a time-displacement diagram of skin particles in the skin stretching area. Figure 8 (a) is a schematic diagram of the three-dimensional model of the skin stretching area. Figure 8 (b) is a schematic diagram of the three-dimensional model of the skin accumulation area.

[0140] In summary, compared with the two-dimensional tensile deformation detection research of speckle, the method of this embodiment can not only make a realistic three-dimensional estimation of the skin stretching and accumulation area caused by skin stress deformation, but also capture images of the effective contact area through a miniature camera when the actuator occludes the skin pressure area, so as to realize the skin stress deformation estimation in that area; compared with the existing region detection network, the network computation of this method is reduced, which can meet the needs of real-time detection.

[0141] This embodiment also provides a device for measuring skin deformation under stress, including:

[0142] At least one processor;

[0143] At least one memory for storing at least one program;

[0144] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 3 The method shown.

[0145] This embodiment of the device for measuring skin deformation under stress can execute the method for measuring skin deformation under stress provided in the method embodiment of the present invention. It can execute any combination of the steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0146] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 3 The method shown.

[0147] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0148] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0149] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0152] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0153] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0154] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0155] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A system for measuring skin deformation under stress, characterized in that, include: At the end of the actuator, there is a transparent pressing part, which is used to press the skin and move in a preset direction; The skin features a characteristic pattern consisting of multiple skin particles; A miniature camera is disposed within the end of the actuator and captures a first image through the pressing portion; A stereo vision camera, mounted externally at the end of the actuator, is used to acquire a second image of the skin stretching area and / or the skin accumulation area; the skin stretching area is the region behind the end of the actuator in the direction of movement, and the skin accumulation area is the region in front of the end of the actuator in the direction of movement. The data processing module obtains the effective contact area between the pressing part and the skin based on the first image, obtains the three-dimensional coordinates of the skin particles in the effective contact area based on the first image, obtains the three-dimensional coordinates of the skin particles in the skin stretching area and / or skin accumulation area based on the second image, obtains the skin deformation based on the obtained three-dimensional coordinates of the skin particles, and performs three-dimensional estimation of the skin deformation under force.

2. A method for measuring skin deformation under stress, characterized in that, Includes the following steps: Design a feature pattern in the skin treatment area, the feature pattern comprising multiple skin particles. , It is the number of skin particles in the skin treatment area; Based on the pressure applied, the skin is divided into effective contact area, skin accumulation area, and skin stretching area; A first image is acquired using a miniature camera, and the effective contact area is determined based on the first image. The three-dimensional coordinates of skin particles in the effective contact area are obtained from the first image. A second image is acquired on the skin accumulation area and / or skin stretching area using a stereo vision camera, and the three-dimensional coordinates of skin particles in the skin accumulation area and / or skin stretching area are obtained based on the second image. The deformation of the skin is obtained by acquiring the three-dimensional coordinates of the skin particles, and the three-dimensional estimation of the skin's stress deformation is performed.

3. The method for measuring skin deformation under stress according to claim 2, characterized in that, The first image is a color image. ; The second image includes a color image. With depth images ; in, Representing the time, Representing the Color images captured by a time-lapse stereo vision camera. Representing the Color images captured by a miniature camera at any moment. Representing the Depth images captured by a time-lapse stereo camera.

4. The method for measuring skin deformation under stress according to claim 2, characterized in that, The step of obtaining the effective contact area based on the first image includes: The first image is input into the trained region recognition model, which outputs a feature map marked with the effective contact area. The training set used to train the region recognition model is obtained in the following way: The effective contact area is marked using a masking method to obtain a mask image. , as training samples, are used to obtain the training set.

5. The method for measuring skin deformation under stress according to claim 4, characterized in that, The region identification model is trained in the following way: For sample images Scaling and normalization are performed to obtain a scaled and normalized image. ; Scaling and normalizing the image Input a feature extraction network to obtain deep information of the image; Deep information layer Input category extraction network, through upsampling, downsampling and convolution, extracts deep information layers Convolutional category information layer ; Category information layer Perform bilinear interpolation to obtain the output layer. Output layer The numerical value at each level represents the probability of belonging to that category, based on the reference probability. Outputs the actual range of the pressed area; During network training, positive sample data for training is selected from a large number of negative samples output by the network using a positive sample matching method, and then a loss function is constructed for regression adjustment of the convolution kernel coefficients.

6. The method for measuring skin deformation under stress according to claim 5, characterized in that, The loss function From the output layer With marker mask image It consists of direct cross-entropy loss; Loss function formula The expression is: in, For output layer The number of parameters, The total probability of effective contact area. The total probability of the background. Refers to the output layer No. The parameter represents the probability of an effective contact area. Refers to the output layer No. The parameter represents the probability of the background.

7. The method for measuring skin deformation under stress according to claim 3, characterized in that, The step of obtaining the three-dimensional coordinates of skin particles in the skin accumulation area and / or skin stretching area based on the second image includes: Based on the second image, identify skin particles in the skin accumulation area and / or skin stretching area, and obtain the two-dimensional image matching relationship of the skin particles; Establish a base coordinate system on the actuator base. According to the Depth images captured by a time-lapse stereo camera By using a depth-first search algorithm and 3D mapping, the first... Depth information of skin particles in areas of skin accumulation and / or stretching at all times. ; Based on the calibration relationship between the actuator and the stereo vision camera Obtain the first coordinate in the actuator coordinate system The three-dimensional coordinates of skin particles in the skin accumulation area and / or skin stretching area at any given time ; According to skin texture The two-dimensional image matching relationship is obtained to obtain the first The three-dimensional coordinates of each skin point at any given time .

8. The method for measuring skin deformation under stress according to claim 3, characterized in that, The step of obtaining the three-dimensional coordinates of skin particles in the effective contact area based on the first image includes: Based on the first image, identify skin particles in the effective contact area and obtain the two-dimensional image matching relationship of the skin particles; Based on the actuator's device dimensions and three-dimensional mapping, obtain the first... Depth information of skin particles in the contact area at all times ; Based on the calibration relationship between the actuator and the miniature camera Obtain the first coordinate in the actuator coordinate system Three-dimensional coordinates of skin particles in the area of ​​constant effective contact ; According to skin texture The two-dimensional image matching relationship is obtained to obtain the first The three-dimensional coordinates of each skin point at any given time ; in Representing the The number of skin texture points identified at any given time. .

9. The method for measuring skin deformation under stress according to claim 8, characterized in that, The step of identifying skin particles in the effective contact area based on the first image and obtaining the two-dimensional image matching relationship of the skin particles includes: For the Color images captured by a miniature camera Perform grayscale histogram analysis and binarize the grayscale histogram. Filtering color images After binarization, the contour area is greater than the preset pixel threshold. outline ; Based on the line segment extraction algorithm, the contour is... Processing is performed to obtain the contour line segments. The outline segment It consists of continuous contour line segments and points; Obtain the contour line segments using the nine-point scanning method. particles inside and point mass In color images Two-dimensional coordinates on; The mass will be moved according to the direction of the actuator. With point mass Matching is performed to obtain the point mass. With skin texture A one-to-one correspondence.

10. A device for measuring skin deformation under stress, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 2-9.

Images