Pressure-modulated optical tracking hardness sensor, hardness detection method, medium and equipment

Abstract

The present application provides a pressure-modulated optical tracking hardness sensor, a hardness detection method, a medium and a device, comprising: a sensor housing; an elastic membrane is provided at one end of the sensor housing, and a plurality of visual tracking points are provided on the inner surface of the elastic membrane; an image acquisition device and a pneumatic pipeline are inserted into the other end of the sensor housing; the pneumatic pipeline is connected to a pressure regulator and the interior of the sensor; the pressure regulator controls the internal pressure of the sensor to determine the hardness of the tissue to be tested by adjusting the compliance of the sensor. The present application can quantify the elasticity of the phantom without an external position reference.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to a pressure-modulated optical tracking hardness sensor, a hardness detection method, a medium and a device. Background Art

[0002] Stiffness sensing is critical to understanding the objects that robots manipulate, and in grasping and manipulation tasks, objects of different stiffness are handled differently to maintain maximum control. In medical robotics, stiffness can convey information about tissue type and tissue health, as unhealthy tissue often has a different stiffness than the tissue around it. For example, breast cancer tissue may be ten times stiffer than normal breast tissue. To ensure that users are fully informed, sensors should be able to digitally measure sample stiffness and detect changes in stiffness during sensor movement.

[0003] Stiffness is used to refer to two different measurements. Traditionally, stiffness refers to the elastic constant (k) of an object. On the other hand, tissue stiffness is measured by the elastic modulus (E) of the material. The elastic constant (k) describes a structural property, while the elastic modulus (E) describes a material property, but the two can be related by the following formula (1):

[0004]

[0005] Among them, k represents the elastic constant, E represents the elastic modulus, A represents the area of ​​the object being measured, and L0 represents the original length of the object being measured. Therefore, when measuring objects of the same size, the area A and the original length L0 remain unchanged, so k is proportional to E.

[0006] Currently, there are several techniques used to measure hardness, which are as follows:

[0007] Hardness sensors that measure force-displacement relationships measure hardness by tracking the force and displacement of an actuator acting on a sample. One sensor type is a tactile switch mounted on a force sensor above a trapezoid, allowing the indentation depth and force to be measured. This method was used to observe the sensor response to different indentation rates for tissue samples of different softness. Another sensor type drives the sensor probe into the tissue with a known displacement and measures the resulting force, which shows that samples of different hardness have different force-displacement graphs.

[0008] The stiffness is measured by measuring the conformity of the sample during indentation or aspiration. During the absorption or indentation process, the deformation of the tissue is used to calculate the stiffness. The probe measures the aspiration pressure and the displacement of the tissue into the sensor. This method can be applied to in vivo experiments. When inverse finite element analysis is applied, it can estimate the modeling coefficients of synthetic samples and tissues. This sensor uses a rigid component to press on the edge of the aspiration area.

[0009] Gubenko et al. developed a tissue stiffness sensor that measures stiffness by measuring the vertical displacement of a central mirror and recording the contact area radius with a camera. These measurements were compared with a finite element model to measure the tissue stiffness of two samples with elasticities of approximately 2300 kPa and 200 kPa. These stiffnesses were calculated with errors of "higher than 10%" and "close to 5%." This sensor is soft-tipped. Nguyen et al. developed a sensing chip that uses three piezoresistive cantilevers to measure tissue stiffness. One cantilever was placed in the center of the chip, while the other two cantilevers were located at the edges. These cantilevers were then covered by an elastic pad. Softer soft tissue will conform more to the sensor, causing the edge cantilevers to experience greater forces relative to the center cantilever. This difference can be measured from the resistance of the cantilever. The sensor was pressed against six samples with elasticities ranging from 57 kPa to 3.2 GPa. When the resistance ratio was plotted, each sample produced a unique curve.

[0010] Yuan et al. also used tissue deformation properties to enable the Gelsight sensor to sense hardness. When the sensor was pushed into samples of different hardness, the movement of the sensor surface was recorded. The deformation was input into the neural network, and after a few frames, the hardness of the sample could be found. It is worth noting that in soft materials, these two properties are not always correlated. The hardness of the samples used ranged from 00-8 to 00-87. These samples were made of elastomers, and their elasticity ranged from about 55kPa to about 827kPa. The root mean square error (RMSE) of the measurements of samples whose shapes were trained but whose hardness was not trained was 00-5.18, accounting for 5.95% of the range. The Gelsight sensor was also shown to be able to measure forces when interacting with rigid objects and detect embedded nodules when statically held under pressure against the sample.

[0011] The shift in the resonant frequency of the displacement has also been used to measure tissue hardness. Omata et al. showed that the shift in the resonant frequency of the sensor under a set indentation pressure is related to the Young's modulus and the spring constant. Samples with elasticity ranging from 10kPa to 100GPa were tested. Zhang et al. tested a resonant sensor that can operate at angles up to 45 degrees to classify samples of different hardness. Samples with gradually increasing hardness were correctly classified from 9.3% to 87.5%, with an accuracy of 92%.

[0012] Comparing the deformation of indenters of different stiffness is used to sense hardness. Faragasso et al. used a camera to measure the displacement of two pairs of indenters on springs of different stiffness. The sensor was tested on four silicon samples, one for each sample. The spring stiffness of these samples ranged from 0.0856N / m to 2.2373N / m, and the calculated error was between 1% and 5% of the sample spring stiffness. The average error of the sample spring stiffness was 3.5%. This sensor was modified to fit an endoscope. It uses four beams, three of which are highly flexible and one with low compliance that can be traced using a micro-helix. The device was tested on a monolithic model with an error of 5.386%. Faragasso et al. adapted their sensor. The sensor uses three high-compliance low-stiffness springs surrounded by a spring indenter. The results showed that the spring stiffness was 0.29N / mm and could measure a sample of 0.62N / mm at angles up to 20°, with a maximum error of 6.4% of the sample stiffness. The sensor uses a hard indenter to interact with the sample.

[0013] Nagatom and Miki invented a sensor that can measure sample stiffness by comparing the stress on materials with different compliances during shear forces. Four strain gauges are arranged around the contact pin, two embedded in lower compliance PDMS and two embedded in higher compliance PDMS conforming to the PDMS standard. The probe's stiffness was measured to be about 40GPa when the sensor was loaded under shear load. The sensor was not tested using samples the size of human soft tissue or contact palpation with a lubricant representing fluids in the body. The sensor uses a hybrid tip-soft material made of hard and flexible materials.

[0014] However, the hardness sensors listed above further demonstrate that flexible tip indenter sensors have not yet been demonstrated to be useful for quantifying sample hardness. Summary of the invention

[0015] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a pressure-modulated optical tracking hardness sensor, a hardness detection method, a medium and an apparatus to solve the technical problem that the current flexible tip pressure head sensor cannot be used to quantify the hardness of a sample.

[0016] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a pressure-modulated optical tracking hardness sensor, comprising: a sensor housing; an elastic membrane is provided at one end of the sensor housing, and a plurality of visual tracking points are provided on the inner surface of the elastic membrane; an image acquisition device and a pneumatic pipeline are inserted into the other end of the sensor housing; the pneumatic pipeline is connected to a pressure regulator and the interior of the sensor; the pressure regulator controls the internal pressure of the sensor to determine the hardness of the tissue to be tested by adjusting the compliance of the sensor.

[0017] In some embodiments of the first aspect of the present application, the compliance of the sensor increases with the increase of internal pressure; the increase in compliance is manifested as: the mass of the internal gas increases with the increase of internal pressure, thereby increasing the internal hardness of the sensor; the stress of the elastic membrane increases with the increase of internal pressure, thereby causing the visual tracking point on the inner surface of the elastic membrane to shift.

[0018] In some embodiments of the first aspect of the present application, the process of determining the hardness of the tissue to be tested by adjusting the compliance of the sensor includes: controlling the internal pressure of the sensor through the pressure regulator to obtain internal hardness data under at least two internal pressures and displacement data of the visual tracking point, and the acquisition method includes: measuring the internal hardness data under two consecutive internal pressures and the displacement data of the visual tracking point, and repeating this process to collect multiple sets of data, calculating the characteristic values ​​of the multiple sets of data, so as to obtain the final internal hardness data and displacement data of the visual tracking point.

[0019] In some embodiments of the first aspect of the present application, the calculation method of the hardness of the tissue to be measured includes:

[0020]

[0021] Among them, k t Indicates the hardness of the tissue to be tested; k s1 represents the first spring stiffness, which refers to the internal hardness of the sensor at the first internal pressure; k s2 Indicates the second spring stiffness, which refers to the internal hardness of the sensor under the second internal pressure; ΔX s1 Indicates the first spring deformation, which refers to the displacement data generated by the visual tracking point under the first internal pressure; ΔX s2 Indicates the deformation of the second spring, which refers to the displacement data generated by the visual tracking point under the second internal pressure.

[0022] In some embodiments of the first aspect of the present application, the internal gas mass increases with increasing internal pressure, which includes: Among them, n represents the internal gas mass; R represents the ideal gas constant, T represents the gas temperature, P represents the internal gas pressure, and V represents the internal gas volume.

[0023] In some embodiments of the first aspect of the present application, a plurality of visual tracking points are arranged on the inner surface of the elastic membrane, one of which is located at the center of the inner surface, and the remaining visual tracking points are evenly arranged on a circumference around the center of the inner surface.

[0024] In some embodiments of the first aspect of the present application, the sensor housing is printed from a photocurable resin material.

[0025] In some embodiments of the first aspect of the present application, the image acquisition device includes a USB endoscope camera.

[0026] In some embodiments of the first aspect of the present application, the elastic membrane is made of platinum-catalyzed silicone rubber Ecoflex 00-50 material.

[0027] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a tissue hardness detection method, which is applied to a computer device; the computer device is connected to a pressure-modulated optical tracking hardness sensor as described in any one of claims 1 to 9; the detection method includes: acquiring at least two sets of sensor data; each set of sensor data corresponds to internal hardness data and displacement data of visual tracking points under each internal pressure; based on the internal hardness data, displacement data of visual tracking points, and the difference between the embedding depth data of the sensor corresponding to the tissue to be tested, the hardness of the tissue to be tested is calculated and generated.

[0028] In some embodiments of the second aspect of the present application, when the tissue to be tested is in contact with a non-flat dome-shaped membrane, the average radial displacement of the visual tracking points of the pressure-modulated optical tracking hardness sensor under multiple internal pressures is input into a trained recurrent neural network, and the recurrent neural network outputs the final hardness data of the tissue to be tested after nonlinear calculation.

[0029] In some embodiments of the second aspect of the present application, the recurrent neural network includes a feedforward fully connected recurrent neural network, which includes an input layer, multiple hidden layers, and an output layer; the input layer includes multiple input nodes, each hidden layer includes multiple hidden nodes, and each hidden node applies a linear activation function to its input data according to the weights established in training; the output layer outputs a predicted value, and uses the test results of the compression test as the benchmark true value for model training.

[0030] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the tissue hardness detection method when executed by a processor.

[0031] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the tissue hardness detection method.

[0032] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the tissue hardness detection method.

[0033] As described above, the pressure-modulated optical tracking hardness sensor, hardness detection method, medium and equipment of the present application have the following beneficial effects: The present application measures tissue hardness by adjusting the internal compliance of the sensor, thereby being able to quantify the elasticity of the phantom without an external position reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1A Shown is a three-dimensional structural diagram of a pressure-modulated optical tracking hardness sensor in one embodiment of the present application.

[0035] Figure 1B Shown is a cross-sectional view taken along line AA' of a pressure-modulated optical tracking hardness sensor according to an embodiment of the present application.

[0036] Figure 2A It is a schematic diagram of the force interaction between a solid spring and an object before contact in one embodiment of the present application.

[0037] Figure 2B It is a schematic diagram showing the force interaction between a solid spring and an object after contact in one embodiment of the present application.

[0038] Figure 2C It is a schematic diagram showing the force interaction between the sensor spring and the tissue before contacting in one embodiment of the present application.

[0039] Figure 2D It is a schematic diagram showing the force interaction between the sensor spring and the tissue after contacting in one embodiment of the present application.

[0040] Figure 3 Shown is a structural schematic diagram of a feedforward fully connected recurrent neural network in one embodiment of the present application.

[0041] Figure 4 Shown is a flowchart of an image processing process in an embodiment of the present application.

[0042] Figure 5 Shown is a schematic diagram of the elastic modulus range of tissue and phantom in one embodiment of the present application.

[0043] Figure 6 Shown is a schematic diagram of the displacement of a visual tracking point in one embodiment of the present application.

[0044] Figure 7 Each phantom in an embodiment of the present application is shown as - Schematic diagram of internal pressure curve.

[0045] Figure 8 Shown is a flow chart of a tissue hardness detection method in one embodiment of the present application.

[0046] Fig. 9 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0049] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0050] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0052] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0053] <1> PMOT (Pressure Modulate Optical Tracking) sensor: A pressure modulated optical tracking sensor combines pressure sensitive elements and optical detection technology to measure and track pressure changes. This sensor is commonly used in industrial process control, pneumatic measurement, flow monitoring and other fields.

[0054] <2> Feed-forward Fully Connected Regression Neural Network: It consists of multiple layers of nodes (or neurons), and the nodes of each layer are fully connected to the nodes of the next layer without loops or feedback. It is mainly used for regression analysis, that is, predicting continuous value outputs. The network structure of a feed-forward fully connected regression neural network usually includes an input layer, a hidden layer, and an output layer.

[0055] In order to solve the technical problems described in the above background technology, this application uses a pressure modulated optical tracking sensor (PMOT) to measure the characteristics of a variable range of force through the compliance control of a soft membrane, explores a new sensing method of a pressure modulated optical tracking sensor and further expands its functions. This flexible tip indenter can quantify the elasticity of a phantom without an external position reference.

[0056] It should be understood that the PMOT (Pressure Modulate Optical Tracking) sensor is a pressure modulated optical tracking sensor that combines a pressure sensitive element and optical detection technology to measure and track pressure changes. This sensor is commonly used in industrial process control, pneumatic measurement, flow monitoring and other fields. In a pressure modulated optical tracking sensor, the pressure sensitive element is usually a diaphragm or a similar structure that can sense pressure changes and produce corresponding deformations. This deformation is measured and tracked by optical detection technology. Commonly used optical detection technologies include reflective interferometers, beam deflectors, beam benders, etc. When the pressure changes, the diaphragm deforms, causing the path of the light beam to change. By detecting the change in the light beam path, the change in pressure can be calculated. This sensor has the advantages of high precision, high sensitivity, low hysteresis and fast response, so it is widely used in various occasions where accurate measurement of pressure changes is required.

[0057] like Figure 1A and 1B As shown, a schematic diagram of the structure of a pressure-modulated optical tracking hardness sensor in an embodiment of the present application is shown. Figure 1A This is a three-dimensional structural diagram of the pressure-modulated optical tracking hardness sensor. Figure 1B 1 is a cross-sectional view of the pressure modulated optical tracking hardness sensor taken along the line A-A'. For ease of description, the pressure modulated optical tracking hardness sensor is referred to as PMOT sensor in the following text.

[0058] In the embodiment of the present application, the PMOT sensor includes a sensor housing 11, an elastic membrane 12 is provided at one end of the sensor housing 11, and a plurality of visual tracking points 121 are provided on the inner surface of the elastic membrane 12. An image acquisition device 13 and a pneumatic tube 14 are inserted into the other end of the sensor housing 11; the pneumatic tube 14 is connected to a pressure regulator (not shown) and the interior of the sensor; the pressure regulator controls the internal pressure of the sensor to determine the hardness of the tissue to be tested by adjusting the compliance of the sensor, and / or to sense the change of hardness by moving on the tissue to be tested during dynamic palpation.

[0059] It should be understood that a visual tracking point refers to a specific point or feature in a video or image used for object tracking or dynamic analysis. In the field of computer vision and image processing, a visual tracking point is a marker or feature point used to determine the position, motion trajectory or other related attributes of an object. By using visual tracking points, video or image sequences can be automatically analyzed to detect and track specific target objects.

[0060] In the embodiments of the present application, the compliance of the sensor increases with the increase of internal pressure; the increase in compliance is manifested as: the mass of the internal gas increases with the increase of the internal pressure, thereby increasing the internal hardness of the sensor; the stress of the elastic membrane increases with the increase of the internal pressure, thereby causing the visual tracking point on the inner surface of the elastic membrane to be displaced.

[0061] Furthermore, the process of determining the hardness of the tissue to be tested by adjusting the compliance of the sensor includes: controlling the internal pressure of the sensor by the pressure regulator to obtain internal hardness data and displacement data of the visual tracking point, and the acquisition method includes: measuring the internal hardness data and displacement data of the visual tracking point under two consecutive internal pressures, and repeating this process to collect multiple sets of data, and calculating the characteristic values ​​of the multiple sets of data, so as to obtain the final internal hardness data and displacement data of the visual tracking point. Exemplarily, the characteristic values ​​of the multiple sets of data include but are not limited to the average value, root mean square value, median, weighted average value, etc. of the multiple sets of data.

[0062] The calculation method of the hardness of the tissue to be tested includes:

[0063]

[0064] Among them, k t Indicates the hardness of the tissue to be tested; k s1 represents the first spring stiffness, which refers to the internal hardness of the sensor at the first internal pressure; k s2 Indicates the second spring stiffness, which refers to the internal hardness of the sensor under the second internal pressure; ΔX s1 Indicates the first spring deformation, which refers to the displacement data generated by the visual tracking point under the first internal pressure; ΔX s2 Indicates the deformation of the second spring, which refers to the displacement data generated by the visual tracking point under the second internal pressure.

[0065] In order to facilitate those skilled in the art to better understand the calculation principle of the present invention, the derivation process of the above formula (2A) will be further explained below:

[0066] First, the state of the sensor under the first internal pressure is understood as the sensor spring 1 (elastic stiffness K s1 ), the state of the sensor under the second internal pressure is understood as the sensor spring 2 (elastic stiffness is K s2 ), and set K s1 >K s2 .

[0067] Then, combined Figures 2A to 2D Graphics displayed, Figure 2A It shows the force interaction between the solid spring and the object before contact. Figure 2BIt shows the force interaction between the solid spring and the object after contact. Figure 2C Shown is the force interaction between the sensor spring and tissue before contact. Figure 2D Shown is the force interaction after the sensor spring contacts the tissue.

[0068] F is the force of the system without load, F1 is the force of equal value and opposite direction acting between the solid spring and the object, and F2 is the force of equal value and opposite direction acting between the solid spring and the object. ΔX s1 and ΔX s2 They represent the deformation of sensor spring 1 and sensor spring 2 respectively, and X0 represents the initial length of the two sensor springs. When sensor spring 1 and sensor spring 2 contact the tissue, a force will be generated on the contact surface. These forces acting on the tissue and the tissue deformation can be used to calculate the tissue hardness.

[0069] After contact, the force acting on each tissue spring contact point is expressed as:

[0070] F1=k t ΔX t1 =k s1 ΔX s1 ;Formula (2B)

[0071] F2=k t ΔX t2 =k s2 ΔX s2 ;Formula (2C)

[0072] The hardness K of the tissue to be tested is calculated by calculating the difference between the two forces. t :

[0073] F1-F2=k t ΔX t1 -k t ΔX t2 ; Formula (2D)

[0074]

[0075] Since the springs are connected to each other, their displacements are equal:

[0076] ΔX t1 +ΔX s1 =ΔX t2 +ΔX s2 ;Formula (2F)

[0077] ΔX t1 -ΔX t2 =ΔX a2 -ΔX s1 ;Formula (2G)

[0078] Substituting formula (2B), (2C), and (2G) into formula (2E), we obtain the following:

[0079]

[0080] Based on formula (2I), the following formula is obtained:

[0081]

[0082] The hardness of the tissue to be tested can be calculated using the above formula (2J).

[0083] It is worth noting that in order to obtain the hardness of an object, it is usually necessary to measure the deformation of at least two springs with different stiffnesses. When two springs with different stiffnesses are connected to a common substrate and in normal contact with a flat tissue sample, the tissue hardness can be calculated using formula (2K).

[0084]

[0085] In this case, k s Represents the spring constant of the sample, k1 and k2 represent the spring constants of the two springs, F1 and F2 represent the forces between the two springs and the tissue, Δd represents the difference in the embedding depth of the two springs, and Δx1 and Δx2 represent the deformation of the two springs. In connection with the embodiments of the present application, as long as the pressure modulated optical tracking hardness sensor and the tissue remain rigid, the pressure modulated optical tracking hardness sensor can measure the tissue hardness through a single contact. The pressure modulated optical tracking hardness sensor can set more than two compliances to associate tissue deformation with tissue hardness. At each internal pressure, the compliance of the membrane tip is related to the two spring constants k in the above formula (2K) s1 , k s2 The difference between the position of the membrane tip when it is in contact with and not in contact with the membrane is proportional to the deformation ΔX of the two springs in the above formula (2K) s1 , ΔX s2 Proportional to the membrane. Specifically, when the membrane tip is in contact with the membrane, it is subjected to pressure and its position changes; when the membrane tip is not in contact with the membrane, its position remains unchanged. Therefore, there is a difference between the position of the membrane tip when it is in contact with and when it is not in contact with the membrane under the same internal pressure, and this difference can be used to measure the degree of membrane deformation or tissue stiffness. This difference can be obtained through the displacement of the membrane tip or the displacement of the tracking point and correlated with parameters such as spring stiffness to calculate tissue stiffness.

[0086] Further, the internal gas mass increases with the increase of internal pressure, which includes:

[0087]

[0088] Among them, n represents the internal gas mass; R represents the ideal gas constant, T represents the gas temperature, P represents the internal gas pressure, and V represents the internal gas volume.

[0089] Therefore, according to the above formula (3A), the relationship between the internal gas pressure (P), the internal gas volume (V), the internal gas mass (n), the ideal gas constant (R) and the gas temperature (T) can be expressed as follows:

[0090] P×V=n×R×T;Formula (3B)

[0091] In an embodiment of the present application, when the tissue to be tested is in contact with a non-flat dome-shaped membrane, the average radial displacement of the visual tracking points of the pressure-modulated optical tracking hardness sensor under multiple internal pressures is input into a trained recurrent neural network, and the recurrent neural network outputs the final hardness data of the tissue to be tested after nonlinear calculation.

[0092] Specifically, the calculation method of tissue hardness recorded in the above formula (2A) cannot fully capture the interaction between tissue and dome-shaped membrane, because the force is applied differently compared to the planar surface applied to the tip of the membrane. In view of this, an embodiment of the present invention uses a recurrent neural network (NN) to handle this complex relationship, using elastic modulus to replace the spring constant for cross-reference. When working as a force sensor, the PMOT sensor can also perform dynamic palpation. During palpation, the PMOT sensor moves on the tissue, and as the tissue hardness increases, the force acting on the PMOT sensor also increases. By changing the compliance parameter of the PMOT sensor, its range can adapt to tissue hardness.

[0093] It should be understood that in the biomedical field, tissue hardness is usually assessed by elastic modulus, which is a physical quantity that describes the material's ability to resist deformation when subjected to external forces. For soft tissues, such as various tissues in the human body, the elastic modulus can provide information about their hardness. By measuring the elastic modulus of tissue, the mechanical properties of the tissue can be understood, which is of great significance for the diagnosis and treatment of certain diseases. For example, diseases such as cancer, inflammation, and fibrosis can cause changes in tissue hardness, and measuring the elastic modulus can help diagnose and monitor these diseases.

[0094] The structure of the recurrent neural network is as follows Figure 3As shown, a feedforward fully connected recurrent neural network (NN) is used to identify the sample hardness from the sensor output. The structure of the feedforward fully connected recurrent neural network (NN) includes an input layer, a hidden layer, and an output layer. The input layer includes multiple input nodes, each of which inputs the average radial displacement of the visual tracking point of the PMOT sensor under different pressures. The feedforward fully connected recurrent neural network (NN) includes multiple hidden layers, each of which includes multiple hidden nodes, and each hidden node applies a linear activation function to its input according to the weight established in the training. The feedforward fully connected recurrent neural network (NN) is trained using the phantom elastic modulus value obtained from the compression test as the benchmark true value.

[0095] For example, in Figure 3 In , a feed-forward fully connected recurrent neural network (NN) is used to identify the hardness of the sample from the sensor output. The displacement of the average tracking point at each pressure is input into the feed-forward fully connected recurrent neural network (NN). The feed-forward fully connected recurrent neural network (NN) has three hidden layers, each with five nodes. Each node applies a rectified linear unit (ReLU) activation function to its output, which is generated based on the weights established during training. This means that at each node, each input (x) is multiplied by a weight (w) and added to the bias (b). If this sum is positive, the result is passed to the next node, and if the sum is negative, 0 is output. For a node in the previous layer with n nodes, this can be written as:

[0096]

[0097] The network was trained using the corresponding phantom elasticity obtained from compression tests as the ground truth output. Before training, all data were normalized between -1 and 1. The network was built using MATLAB's fitrnet function. This function does not allow the batch size to be set, so all training data is used in each batch, resulting in one iteration being the same as one cycle. The network was supervised using the Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm with limited memory, and the results were compared with the validation data every five iterations. Training was repeated until the function loss was less than 1e-16.

[0098] Further, when using neural networks for machine learning, the process of data preparation, standardization, training, and validation is as follows:

[0099] Step S31: Arrange the training data, verification data and test data into a matrix; each column of the matrix represents a variable, and each row represents a data sample, so that the data can be easily input into the neural network for processing in the form of a matrix.

[0100] Step S32: Use formula x s =(2*(x-min) / (max-min))-1, standardize the data of each variable to the range of [-1,1]. Standardization is an important step in preprocessing data. It can convert data of different ranges and units to the same scale, which helps the neural network learn and converge better.

[0101] Step S33: Run the training data through the neural network by applying the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimization algorithm to fit the network weights and biases to the data.

[0102] During the training phase, the L-BFGS algorithm is used to adjust the weights and biases of the neural network in order to minimize the loss function and make the network's predictions as close as possible to the actual values ​​of the training data. The L-BFGS algorithm is an efficient optimization algorithm that is particularly suitable for optimization problems in large-scale data sets and high-dimensional spaces.

[0103] Step S34: Every five iterations, run the validation data through the neural network and record the accuracy.

[0104] During the training process, it is very important to regularly use validation data to evaluate the performance of the model, so as to monitor whether the model is overfitting or underfitting and adjust the training strategy in time. Recording accuracy is a key indicator of model performance.

[0105] Step S35: Stop running when the function loss is less than 1e-16, and export the network weights and biases that produce the best results on the validation data. Function loss, also known as loss function, is an indicator that measures the difference between the model's predicted value and the actual value. When the value of the loss function is lower than a certain threshold (here 1e-16), it means that the performance of the model has reached a very high level. At this point, stop training and save the current optimal model parameters, which can be used in subsequent tests and practical applications.

[0106] For example, the input layer has 4 input nodes, corresponding to the average radial displacement of the visual tracking point under the internal pressure of 0kPa, 2kPa, 4kPa and 6kPa These data measured by the sensors were fed into 3 hidden layers with 5 fully connected nodes in each hidden layer, from which the phantom stiffness was output. A feed-forward fully connected regression neural network (NN) was built using the fitrnet function of MATLAB. This function does not allow the batch size to be set, so each batch of training data was used for one iteration. Supervised learning was performed using the L-BFGS algorithm, and the results were compared with the validation data, with comparisons performed every five iterations. Training was repeated until the function loss was less than 1e-16. The trained neural network produced an RMSE (Root Mean Squared Error) of 0.321 kPa on the validation data.

[0107] Preferably, the image acquisition device is a USB endoscopic camera. A USB endoscopic camera is a camera that can be plugged into a USB port and is typically used in medical, industrial, or security fields to allow real-time video transmission when direct observation is not possible. This camera has high resolution and high sensitivity, and can capture clear images and transmit them to a connected device, such as a computer or other video monitor. With a USB endoscopic camera, a doctor or professional can observe the inside of a patient or device without the need for incisions or punctures.

[0108] Preferably, the sensor housing is printed from a photocurable resin material. The photocurable resin material is preferably Formlabs Tough 2000, which has the characteristics of being hard, tough and difficult to bend, and is suitable for applications that need to withstand wear and high stress, such as housings, fixtures and fixtures, mechanical connectors, and hands that need to withstand wear and tear.

[0109] Preferably, one of the several visual tracking points on the inner surface of the elastic mold is located at the center of the inner surface, and the remaining visual tracking points are evenly arranged on a circumference around the center of the surface. For example, there are 9 hemispherical visual tracking points with a diameter of 1.4 mm, one of which can be cast at the center of the inner surface, and the remaining 8 can be located on a circumference with a radius of 4 mm.

[0110] Preferably, the edge of the elastic membrane is adhered to the sensor housing and wrapped with polytetrafluoroethylene (PTFE) tape to ensure air tightness.

[0111] Preferably, the pressure regulator is an SMC ITV0010 pressure regulator, which has the characteristics of high-precision pressure control, fast response, good stability, easy installation and maintenance, etc.

[0112] Preferably, the camera is a ROTEK USB Otoscope camera with a diameter of 5.5 mm and 1.3 MP, connected to a computer. It should be understood that the OTEK USB Otoscope camera is a medical device that combines otoscope and camera technology.

[0113] The camera uses the OpenCV library to extract the position of the visual tracking point from each frame. The specific execution steps are as follows Figure 4 As shown, including:

[0114] Step S41: blurring the collected image based on a convolution kernel of a preset size to reduce noise on the image.

[0115] Exemplarily, the image is subjected to 7x7 blurring, that is, a 7x7 convolution kernel (or mask) is used in the image processing process to perform blurring operation, the purpose of which is to reduce the noise of the image and improve the smoothness of the image. It should be understood that in blurring, the convolution kernel (or mask) is a key tool. The 7x7 convolution kernel is a matrix whose central element corresponds to each pixel in the image. The convolution kernel then moves on the image, multiplies the value on the convolution kernel with the pixel value of the image it covers, and sums the multiplication results, and then replaces the original pixel value with this sum. This process traverses the entire image to achieve a blurring effect. The specific effect of the 7x7 blurring depends on the value of the convolution kernel, which is usually predefined, such as mean filtering and Gaussian filtering. In mean filtering, all values ​​of the 7x7 convolution kernel are equal, while in Gaussian filtering, the value of the convolution kernel is set according to the Gaussian distribution.

[0116] Furthermore, blurring may result in pixels at the edge of the image not being fully processed, as the number of pixels around these pixels may not be sufficient to form a complete convolution kernel size (e.g., a 7x7 convolution kernel). In this case, edge padding or reducing the size of the convolution kernel can be used to handle this.

[0117] Step S42: extracting visual tracking points from the denoised image through color masking to produce a grayscale mask image.

[0118] It should be noted that a color mask is used to manipulate the color value of a pixel based on a color space (such as RGB or CMYK). A color mask is typically an image with a specific color channel, which can be used to select, modify or replace a specific color or color range in an original image. In an embodiment of the present application, a color mask is typically implemented as an image or matrix with a specific color value. When a color mask is applied to an original image, an algorithm compares the color value of the color mask and the original image pixel by pixel, and performs corresponding extraction operations according to the comparison result.

[0119] Step S43: Use an edge detection algorithm to extract the boundary from the grayscale mask image, and use an edge contour detection algorithm to extract the boundary and its center point.

[0120] Specifically, edge detection algorithm is an important step in image processing, which is mainly used to identify edge lines in images. These lines can be the boundary lines between objects or the shadow areas of things. The purpose of edge detection is to use binary graphs to show the edge information in the image, which can be used for tasks such as image segmentation and image shape extraction.

[0121] The edge detection algorithm in the embodiment of the present application is preferably a Canny edge detection algorithm. The Canny edge detection algorithm is based on a multi-level edge detection algorithm and can meet the general standards of edge detection, including detecting edges with a low error rate, accurately locating the position of the edge, and marking a given edge only once. Its implementation process includes steps such as smoothing the image, calculating the gradient, non-maximum suppression, dual threshold processing, and hysteresis tracking.

[0122] The edge contour detection algorithm can specifically use the find contours algorithm, which is an algorithm used to detect edge contours in binary images. The algorithm is based on image processing and computer vision technology. By performing a series of processing operations on the image, the edge contour information in the image is extracted, and the boundary and center point of the image are extracted.

[0123] In order to further explain the principle of the PMOT sensor in the embodiment of the present application to detect tissue hardness, the following will combine multiple experiments and describe the experimental construction conditions and implementation process to assist in the explanation.

[0124] Experiment 1: Tissue stiffness sensing.

[0125] Experiment 1A: Tissue stiffness sensing experimental method.

[0126] Experiment 1 aims to evaluate the hardness sensing capability of the PMOT sensor. To this end, a set of phantoms with different hardness were made to evaluate how the sensor output changes with pressure when in contact with samples of different hardness. Finally, the sensor's ability to quickly sense hardness using a neural network was studied.

[0127] like Figure 5 , a schematic diagram showing the range of elastic moduli of tissues and phantoms is shown, in which the elasticity of a range of healthy tissues (black) and abnormal tissues (white) is shown as horizontal bars, and the elastic modulus of the phantom used in Experiment 1 is shown as a vertical black dashed line.

[0128] Experiment 1A-1) In order to evaluate the ability of the sensor to measure tissue hardness, phantoms with a hardness range similar to that of human tissue were made. Many healthy human tissues, including liver, prostate and skin, have elastic moduli between 3kPa and 100kPa. Conditions such as liver fibrosis, papillary adenocarcinoma and prostate cancer have been shown to increase the hardness of their respective healthy tissues by 1.9 to 9 times. Therefore, in order to demonstrate that the sensor in this application can distinguish between different tissues and healthy and abnormal tissues, a series of phantoms with hardness ranging from 4kPa to 180kPa were designed, with an average step factor of 1.38.

[0129] The phantoms of different hardness are made of Smooth-On platinum-cured silicone rubber. Smooth-On platinum-cured silicone rubber is an organic rubber, also known as addition-cured silicone rubber. It has the lowest long-term shrinkage value and the longest shelf life of all mold rubbers, making it suitable for casting various molds. In addition, platinum-cured silicone has applications and physical properties that other rubbers do not have, and can be used in some areas that require "skin safety" and directly on human skin. It can also be used to make prosthetics and orthotics.

[0130] Table 1 below summarizes the materials and proportions used to make these phantoms, as well as the elastic modulus obtained by compression testing. The phantoms in the left column of Table 1 are used to measure displacement under internal pressure and for training and validation testing; the phantoms in the right column are used as untrained phantoms for testing.

[0131] Take "4Ecoflex 00-10Part A:4Ecoflex 00-10Part B:8Silicone Thinner" as an example. Ecoflex 00-10Part A and Ecoflex 00-10Part B: This is usually a two-part base material for silicone. Most two-component silicone systems are made up of two parts: Part A (usually a base polymer) and Part B (usually a curing agent or catalyst). When these two parts are mixed in the specified proportions, they begin to react chemically, causing the silicone to cure or harden. 8Silicone Thinner is a silicone thinner used to adjust the viscosity or fluidity of silicone. Thinners can reduce the viscosity of silicone, making it easier to pour into molds or apply to detailed parts. However, too much thinner may reduce the strength and durability of silicone. When mixing these materials together, you need to follow the proportions and recommendations provided by the manufacturer. For example, for Ecoflex 00-10, the general recommendation is to mix Part A and Part B in a 1:1 ratio, and then add the appropriate amount of thinner as needed to adjust the viscosity. The meaning of the remaining items in the table can be deduced in the same way.

[0132] The silicone mixture was degassed and cast into a cylindrical phantom with a diameter of 50 mm and a height of 30 mm. The compression test was performed on a Zwick / Roell Z1.0 using a 1 kN load cell. It should be understood that the Zwick / Roell Z1.0 compression test is a material testing method that uses Zwick / Roell testing equipment and software to evaluate the performance of materials under compressive stress. The specific operation involves placing the sample to be tested in the center area of ​​the fixed platform and filling in the relevant information of the sample on the software, such as thickness, width and length.

[0133] During the compression test, the motion module of the test machine applies the compressive force while measuring the stress change of the specimen. Through this test, the compressive stress curve of the specimen can be obtained, which is usually divided into three stages: linear elastic section, plateau section and densification section. The linear elastic stage is relatively short and mainly shows the stress change when the bracket is intact; the plateau stage is the time period between the bracket being intact and the hole wall starting to collapse; the densification stage starts when the hole wall is damaged. As the test machine's moving module presses down, the gap becomes smaller and smaller, and the stress increases with the increase of strain. Specifically, the phantom is compressed by 20% from a preload of 0.1N. The elastic modulus is calculated during the first 10% of the compression. Each phantom is repeated 10 times, 5 times on each side. The first repetition on each side is excluded, and there are a total of 8 analyzed repetitions on each side. This produces 3976 rows of data for each phantom for calculating the elastic modulus, with an average standard deviation of 0.431%.

[0134] Table 1: Materials and elastic modulus of the soft phantom

[0135]

[0136]

[0137] Experiment 1A-2) Use internal pressure to measure the displacement of the visual tracking point.

[0138] The relationship between the displacement and internal pressure of the pressure modulated optical tracking hardness sensor was evaluated as a function of the elastic modulus of the phantom when in contact with samples of different hardness. First, the pressure modulated optical tracking hardness sensor was set parallel and coaxial to the phantom, and the membrane was slowly inflated while the displacement of the visual tracking point was continuously monitored. When the height of the membrane exceeded 10 mm, the position of the visual tracking point was recorded for calibration. After applying a layer of lubricant on the elastic membrane of the sensor and the phantom, the pressure modulated optical tracking hardness sensor was lowered into contact with the phantom.

[0139] like Figure 6 As shown, the solid circular trajectory line represents the original position of the visual tracking point, and the dotted circular trajectory line represents the position of the visual tracking point after displacement.

[0140] The visual tracking point is inside the circle and its pixel range is checked to be between 24 and 29. This range was chosen because it was met by all phantoms in 20 calibration attempts and it is the upper limit that can be achieved by the softest phantom. After every five tests, the robot's motion pattern was reprogrammed until the average displacement was within the previously mentioned range, ensuring repeatable contact forces. Once in contact, the internal pressure was increased in steps of 0.061 kPa. In each step, a pause of 1 s was made to adjust the membrane, and then the position of the visual tracking point was measured five times with a 0.2 s interval between measurements. This process was repeated until the pressure reached 9.77 kPa, and then the experiment was reset. These experiments were performed on every other phantom listed in the left column of Table 1 above. Each phantom was tested five times, resulting in 805 rows of data per phantom.

[0141] exist Figure 6 In the , the position data of each visual tracking point at each pressure is extracted and converted into a polar coordinate system around the center point. The radial displacement Δr is measured in pixels px and the angular displacement θ is measured in degrees (°). The radial displacement Δr measures the change in the distance of the tracking point from the center point, while the angular displacement θ measures the change in angular displacement around a circle starting from the center point. This makes it possible to evaluate the movement of the visual tracking point while also eliminating torsion effects on the membrane. The average radial displacement Δr is obtained by averaging the Δr of all tracking points except the center point at each measurement.

[0142] Experiment 1A-3) Stiffness sensing method: In order to reduce the time required to measure tissue stiffness, four internal pressures of the sensor (0kPa, 2kPa, 4kPa, and 6kPa) were used for sensing. At each of the four pressures, the membrane was given an adjustment time of 0.1s; then, the position of the visual tracking point was measured four times, with an interval of 0.1s between each measurement. The data was extracted and converted into radial displacement Δr. The experiment was repeated 24 times on every other phantom in the left column of Table 1 to create training and validation data sets, with 20 training times and 4 validation times. A total of 140 rows of training data and 28 rows of validation data were generated. The experiment was then repeated 16 times on each phantom to generate a test data set. A total of 208 rows of test data were generated. Each experimental repetition generates a small amount of data, while neural networks require a larger amount of data than other models when training and testing. Therefore, this experiment was repeated more times than other experiments.

[0143] Experiment 1A-4) Structure and training of recurrent neural networks: The architecture of the network is as follows Figure 3 As shown. The average tracking point displacement at each pressure is passed as input to the neural network. The feed-forward fully connected recurrent neural network has three hidden layers, each with five nodes. Each node applies a linear activation function to its input according to the weight established in training. The network is trained using the corresponding phantom elasticity obtained from compression testing as the ground truth output. Prior to training, all data are normalized between -1 and 1.

[0144] The input layer of the feedforward fully connected recurrent neural network (NN) has 4 input nodes, corresponding to the average radial displacement of the visual tracking point under internal pressures of 0 kPa, 2 kPa, 4 kPa, and 6 kPa. The data measured by the sensors are input into 3 hidden layers, each with 5 fully connected nodes, from which the phantom stiffness is output.

[0145] A feed-forward fully connected recurrent neural network (NN) was built using the fitrnet function of MATLAB. The function does not allow for setting the batch size, so each batch of training data was used for one iteration. Supervised learning was performed using the L-BFGS algorithm, and the results were compared with the validation data every five iterations. The training was repeated until the function loss was less than 1e-16. The trained neural network produced an RMSE (Root Mean Squared Error) of 0.321 kPa on the validation data.

[0146] Experiment 1B. Tissue stiffness sensing results.

[0147] Experiment 1B-1) Using internal pressure to measure displacement

[0148] Figure 7 Each phantom is drawn in -Internal pressure curve. -The pressure curve changes with the change of the phantom hardness. Table 2 shows the slope of the linear fit of the points of each phantom at 1 kPa internal pressure. As the hardness of the phantom increases, the slope increases.

[0149] Table 2: Initial pressure-displacement gradient (up to 1 KPA) and phantom hardness

[0150]

[0151] Experiment 1B-2) Hardness Sensing:

[0152] Use internal pressure to measure displacement, Hardness Sensing: Test data was input into the neural network and the output was analyzed to evaluate the hardness sensing capability of the sensor. To quantify the accuracy of the sensor, the output was compared to the phantom hardness to calculate the root mean square error (RMSE) and non-repeatability for each phantom and phantom range. The average RMSE was 2.47 kPa and the non-repeatability was 16.24% of the phantom range.

[0153] Table 3 shows the root mean square error (RMSE) for each phantom stiffness. The average RMSE for the trained phantoms was 0.25 kPa and the average RMSE for the untrained phantoms was 5.07 kPa (2.85% of the range). For the untrained data, the Pearson product moment correlation coefficient indicated that the accuracy of the sensor decreased with increasing phantom stiffness, r(95) = 0.65, p < 0.001. In addition, if the sensor range was limited to 64 kPa, the sensor performance improved, with the average RMSE decreasing to 0.98 kPa, while the non-repeatability became 4.68% of the range. To compensate for this nonlinear behavior, the root mean square errors (RMSEs) were also calculated as a percentage of the phantom stiffness for each reading, which is also shown in Table 3. The average percent root mean square error (%RMSE) for all untrained phantoms was 7.72% of the sample stiffness. For comparison with those papers that do not calculate RMSE, the mean absolute error (MAE) for all untrained phantoms was also calculated. The mean absolute error is 4.55 kPa, which is equivalent to 7.07% of the sample hardness or 2.56% of the sensor range.

[0154] Table 3: Accuracy of hardness sensing

[0155]

[0156] Experiment 1C. Tissue Stiffness Sensing Discussion.

[0157] Experiment 1C-1) Using internal pressure to measure displacement, Δr:

[0158] Figure 7 The shape of the Δr-internal pressure curve is shown to change as the stiffness of the phantom changes. It can be seen that as the stiffness of the phantom increases, the initial slope of the curve also increases. This is further demonstrated in Table 2. Although the stiffness of the phantom increases exponentially, the slope does not increase, indicating a nonlinear relationship. These results indicate that the PMOT sensor will be able to sense the stiffness of the phantom at a selected known pressure using Δr, and that lower stiffness will be sensed more accurately than higher stiffness.

[0159] Experiment 1C-2) Hardness Sensing:

[0160] From the results in Table 3, it can be seen that the PMOT sensor can measure the hardness of the phantom, especially at low hardness, and can identify healthy and unhealthy tissues. The error of the untrained phantom is higher than that of the trained phantom, which indicates that the model is biased towards data similar to its training set. This difference can be reduced by introducing more intermediate training samples. Compared with the sensor sensing performance recorded in the existing literature, the RMSE of the PMOT sensor on the untrained phantom is 2.85% of the sensor range, which is lower than 5.95% in the prior art; the MAE of the PMOT sensor on the untrained phantom is 7.07% of the sample hardness, which is comparable to the error range in the prior art.

[0161] like Figure 8 As shown, a schematic diagram of a process flow of a tissue hardness detection method in an embodiment of the present application is shown. The process flow method in this embodiment is applied to a computer device; the computer device is connected to the PMOT sensor in the above embodiment. The detection method includes the following steps:

[0162] Step S801: Acquire at least two sets of sensor data; each set of sensor data corresponds to internal hardness data under each internal pressure and displacement data of a visual tracking point.

[0163] Step S802: Calculate and generate the hardness of the tissue to be tested based on the internal hardness data, the displacement data of the visual tracking point, and the difference between the embedding depth data of the sensor corresponding to the tissue to be tested.

[0164] Furthermore, when the tissue to be tested contacts a non-flat dome-shaped membrane, the average radial displacement of the visual tracking points of the pressure-modulated optical tracking hardness sensor under multiple internal pressures is input into a trained recurrent neural network, and the recurrent neural network outputs the final hardness data of the tissue to be tested after nonlinear calculation.

[0165] Furthermore, the recurrent neural network includes a feedforward fully connected recurrent neural network, which includes an input layer, multiple hidden layers, and an output layer; the input layer includes multiple input nodes, each hidden layer includes multiple hidden nodes, and each hidden node applies a linear activation function to its input data according to the weights established in training; the output layer outputs the predicted value, and uses the test results of the compression test as the benchmark true value for model training.

[0166] It should be noted that the implementation principle and process of the tissue hardness detection method in the embodiment of the present application are similar to those of the PMOT sensor mentioned above, and will not be repeated here.

[0167] like Fig. 9 As shown, a schematic diagram of the structure of a computer device in an embodiment of the present application is shown. The computer device 900 can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The computer device 900 includes: at least one processor 901, a memory 902, at least one network interface 904 and a user interface 906. The various components in the device are coupled together through a bus system 905. It can be understood that the bus system 905 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 905 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Fig. 9 In the specification, various buses are labeled as bus systems.

[0168] The user interface 906 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0169] It is understood that the memory 902 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0170] The memory 902 in the embodiment of the present invention is used to store various categories of data to support the operation of the computer device 900. Examples of these data include: any executable program for operating on the computer device 900, such as an operating system 9021 and an application 9022; the operating system 9021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 9022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The tissue hardness detection method provided in the embodiment of the present invention can be included in the application 9022.

[0171] The method disclosed in the above embodiment of the present invention can be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 901 or the instruction in the form of software. The above processor 901 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 901 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0172] In an exemplary embodiment, the computer device 900 may be one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0173] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0174] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0175] In summary, the present application provides a pressure-modulated optical tracking hardness sensor, a hardness detection method, a medium and an apparatus. The present application can quantify the elasticity of a phantom without an external position reference, so the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0176] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A pressure modulated optical tracking hardness sensor, characterized in that: include: Sensor housing; An elastic membrane is provided at one end of the sensor housing, and a plurality of visual tracking points are provided on the inner surface of the elastic membrane; An image acquisition device and a pneumatic tube are inserted into the other end of the sensor housing; the pneumatic tube is connected to a pressure regulator and the interior of the sensor; the pressure regulator controls the internal pressure of the sensor to determine the hardness of the tissue to be tested by adjusting the compliance of the sensor.

2. The pressure modulated optical tracking hardness sensor according to claim 1, characterized in that: The compliance of the sensor increases with increasing internal pressure; the increase in compliance is manifested as: The internal gas mass increases with the internal pressure, which in turn increases the internal hardness of the sensor; The stress of the elastic membrane increases with the increase of internal pressure, which in turn causes the visual tracking point on the inner surface of the elastic membrane to displace.

3. The pressure modulated optical tracking hardness sensor according to claim 2, characterized in that: The process of determining the hardness of the tissue to be measured by adjusting the compliance of the sensor includes: The internal pressure of the sensor is controlled by the pressure regulator to obtain internal hardness data and displacement data of the visual tracking point, wherein the acquisition method includes: measuring the internal hardness data and displacement data of the visual tracking point under two consecutive internal pressures, and repeating this process to collect multiple sets of data, and calculating the characteristic values ​​of the multiple sets of data, thereby obtaining the final internal hardness data and displacement data of the visual tracking point; The hardness of the tissue to be measured is calculated and generated based on the internal hardness data, the displacement data of the visual tracking point, and the difference between the embedding depth data of the sensor corresponding to the tissue to be measured.

4. The pressure modulated optical tracking hardness sensor according to claim 3, characterized in that: The calculation method of the hardness of the tissue to be tested includes: Among them, k t Indicates the hardness of the tissue to be tested; k s1 represents the first spring stiffness, which refers to the internal hardness of the sensor at the first internal pressure; k s2 represents the second spring stiffness, which refers to the internal hardness of the sensor under the second internal pressure; ΔX s1 Indicates the first spring deformation, which refers to the displacement data generated by the visual tracking point under the first internal pressure; ΔX s2 Indicates the deformation of the second spring, which refers to the displacement data generated by the visual tracking point under the second internal pressure.

5. The pressure modulated optical tracking hardness sensor according to claim 2, characterized in that: The internal gas mass increases with the increase of internal pressure, which includes: Among them, n represents the internal gas mass; R represents the ideal gas constant, T represents the gas temperature, P represents the internal gas pressure, and V represents the internal gas volume.

6. The pressure modulated optical tracking hardness sensor according to claim 1, characterized in that: A plurality of visual tracking points are arranged on the inner surface of the elastic membrane, one of which is located at the center of the inner surface, and the other visual tracking points are evenly arranged on a circumference around the center of the inner surface.

7. The pressure modulated optical tracking hardness sensor according to claim 1, characterized in that: The sensor housing is printed from a light-curing resin material.

8. The pressure modulated optical tracking hardness sensor according to claim 1, characterized in that: The image acquisition device includes a USB endoscope camera.

9. The pressure modulated optical tracking hardness sensor according to claim 1, characterized in that: The elastic membrane is made of platinum-catalyzed silicone rubber Ecoflex 00-50 material.

10. A method for detecting tissue hardness, characterized in that: Applied to computer equipment; the computer equipment is connected to the pressure modulated optical tracking hardness sensor as described in any one of claims 1 to 9; the detection method comprises: Acquire at least two sets of sensor data; each set of sensor data corresponds to internal hardness data under each internal pressure and displacement data of a visual tracking point; The hardness of the tissue to be tested is calculated based on the internal hardness data, the displacement data of the visual tracking point, and the difference between the embedding depth data of the sensor corresponding to the tissue to be tested.

11. The tissue hardness detection method according to claim 10, characterized in that: When the tissue to be tested contacts a non-flat dome-shaped membrane, the average radial displacement of the visual tracking points of the pressure-modulated optical tracking hardness sensor under multiple internal pressures is input into a trained recurrent neural network, which then outputs the final hardness data of the tissue to be tested after nonlinear calculation.

12. The tissue hardness detection method according to claim 11, characterized in that: The recurrent neural network comprises a feedforward fully connected recurrent neural network, which comprises an input layer, a plurality of hidden layers, and an output layer; the input layer comprises a plurality of input nodes, each hidden layer comprises a plurality of hidden nodes, each hidden node applies a linear activation function to its input data according to a weight established in training; the output layer outputs a predicted value, and performs model training using the test result of the compression test as a benchmark true value.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the tissue hardness detection method according to any one of claims 10 to 12 is implemented.

14. A computer program product, characterized in that A method for detecting tissue hardness according to any one of claims 10 to 12.

15. A computer device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the computer device executes the tissue hardness detection method according to any one of claims 10 to 12.

Images