Acupuncture point pose estimation method and system based on pseudo stereoscopic vision

Abstract

The invention discloses an acupoint pose estimation method and system based on pseudo-stereoscopic vision, and relates to the technical field of robot control, and the method comprises the steps: driving a global camera and a hand-eye camera to dynamically collect a patient image; acquiring a closed solution of an acupoint elliptical contour parameter according to the patient image based on the arithmetic distance; optimizing the closed solution based on approximate geometric iteration to obtain an optimal solution of the acupuncture point elliptic contour parameters; reconstructing according to the optimized solution based on monocular vision to obtain multiple groups of solutions of the acupoint position and the normal vector; determining unique solutions of the radius, the position and the normal vector of the acupoint according to the multiple groups of solutions based on pseudo-stereoscopic vision; and optimizing the pose of the acupoint according to the unique solution of the radius, the position and the normal vector of the acupoint. The method can accurately estimate the three-dimensional position and the two-dimensional normal vector of the acupoint at the same time, can improve the estimation precision of the acupoint pose, and is of great significance for improving the performance and clinical application value of a traditional Chinese medicine robot.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a method and system for estimating acupoint posture based on pseudo-stereo vision. Background Art

[0002] As a fusion of traditional Chinese medicine and modern technology, TCM robots are of great significance in promoting the modernization of TCM. They can achieve precise acupoint positioning and treatment operations, improve the standardization of diagnosis and treatment, reduce human errors, and enable TCM to better serve patients around the world. Acupoint pose estimation technology is the core of TCM robots, and its accuracy directly affects the treatment effect and safety. This technology aims to accurately reconstruct the three-dimensional position and two-dimensional normal vector of acupoints, so that robots can accurately perform acupuncture, massage and other operations. However, existing technologies have many problems. Monocular vision-based methods cannot capture scale information, and the offline model differs from the actual patient, limiting the accuracy of position estimation. Binocular vision-based methods recover acupoint scale information through redundant feature matching, but cannot estimate the normal vector of the acupoint, which is crucial for precise treatment. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a method and system for estimating acupoint posture based on pseudo-stereo vision to improve the estimation accuracy of acupoint posture.

[0004] To achieve the above objectives, an embodiment of the present application provides a method for estimating acupoint posture based on pseudo-stereoscopic vision, the method comprising the following steps:

[0005] Drive the global camera and hand-eye camera to dynamically capture patient images;

[0006] Obtaining a closed solution for the acupoint ellipse contour parameters from the patient image based on the arithmetic distance;

[0007] Optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters;

[0008] Reconstructing multiple sets of solutions of acupoint positions and normal vectors based on the optimized solution based on monocular vision;

[0009] Determine a unique solution for acupuncture point radius, position and normal vector based on the multiple sets of solutions based on pseudo-stereoscopic vision;

[0010] The pose of the acupoint is optimized according to the unique solution of the acupoint radius, position and normal vector.

[0011] In some embodiments, obtaining a closed solution for the acupoint ellipse contour parameters according to the patient image based on the arithmetic distance comprises the following steps:

[0012] Define variable ε = [a,b,c,d,e,f]T , for the pixel point s of the elliptical contour of the i-th acupoint in the patient image i =[u i ,v i ] T Satisfies the following vector form:

[0013]

[0014] Defining variables The pixel points of all acupoint elliptical contours are defined as:

[0015]

[0016] Defining variables The parameter ε of the acupoint ellipse contour is optimized to minimize the bi-norm error of Gε, and the discriminant constraint of the ellipse is introduced;

[0017] Among them, the constrained optimization problem about ε is:

[0018]

[0019] Among them, D A (ε) is the arithmetic distance of the acupoint ellipse contour fitting:

[0020] ζ is the parameter matrix of the elliptic discriminant constraint,

[0021] Define λ + and q + Represents G T G has a unique positive eigenvalue relative to ζ and a corresponding unit eigenvector. According to the relevant theorem of ellipse fitting, the optimal solution of the constrained optimization problem is determined to be ε * =q + .

[0022] In some embodiments, optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters comprises the following steps:

[0023] Iteratively optimizing the parameters of the closed solution of the acupoint ellipse contour to minimize the Sampson distance of the acupoint ellipse contour fitting;

[0024] The expression for minimizing the Sampson distance of the acupoint ellipse contour fitting is:

[0025]

[0026] stε T ζε>0;

[0027] Among them, DS (ε) is the Sampson distance of the acupoint ellipse contour fitting;

[0028] W i is g i is s i is the partial derivative matrix of s

[0029] The elliptic discriminant constraint in the minimization of the Sampson distance of the acupoint ellipse contour fitting is converted into a penalty term of the Sampson distance D S (ε) by using the penalty function method, and an unconstrained optimization problem is obtained as follows:

[0030]

[0031] where σ S is the penalty factor of the elliptic discriminant constraint, and σ S > 0;

[0032] The Levenberg-Marquardt algorithm is used to solve the unconstrained optimization problem, including the following steps:

[0033] The partial derivative matrix M of h(ε) with respect to ε is calculated:

[0034]

[0035] where h i (ε) is the i-th element of h(ε);

[0036] Define t max , which represents the iteration number, ε t , which represents the acupoint ellipse contour parameter at time t, and M t , which represents the partial derivative matrix of h(ε t ) with respect to ε t , where t = 1, 2, …, t max ; further, at time t, the update equation of the acupoint ellipse contour parameter is represented as:

[0037]

[0038] where α t is the damping factor, and α t > 0.

[0039] In some embodiments, the plurality of solutions of acupoint positions and normal vectors reconstructed according to the optimization solution based on monocular vision includes the following steps:

[0040] Define the quadratic form of the general equation of an ellipse as:

[0041]

[0042] Wherein, E is the parameter matrix of the quadratic form of the general elliptic equation;

[0043] Define p = [x, y, z] T Representing the Cartesian coordinates corresponding to the pixel coordinate s in the camera space, the camera projection model is obtained as:

[0044]

[0045] Among them, (f u ,f v ,u0,v0) are the internal parameters of the camera, and the internal parameters constitute the internal parameter matrix K;

[0046] Combining the quadratic form of the general equation of the ellipse and the projection model of the camera, the quadratic form equation of the oblique cone is obtained as follows:

[0047] p T Hp=0;

[0048] Where H is the parameter matrix of the oblique conic quadratic equation, H = K T EK;

[0049] Define λ1, λ2, and λ3 to represent the three eigenvalues ​​of H, and λ1≥λ2>0>λ3, q1, q2, and q3 to represent the corresponding unit eigenvectors. Furthermore, according to the quadratic equation of the oblique cone in the camera space, the normalized transformation matrix of the quadratic equation of the oblique cone is expressed as P=[o1,o2,o3], where q 3,z represents the third element of q3, sgn(·) represents the sign function, o2=q2, o1=o2×o3;

[0050] Furthermore, the normal vector and position of the acupoint are expressed as:

[0051]

[0052]

[0053] Among them, r ap is the radius of the acupoint.

[0054] In some embodiments, the method of determining a unique solution for acupoint radius, position, and normal vector from the multiple sets of solutions based on pseudo-stereoscopic vision comprises the following steps:

[0055] definition hec R gc and hec t gc They represent the rotation matrix and translation vector of the coordinate system {gc} relative to the coordinate system {hec} respectively;

[0056] The normal vectors of acupoints measured by different monocular vision are defined in the coordinate system {hec} as follows:

[0057]

[0058] By determining the minimum angle between the normal vectors of the acupoints, the unique solution of the acupoint posture is obtained; and The angle in the coordinate system {hec} is expressed as:

[0059]

[0060] in,<x,y> is the inner product between vectors x and y;

[0061] The position of the acupuncture points is defined in the coordinate system {hec} as:

[0062]

[0063] The translation vector obtained according to the normal vector and position of the acupoint is:

[0064]

[0065] Defining variables According to the relevant lemma of generalized inverse, the least squares solution of the translation vector is determined to be

[0066] A multi-constrained nonlinear acupoint position optimization model is established to optimize the parameters of the acupoint ellipse contour and the radius of the acupoint, thereby minimizing the joint error of the acupoint position.

[0067] The acupoint posture optimization model is:

[0068]

[0069] The particle swarm optimization algorithm is used to solve the acupoint posture optimization model to obtain a unique solution for the acupoint radius, position and normal vector.

[0070] To achieve the above objectives, another aspect of the present application provides an acupoint position estimation system based on pseudo-stereoscopic vision, the system comprising:

[0071] An image acquisition unit, used to drive the global camera and the hand-eye camera to dynamically acquire patient images;

[0072] a closed solution determining unit, configured to obtain a closed solution of the acupoint ellipse contour parameters according to the patient image based on arithmetic distance;

[0073] an optimization solution determination unit configured to optimize the closed solution based on approximate geometric iteration to obtain an optimization solution of the acupoint elliptical contour parameter;

[0074] a multiple-group solution determination unit configured to reconstruct, based on monocular vision, a multiple-group solution of acupoint position and normal vector from the optimization solution;

[0075] a unique solution determination unit configured to determine, based on pseudo-stereovision, a unique solution of acupoint radius, position and normal vector from the multiple-group solution;

[0076] a pose determination unit configured to optimize the pose of the acupoint according to the unique solution of acupoint radius, position and normal vector.

[0077] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0078] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0079] The embodiments of the present application at least have the following beneficial effects:

[0080] The present application can drive a global camera and a hand-eye camera to dynamically collect patient images, obtain a closed solution of acupoint elliptical contour parameter from the patient images based on arithmetic distance, optimize the closed solution based on approximate geometric iteration to obtain an optimization solution of the acupoint elliptical contour parameter, reconstruct, based on monocular vision, a multiple-group solution of acupoint position and normal vector from the optimization solution, determine, based on pseudo-stereovision, a unique solution of acupoint radius, position and normal vector from the multiple-group solution, and optimize the pose of the acupoint according to the unique solution of acupoint radius, position and normal vector. The present application can simultaneously accurately estimate the three-dimensional position and two-dimensional normal vector of the acupoint, can improve the estimation accuracy of the acupoint pose, and has important significance for improving the performance and clinical application value of a traditional Chinese medicine robot. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0082] Figure 1 A flowchart of an acupoint pose estimation method based on pseudo-stereovision provided by the embodiments of the present application;

[0083] Figure 2 An example flow chart of a method for estimating acupoint posture based on pseudo-stereoscopic vision provided in an embodiment of the present application;

[0084] Figure 3 The orthogonal distance histogram of the acupoint ellipse contour fitting provided in the embodiment of the present application;

[0085] Figure 4 The error histogram of acupoint position estimation provided in the embodiment of the present application;

[0086] Figure 5 A schematic diagram of the structure of an acupoint posture estimation system based on pseudo-stereoscopic vision provided in an embodiment of the present application;

[0087] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0089] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0090] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0091] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0092] Some parameters are described as follows:

[0093] {0} and {e} represent the base coordinate system and end coordinate system of the manipulator, respectively;

[0094] {gc} represents the global camera coordinate system;

[0095] {hec} represents the coordinate system of the hand-eye camera;

[0096] {an} represents the coordinate system of the end effector of the robot arm;

[0097] {ap} represents the coordinate system of the acupoint.

[0098] Reference Figure 1 The embodiment of the present application provides a method for estimating acupoint position based on pseudo-stereoscopic vision. The method may include but is not limited to S100 to S150, as follows:

[0099] S100: drives the global camera and hand-eye camera to dynamically acquire patient images;

[0100] S110: Obtaining a closed solution of acupoint ellipse contour parameters according to the patient image based on arithmetic distance;

[0101] S120: Optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters;

[0102] S130: reconstructing multiple sets of solutions of acupoint positions and normal vectors according to the optimized solution based on monocular vision;

[0103] S140: Determine a unique solution for acupoint radius, position, and normal vector according to the multiple sets of solutions based on pseudo-stereoscopic vision;

[0104] S150: Optimizing the position and posture of the acupoint according to the unique solution of the acupoint radius, position and normal vector.

[0105] Optionally, the step of obtaining a closed-form solution of acupoint ellipse contour parameters from the patient image based on the arithmetic distance comprises the following steps:

[0106] Define variable ε = [a,b,c,d,e,f] T , for the pixel point s of the elliptical contour of the i-th acupoint in the patient image i =[u i ,v i ]T Satisfies the following vector form:

[0107]

[0108] Defining variables The pixel points of all acupoint elliptical contours are defined as:

[0109]

[0110] Defining variables The parameter ε of the acupoint ellipse contour is optimized to minimize the bi-norm error of Gε, and the discriminant constraint of the ellipse is introduced;

[0111] Among them, the constrained optimization problem about ε is:

[0112]

[0113] Among them, D A (ε) is the arithmetic distance of the acupoint ellipse contour fitting:

[0114] ζ is the parameter matrix of the elliptic discriminant constraint,

[0115] Define λ + and q + Represents G T G has a unique positive eigenvalue relative to ζ and a corresponding unit eigenvector. According to the relevant theorem of ellipse fitting, the optimal solution of the constrained optimization problem is determined to be ε * =q + .

[0116] Optionally, optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters comprises the following steps:

[0117] Iteratively optimizing the parameters of the closed solution of the acupoint ellipse contour to minimize the Sampson distance of the acupoint ellipse contour fitting;

[0118] The expression for minimizing the Sampson distance of the acupoint ellipse contour fitting is:

[0119]

[0120] stε T ζε>0;

[0121] Among them, D S (ε) is the Sampson distance of the acupoint ellipse contour fitting;

[0122] W i g iThe partial derivative matrix of s i ,

[0123] The penalty function method is used to convert the ellipse discriminant constraint in the Sampson distance of the minimum acupoint ellipse contour fitting into a penalty term of the Sampson distance D S (ε) to obtain an unconstrained optimization problem as follows:

[0124]

[0125] Wherein, σ S is a penalty factor of the ellipse discriminant constraint, and σ S >0;

[0126] The Levenberg-Marquardt algorithm is used to solve the unconstrained optimization problem, including the following steps:

[0127] The partial derivative matrix M of h(ε) with respect to ε is calculated:

[0128]

[0129] Wherein, h i (i)(ε) is the i-th element of h(ε);

[0130] Define t max , which represents the iteration number, ε t , which represents the acupoint ellipse contour parameter at t time, M t , which represents the partial derivative matrix of h(ε t ) with respect to ε t , wherein t=1,2,…,t max ; further, at t time, the update equation of the acupoint ellipse contour parameter is represented as:

[0131]

[0132] Wherein, α t is a damping factor, and α t >0.

[0133] Optionally, the plurality of sets of solutions of acupoint positions and normal vectors reconstructed according to the optimization solution based on monocular vision includes the following steps:

[0134] Define the quadratic form of the ellipse general equation as:

[0135]

[0136] Wherein, E is a parameter matrix of the quadratic form of the ellipse general equation;

[0137] Define p=[x,y,z] TRepresenting the Cartesian coordinates corresponding to the pixel coordinate s in the camera space, the camera projection model is obtained as:

[0138]

[0139] Among them, (f u ,f v ,u0,v0) are the internal parameters of the camera, and the internal parameters constitute the internal parameter matrix K;

[0140] Combining the quadratic form of the general equation of the ellipse and the projection model of the camera, the quadratic form equation of the oblique cone is obtained as follows:

[0141] p T Hp=0;

[0142] Where H is the parameter matrix of the oblique conic quadratic equation, H = K T EK;

[0143] Define λ1, λ2, and λ3 to represent the three eigenvalues ​​of H, and λ1≥λ2>0>λ3, q1, q2, and q3 to represent the corresponding unit eigenvectors. Furthermore, according to the quadratic equation of the oblique cone in the camera space, the normalized transformation matrix of the quadratic equation of the oblique cone is expressed as P=[o1,o2,o3], where q 3,z represents the third element of q3, sgn(·) represents the sign function, o2=q2, o1=o2×o3;

[0144] Furthermore, the normal vector and position of the acupoint are expressed as:

[0145]

[0146] Among them, r ap is the radius of the acupuncture point.

[0147] Optionally, determining a unique solution of acupoint radius, position and normal vector according to the multiple sets of solutions based on pseudo-stereoscopic vision comprises the following steps:

[0148] definition hec R gc and hec t gc They represent the rotation matrix and translation vector of the coordinate system {gc} relative to the coordinate system {hec} respectively;

[0149] The normal vectors of acupoints measured by different monocular vision are defined in the coordinate system {hec} as follows:

[0150]

[0151] By determining the minimum angle between the normal vectors of the acupoints, the unique solution of the acupoint posture is obtained; and The angle in the coordinate system {hec} is expressed as:

[0152]

[0153] in,<x,y> is the inner product between vectors x and y;

[0154] The position of the acupuncture points is defined in the coordinate system {hec} as:

[0155]

[0156] The translation vector obtained according to the normal vector and position of the acupoint is:

[0157]

[0158] Defining variables According to the relevant lemma of generalized inverse, the least squares solution of the translation vector is determined to be

[0159] A multi-constrained nonlinear acupoint position optimization model is established to optimize the parameters of the acupoint ellipse contour and the radius of the acupoint, thereby minimizing the joint error of the acupoint position.

[0160] The acupoint posture optimization model is:

[0161]

[0162] The particle swarm optimization algorithm is used to solve the acupoint posture optimization model to obtain a unique solution for the acupoint radius, position and normal vector.

[0163] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0164] Reference Figure 2 , this embodiment may include the following steps:

[0165] Drive the global camera and hand-eye camera to acquire real-time patient images;

[0166] A closed-form solution for obtaining the contour parameters of acupoint ellipse based on arithmetic distance;

[0167] Optimize the parameters of the acupoint ellipse contour based on approximate geometry iteration;

[0168] Reconstruct multiple sets of solutions of acupoint positions and normal vectors based on monocular vision;

[0169] Determine the unique solution of acupoint radius, position and normal vector based on pseudo-stereoscopic vision;

[0170] The position of the acupoint is optimized based on multi-view fusion.

[0171] Specifically, the various technical solutions of the embodiment are described as follows:

[0172] 1. Closed solution for obtaining acupoint ellipse contour parameters based on arithmetic distance.

[0173] Suppose s = [u, v] T represents the pixel coordinates of any point on the ellipse, which satisfies the general equation of the ellipse, that is:

[0174] au 2 +buv+cv 2 +du+ev+f=0(1)

[0175] In the formula, (a, b, c, d, e, f) are parameters of the general equation of the ellipse, which satisfy the discriminant Δ = b 2 -4ac<0.

[0176] Suppose that n pixel points of acupoint ellipse contours can be observed in the image, and define the variable ε = [a, b, c, d, e, f] T For the pixel point s i = [u i ,v i ] T of the i-th acupoint ellipse contour, the vector form of the general equation of the ellipse (3-1) can be expressed as:

[0177]

[0178] Define the variable For all pixel points of the acupoint ellipse contours, formula (2) can be extended as:

[0179]

[0180] Define the variable Due to noise interference and discretization processing in the camera imaging process, formula (3) is usually not established. At this time, the parameters ε of the acupoint ellipse contour need to be optimized to minimize the two-norm error of Gε. In addition, in order to avoid producing false fitting results (i.e., hyperbola and parabola), it is necessary to introduce the discriminant constraint of the ellipse. Without loss of generality, it can be converted into a special equality constraint.

[0181] Thus, a constrained optimization problem about ε is established, that is:

[0182]

[0183] In the formula, D A(ε) is the arithmetic distance of the acupoint ellipse contour fitting;

[0184] ζ is the parameter matrix of the elliptic discriminant constraint, where:

[0185]

[0186] Assume λ + and q + Represents G T G has a unique positive eigenvalue relative to ζ and a corresponding unit eigenvector. According to the relevant theorem of ellipse fitting, the optimal solution of the optimization problem of formula (4) is ε * =q + .

[0187] 2. Optimize the parameters of the acupoint elliptical contour based on approximate geometry iteration.

[0188] The Sampson distance is an approximate geometric distance that is equivalent to the orthogonal distance under the assumption of independent and identically distributed Gaussian noise. It strikes a balance between real-time performance and accuracy, making it suitable for robotic applications. Therefore, the parameters of the acupoint ellipse contour are iteratively optimized based on the Sampson distance. Similarly, a constrained optimization problem with respect to ε is established to minimize the Sampson distance of the acupoint ellipse contour fit, namely:

[0189]

[0190] stε T ζε>0;

[0191] Where D S (ε) is the Sampson distance of the acupoint ellipse contour fitting;

[0192] W i g i to s i The partial derivative matrix of

[0193] The penalty function method is used to convert the elliptic discriminant constraint in problem (3-6) into the Sampson distance D S (ε) penalty term. Therefore, it can be converted into an unconstrained optimization problem, that is:

[0194]

[0195] Where σ S is the penalty factor of the elliptic discriminant constraint, σ S >0.

[0196] Problem (7) is an unconstrained quadratic programming problem. Therefore, the Levenberg-Marquardt algorithm can solve this problem. First, calculate the partial derivative matrix M of h(ε) with respect to ε, that is:

[0197]

[0198] Where h i (ε) is the i-th element of h(ε).

[0199] Assume t max represents the number of iterations, ε t represents the acupoint ellipse contour parameter at time t, M t represents h(ε t ) for ε t The partial derivative matrix of , where t=1,2,…,t max Therefore, at time t, the update equation of the acupoint ellipse contour parameters can be expressed as:

[0200]

[0201] Where α t is the damping factor, α t >0.

[0202] The fitting algorithm based on arithmetic distance provides an iterative initial value that belongs to the feasible domain of problem (3-6), while the penalty term in formula (7) prevents the optimization variable ε from moving towards ε. T ζε = 0. Therefore, it ensures that the fitting result is an ellipse.

[0203] 3. Reconstruct multiple sets of solutions of acupoint positions and normal vectors based on monocular vision.

[0204] The quadratic form of the elliptic general equation of formula (1) can be expressed as:

[0205]

[0206] Where E is the parameter matrix of the elliptic quadratic equation.

[0207] Assume p = [x, y, z] T Represents the Cartesian coordinate corresponding to the pixel coordinate s in the camera space. The projection model of the camera can be expressed as:

[0208]

[0209] Where (f u ,f v ,u0,v0) are the intrinsic parameters of the camera, which constitute the intrinsic parameter matrix K.

[0210] Combining equation (12) and equation (13), the quadric equation of oblique circular cone can be expressed as:

[0211] p T Hp=0(14)

[0212] where H is the parameter matrix of the quadric equation of oblique circular cone, H=K T EK。

[0213] Assume that λ1, λ2 and λ3 represent three eigenvalues of H respectively, and λ1≥λ2>0>λ3, q1, q2 and q3 represent the corresponding unit eigenvectors respectively. Therefore, according to equation (14), the conversion matrix of the quadric standardization of oblique circular cone in the camera space can be expressed as P=[o1,o2,o3], wherein, q 3,z represents the 3rd element of q3, sgn(·) represents the sign function, o2=q2, o1=o2×o3.

[0214] Therefore, the normal vector and position of the acupoint can be respectively expressed as:

[0215]

[0216] where r ap is the radius of the acupoint.

[0217] According to equation (15) and equation (16), the monocular vision-based acupoint pose reconstruction has a multi-solution problem, and the normal vector and position of the acupoint cannot be uniquely determined. In addition, the radius of the acupoint needs to be measured in advance.

[0218] 4. Determining the unique solution of the radius, position and normal vector of the acupoint based on pseudo stereo vision.

[0219] Pseudo stereo vision has redundant measurement information, which can exclude the wrong solution of the acupoint pose, and can also estimate the radius of the acupoint. Assume that hec R gc and hec t gc respectively represent the rotation matrix and translation vector of the coordinate system {gc} relative to the coordinate system {hec}, and other related variables are shown in Table 1.

[0220]

[0221] Table 1

[0222] In theory, the acupoint normal vectors measured by different monocular visions are coincident relative to the same reference coordinate system. For example, in the coordinate system {hec}, there are:

[0223]

[0224] In fact, due to error interference and other factors, formula (17) is established, and there is a small angle deviation between the two. Based on this, the unique solution of the acupoint pose is obtained by judging the minimum included angle between the normal vectors of the acupoints. Among them, and The included angle in the coordinate system {hec} can be expressed as:

[0225]

[0226] In the formula, <x, y> is the inner product between vectors x and y.

[0227] Similarly, for the position of the acupoint, in the coordinate system {hec}:

[0228]

[0229] Combined with formula (16) and formula (19), the following can be obtained:

[0230]

[0231] Define the variable According to the relevant lemma of the generalized inverse, the least square solution of formula (20) is Up to now, the unique solution of the normal vector, position and radius of the acupoint has been obtained.

[0232] Due to the errors in the fitting of the acupoint ellipse contour and the internal and external parameter calibration of the camera, the acupoint poses reconstructed by different cameras are not coincident, that is, formula (17) and formula (19) are usually not established. Therefore, it is further necessary to optimize the parameters of the acupoint ellipse contour and the radius of the acupoint, so as to minimize the joint error of the acupoint pose. Thus, a multi-constrained nonlinear acupoint pose optimization model is established, that is:

[0233]

[0234] Finally, the particle swarm optimization algorithm is used to solve formula (21).

[0235] 5. Simulation.

[0236] In order to verify the correctness and effectiveness of the acupoint pose estimation method provided in the embodiment, simulation data of six acupoints is generated based on Matlab. Among them, AP1-AP6 represent six acupoints. In the hand-eye image, the visible area of AP1 and AP2 is less than 50%, the visible area of AP3 and AP4 is greater than 50%, and AP5 and AP6 are completely visible. On the global image, all acupoints are completely visible. Then, the pixel points on the acupoint ellipse contour are sampled respectively, and Gaussian noise with mean value of 0 and variance of 4 is added.

[0237] Finally, the position and normal vector of the acupoints are reconstructed using a method based on a space circle. The non-optimization method is used to compare with the proposed method. The comparison results of the acupoint ellipse contour fitting and pose estimation are shown in Figure 2. Figure 3 、 Figure 4 As shown. For AP1 and AP2 with smaller visible areas, the non-optimization method uses a fitting algorithm based on arithmetic distance, which easily leads to large errors or even mistakes in pose estimation, which will affect the subsequent acupoint tracking performance of the robot. The proposed method optimizes the acupoint ellipse contour parameters based on approximate geometric distance, thereby reducing the errors of the fitted orthogonal distance and pose estimation. For other acupoints, both methods can obtain pose estimation results with high accuracy. In particular, thanks to the joint optimization of acupoint position and normal vector, the pose estimation accuracy of the proposed method is better than that of the non-optimization method. In the actual acupoint tracking control process, the acupoint ellipse contour is transformed from an incompletely visible state to a fully visible state, and ultimately can meet the requirements of high-precision pose estimation.

[0238] Reference Figure 5 The embodiment of the present application further provides an acupoint position estimation system based on pseudo-stereoscopic vision, which can implement the above-mentioned acupoint position estimation method based on pseudo-stereoscopic vision. The system includes:

[0239] An image acquisition unit, used to drive the global camera and the hand-eye camera to dynamically acquire patient images;

[0240] a closed solution determining unit, configured to obtain a closed solution of the acupoint ellipse contour parameters according to the patient image based on arithmetic distance;

[0241] an optimized solution determining unit, configured to optimize the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters;

[0242] a multi-solution determination unit, configured to obtain multi-solutions of acupoint positions and normal vectors based on the optimized solution reconstruction based on monocular vision;

[0243] a unique solution determining unit, configured to determine a unique solution for acupoint radius, position, and normal vector from the multiple sets of solutions based on pseudo-stereoscopic vision;

[0244] A posture determination unit is used to optimize the posture of the acupoint according to the unique solution of the acupoint radius, position and normal vector.

[0245] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0246] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the present application. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0247] It can be understood that the contents of the above method embodiments are all applicable to the embodiments of the present device, the functions specifically implemented by the embodiments of the present device are the same as those of the method of the present application, and the beneficial effects achieved are also the same as those achieved by the method of the present application.

[0248] See also Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0249] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0250] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the methods of the embodiments of this application.

[0251] Input / output interface 603, used to implement information input and output;

[0252] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0253] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0254] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0255] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the present application is implemented.

[0256] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0257] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0258] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0259] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0260] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0261] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0262] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a recited step or its integral sub-steps or additional steps whether or not readily ascertainable from the description or the like. Further, the words "a" or "an", as used herein in the disclosure and elsewhere, are used indiscriminately and are to be interpreted in the same way, i.e. as meaning "one or more".

[0263] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0264] In several embodiments provided in the application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between systems or units, which can be electrical, mechanical or other forms.

[0265] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0266] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0268] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for estimating acupoint posture based on pseudo-stereo vision, characterized in that: The method comprises the following steps: Drive the global camera and hand-eye camera to dynamically capture patient images; Obtaining a closed solution for the acupoint ellipse contour parameters from the patient image based on the arithmetic distance; Optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters; Reconstructing multiple sets of solutions of acupoint positions and normal vectors based on the optimized solution based on monocular vision; Determine a unique solution for acupuncture point radius, position and normal vector based on the multiple sets of solutions based on pseudo-stereoscopic vision; The pose of the acupoint is optimized according to the unique solution of the acupoint radius, position and normal vector.

2. The acupoint position estimation method based on pseudo-stereoscopic vision according to claim 1, characterized in that: The method of obtaining a closed solution of the acupoint ellipse contour parameters according to the patient image based on the arithmetic distance comprises the following steps: Define variable ε = [a,b,c,d,e,f] T , for the pixel point s of the elliptical contour of the i-th acupoint in the patient image i =[u i ,v i ] T Satisfies the following vector form: Defining variables The pixel points of all acupoint elliptical contours are defined as: Defining variables The parameter ε of the acupoint ellipse contour is optimized to minimize the bi-norm error of Gε, and the discriminant constraint of the ellipse is introduced; Among them, the constrained optimization problem about ε is: Among them, D A (ε) is the arithmetic distance of the acupoint ellipse contour fitting: ζ is the parameter matrix of the elliptic discriminant constraint, Define λ + and q + Represents G T G has a unique positive eigenvalue relative to ζ and a corresponding unit eigenvector. According to the relevant theorem of ellipse fitting, the optimal solution of the constrained optimization problem is determined to be ε * =q + .

3. The acupoint position estimation method based on pseudo-stereoscopic vision according to claim 1, characterized in that: The method of optimizing the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters comprises the following steps: Iteratively optimizing the parameters of the closed solution of the acupoint ellipse contour to minimize the Sampson distance of the acupoint ellipse contour fitting; The expression for minimizing the Sampson distance of the acupoint ellipse contour fitting is: st T ze>0; Among them, D S (ε) is the Sampson distance of the acupoint ellipse contour fitting; W i g i to s i The partial derivative matrix of The penalty function method is used to convert the ellipse discriminant constraint in the Sampson distance of minimizing the acupoint ellipse contour fitting into the Sampson distance D S The penalty term of (ε) is used to obtain the unconstrained optimization problem as follows: Among them, σ S is the penalty factor of the elliptic discriminant constraint, σ S >0; Solving the unconstrained optimization problem using the Levenberg-Marquardt algorithm includes the following steps: Calculate the partial derivative matrix M of h(ε) with respect to ε: Among them, h i (ε) is the i-th element of h(ε); Definition max represents the number of iterations, ε t represents the acupoint ellipse contour parameter at time t, M t represents h(ε t ) for ε t The partial derivative matrix of , where t=1,2,…,t max ; Furthermore, at time t, the update equation of the acupoint ellipse contour parameters is expressed as: Among them, α t is the damping factor, α t >0.

4. The acupoint position estimation method based on pseudo-stereoscopic vision according to claim 1, characterized in that: The method of reconstructing multiple sets of solutions of acupoint positions and normal vectors based on the optimized solution based on monocular vision comprises the following steps: The quadratic form of the general equation of the ellipse is defined as: Wherein, E is the parameter matrix of the quadratic form of the general elliptic equation; Define p = [x, y, z] T Representing the Cartesian coordinates corresponding to the pixel coordinate s in the camera space, the camera projection model is obtained as: Among them, (f u ,f v ,u0,v0) are the internal parameters of the camera, and the internal parameters constitute the internal parameter matrix K; Combining the quadratic form of the general equation of the ellipse and the projection model of the camera, the quadratic form equation of the oblique cone is obtained as follows: p T Hp=0; Where H is the parameter matrix of the oblique conic quadratic equation, H = K T EK; Define λ1, λ2, and λ3 to represent the three eigenvalues ​​of H, and λ1≥λ2>0>λ3, q1, q2, and q3 to represent the corresponding unit eigenvectors. Furthermore, according to the quadratic equation of the oblique cone in the camera space, the normalized transformation matrix of the quadratic equation of the oblique cone is expressed as P=[o1,o2,o3], where q 3,z represents the third element of q3, sgn(·) represents the sign function, o2=q2, o1=o2×o3; Furthermore, the normal vector and position of the acupoint are expressed as: Among them, r ap is the radius of the acupoint.

5. The acupoint position estimation method based on pseudo-stereoscopic vision according to claim 1, characterized in that: The method of determining a unique solution of acupoint radius, position and normal vector according to the multiple sets of solutions based on pseudo-stereoscopic vision comprises the following steps: definition hec R gc and hec t gc They represent the rotation matrix and translation vector of the coordinate system {gc} relative to the coordinate system {hec} respectively; The normal vectors of acupoints measured by different monocular vision are defined in the coordinate system {hec} as follows: By determining the minimum angle between the normal vectors of the acupoints, the unique solution of the acupoint posture is obtained; and The angle in the coordinate system {hec} is expressed as: in,<x,y> is the inner product between vectors x and y; The position of the acupuncture points is defined in the coordinate system {hec} as: The translation vector obtained according to the normal vector and position of the acupoint is: Defining variables According to the relevant lemma of generalized inverse, the least squares solution of the translation vector is determined to be A multi-constrained nonlinear acupoint position optimization model is established to optimize the parameters of the acupoint ellipse contour and the radius of the acupoint, thereby minimizing the joint error of the acupoint position. The acupoint posture optimization model is: The particle swarm optimization algorithm is used to solve the acupoint posture optimization model to obtain a unique solution for the acupoint radius, position and normal vector.

6. A pseudo-stereoscopic vision-based acupoint position estimation system, characterized in that: The system comprises: An image acquisition unit, used to drive the global camera and the hand-eye camera to dynamically acquire patient images; a closed solution determining unit, configured to obtain a closed solution of the acupoint ellipse contour parameters according to the patient image based on arithmetic distance; an optimized solution determining unit, configured to optimize the closed solution based on approximate geometric iteration to obtain an optimized solution for the acupoint ellipse contour parameters; a multi-solution determination unit, configured to obtain multi-solutions of acupoint positions and normal vectors based on the optimized solution reconstruction based on monocular vision; a unique solution determining unit, configured to determine a unique solution for acupoint radius, position, and normal vector from the multiple sets of solutions based on pseudo-stereoscopic vision; A posture determination unit is used to optimize the posture of the acupoint according to the unique solution of the acupoint radius, position and normal vector.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images