A visual servo method and system for a mirror-holding robot

Abstract

The present application discloses a visual servo method and system for a mirror-holding robot, the method comprising: acquiring an image captured by the arthroscope of the mirror-holding robot; determining the coordinates of the tip center of the instrument in the image; solving the angular velocities of each joint corresponding to the mechanical arm of the mirror-holding robot according to the coordinates of the tip center and the coordinates of the target position; and controlling the mechanical arm to drive the arthroscope to move according to each joint angular velocity. The present application can determine the coordinates of the tip center of the instrument based on the image captured by the arthroscope, and then calculate the joint angular velocity corresponding to the coordinates, so as to control the mechanical arm to drive the arthroscope to track the coordinates of the target position in real time, thereby achieving refined control, and thus meeting the control requirements of complex working environments, and can be widely used in the field of robot control technology.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a visual servo method and system for a mirror-holding robot. Background Art

[0002] Mirror-holding robots are widely used in the medical field, but existing mirror-holding robots still cannot meet the control requirements in complex working environments. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a visual servo method and system for a mirror-holding robot to improve the visual servo performance of the mirror-holding robot and thereby achieve refined control in a complex working environment.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application provides a visual servo method for a mirror-holding robot, the method comprising:

[0005] Acquire images captured by the arthroscope of the scope-holding robot;

[0006] determining coordinates of a tip center of the instrument in the image;

[0007] Calculating the angular velocities of each joint of the robot arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position;

[0008] The robot arm is controlled according to the angular velocity of each joint to drive the arthroscope to move.

[0009] In some embodiments, determining the coordinates of the tip center of the instrument in the image includes:

[0010] Detecting one or more device prediction frames in the image using a pre-trained object detection model, wherein the geometric center of each device prediction frame corresponds to a tip center of the device;

[0011] Determining the coordinates of the geometric center of each of the instrument prediction frames as the coordinates of the tip center of the corresponding instrument;

[0012] The coordinates of the center of the tip of the instrument are:

[0013]

[0014] Among them, c u , c v are the horizontal and vertical coordinates of the tip center of the instrument after adjustment, are the horizontal and vertical coordinates of the tip center of the instrument before adjustment, Specifically:

[0015]

[0016] in, represents the horizontal coordinate of the i-th device prediction frame, represents the ordinate of the i-th device prediction frame, represents the width of the prediction box of the i-th device, represents the height of the i-th device prediction box, represents the predicted probability of the type of the instrument in the i-th instrument prediction frame, K is the total number of the instrument prediction frames, and λ is a positive constant.

[0017] In some embodiments, solving the angular velocities of each joint corresponding to the mechanical arm of the mirror-holding robot according to the coordinates of the tip center and the coordinates of the target position includes:

[0018] Determine the expected moving distance according to the coordinates of the center of the tip of the instrument and the coordinates of the target position; determine the expected speed of the end of the arthroscope according to the expected moving distance; determine the angular speed of each joint corresponding to the robotic arm according to the expected speed;

[0019] Alternatively, the first recurrent neural network is used to solve the angular velocities of each joint corresponding to the robotic arm according to the coordinates of the tip center of the instrument.

[0020] In some embodiments, determining the angular velocities of each joint corresponding to the robotic arm according to the expected speed includes:

[0021] Determine the end velocity of the manipulator according to the desired velocity under the remote motion center constraint; solve the joint angular velocity corresponding to the end velocity based on the Jacobian matrix;

[0022] Alternatively, the joint angular velocity corresponding to the expected velocity is solved based on a second recurrent neural network.

[0023] In some embodiments, solving the joint angular velocity corresponding to the terminal velocity based on the Jacobian matrix includes:

[0024] Solve the joint angular velocity corresponding to the terminal velocity according to the first calculation formula;

[0025] The first calculation formula is:

[0026]

[0027] in, represents the joint angular velocity, J b represents the Jacobian matrix, represents the rotation matrix of the end of the robot arm, Scon represents the terminal velocity.

[0028] In some embodiments, solving the joint angular velocity corresponding to the expected velocity based on the second recurrent neural network includes:

[0029] Inputting the expected speed into the second recurrent neural network to obtain the joint angular speed output by the second recurrent neural network;

[0030] The second recurrent neural network is:

[0031]

[0032] Among them, μ is the Lagrangian operator, is the derivative of the Lagrangian operator, γ is a preset constant scalar and γ>0, t is the solution time, Γ is the activation function of the second recurrent neural network, Ψ is the projection function, S is the expected speed, U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW -1 ), A=JW -1 J T , W and J matrices are both full rank and positive definite matrices, I is the identity matrix, x = US-Vμ, x is the output of the second recurrent neural network, represents the joint angular velocity.

[0033] In some embodiments, the method of using the first recurrent neural network to solve the angular velocities of the joints corresponding to the robotic arm according to the coordinates of the tip center of the instrument includes:

[0034] Iteratively solving the second calculation formula using the first recurrent neural network to obtain the angular velocity of each joint;

[0035] The second calculation formula is:

[0036]

[0037] Among them, J b is the Jacobian matrix of the mirror-holding robot, R is the rotation matrix, is the image Jacobian matrix, is the desired speed of the arthroscope, is the joint angular velocity; the expected velocity is determined according to the coordinates of the tip center.

[0038] To achieve the above purpose, another aspect of the embodiment of the present application provides a mirror-holding robot visual servo system, the system comprising:

[0039] An image acquisition module, used to acquire images taken by the arthroscopic robot;

[0040] a coordinate determination module for determining the coordinates of the tip center of the instrument in the image;

[0041] A velocity solving module, used for solving the angular velocity of each joint corresponding to the mechanical arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position;

[0042] The motion control module is used to control the robot arm to drive the arthroscopy to move according to the angular velocity of each joint.

[0043] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0045] The embodiments of the present application include at least the following beneficial effects:

[0046] The present application obtains the image taken by the arthroscope of the mirror-holding robot; determines the coordinates of the tip center of the instrument in the image; solves the angular velocities of each joint corresponding to the mechanical arm of the mirror-holding robot according to the coordinates of the tip center and the coordinates of the target position; and controls the mechanical arm to drive the arthroscope to move according to each joint angular velocity. The present application can determine the coordinates of the tip center of the instrument based on the image taken by the arthroscope, and then calculate the joint angular velocity corresponding to the coordinates, so as to control the mechanical arm to drive the arthroscope to track the coordinates of the target position in real time, realize refined control, and thus meet the control requirements of complex working environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of a visual servo method for a mirror-holding robot provided in an embodiment of the present application;

[0048] Figure 2 An example diagram of a scene for detecting a prediction box in an image provided by an embodiment of the present application;

[0049] Figure 3 An example diagram of a mirror-holding robot system and RCM constraints provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a process for solving a desired joint angle based on a speed level inverse kinematics solving module of a mirror-holding robot provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a recurrent neural network provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a flow chart of solving the desired joint angle by a velocity level inverse kinematics solving module of a mirror-holding robot based on a recurrent neural network provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a flow chart of solving a desired joint angle by a multi-constraint mirror-holding robot solving module based on a recurrent neural network provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a mirror-holding robot visual servo system provided in an embodiment of the present application;

[0055] Fig. 9 A structural block diagram of a mirror-holding robot visual servo system provided in an embodiment of the present application;

[0056] Fig.10 An application scenario diagram of a mirror-holding robot visual servo system provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0059] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0060] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] A visual servo method for a mirror-holding robot provided in an embodiment of the present application relates to the field of robot control technology. The visual servo method provided in an embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the visual servo method, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] Reference Figure 1 The embodiment of the present application provides a visual servo method for a mirror-holding robot, which may include but is not limited to steps S100 to S130, as follows:

[0065] S100: Acquire an image captured by the arthroscope of the scope-holding robot.

[0066] Specifically, this embodiment can obtain images captured by the arthroscopic camera in real time, or images captured most recently.

[0067] S110: Determine the coordinates of the tip center of the instrument in the image.

[0068] Specifically, the instrument in this embodiment may be a surgical instrument or a tool such as a small screwdriver.

[0069] Furthermore, S110 may include:

[0070] Detecting one or more device prediction frames in the image using a pre-trained object detection model, wherein the geometric center of each device prediction frame corresponds to a tip center of the device;

[0071] Determining the coordinates of the geometric center of each of the instrument prediction frames as the coordinates of the tip center of the corresponding instrument;

[0072] The coordinates of the center of the tip of the instrument are:

[0073]

[0074] Among them, c u , c v are the horizontal and vertical coordinates of the tip center of the instrument after adjustment, are the horizontal and vertical coordinates of the tip center of the instrument before adjustment, Specifically:

[0075]

[0076] in, represents the horizontal coordinate of the i-th device prediction frame, represents the ordinate of the i-th device prediction frame, represents the width of the prediction box of the i-th device, represents the height of the i-th device prediction box, represents the predicted probability of the type of the instrument in the i-th instrument prediction frame, K is the total number of the instrument prediction frames, and λ is a positive constant.

[0077] Specifically, taking the above-mentioned instruments as surgical instruments as an example for explanation, this implementation step can be applied to the multi-joint medical instrument autonomous identification and positioning module in the mirror-holding robot. This module can achieve real-time acquisition of the positions of several surgical instruments based on the one-stage target detection model, and obtain the centers of several surgical instrument tips. It first obtains the training image for the target detection model through the arthroscopy, performs labeling and data enhancement operations such as image cropping, splicing, and flipping, and obtains an enhanced data set. Then, based on the data set, the training set and the validation set are divided into inputs into the target prediction network for training to obtain the corresponding pre-trained model, that is, the optimal network weight. Further, the optimal network weight is used to detect and classify the instrument tip. The central pixel coordinates of several surgical instrument tips are obtained through the basic information of the prediction box.

[0078] Optionally, assume that the image of each frame of the arthroscopy is I r , D represents the one-stage target detection model. Through this target detection model, the center of the prediction box (x p ,y p ), width and height are (w p ,h p ) and the predicted probability of each category p=(p1,p2,...,p class ), from which we can get:

[0079] (x p ,y p ,w p ,h p ,p)=D(I r )

[0080] The types and probabilities of prediction boxes are:

[0081] b c = argmax(p)

[0082] Since the candidate box generated by the prior predicts the approximate position of the target, by predicting the offset of the candidate box, the candidate box can be better used as the prediction value. The initial offset is random, so it is converted into a prediction weight here. This weight varies between [0,1] to make it easier to stabilize the training. Based on this, it is assumed that the coordinates of the upper left corner of the unit (c x ,c y ), the width and height of the preset box are (p w ,p h ).

[0083] At the same time, the Sigmoid function is used to process the offset, that is:

[0084]

[0085] Based on this, the upper left corner coordinate of the actual prediction box (b x , b y ) and width and height (b w , b h ) is as follows:

[0086]

[0087] The specific prediction box update method and parameters are as follows Figure 2 As shown, Figure 2 The dashed box represents the prior box, and the solid box represents the predicted box.

[0088] For joint surgery scenarios, K prediction frames will be generated. For single surgical instrument and multiple surgical instrument scenarios, the output should be the center of several prediction frames to ensure that the entire instrument is in the center of the arthroscopic field of view. The specific formula is as follows:

[0089]

[0090] Thus, the center coordinates of the arthroscopic instrument prediction frame in the current frame in the image coordinate system are obtained.

[0091] Here, since the clear field of view of the arthroscopy is small, there is a considerable amount of blurred field of view. The center of the expected field of view is more inclined to the center of the prediction box with a greater prediction probability. This can avoid the impact of misidentification on subsequent modules under blurred field of view to a certain extent, improve the safety of surgery, and make corresponding adjustments according to the specific type of surgery and the manual operation habits of doctors. Therefore, the above formula can be improved to:

[0092]

[0093] where λ∈[0,1] is a positive constant.

[0094] S120: Calculating the angular velocities of each joint of the robot arm of the mirror-holding robot according to the coordinates of the tip center and the coordinates of the target position.

[0095] Specifically, the target position is the desired location of the center of the instrument tip.

[0096] Further, in this embodiment, S120 may include two implementation modes. The first implementation mode may include S121 to S123, and the second implementation mode may include S124, which are specifically as follows:

[0097] S121: Determine the expected moving distance according to the coordinates of the tip center of the instrument and the coordinates of the target position.

[0098] S122: Determine a desired speed of the distal end of the arthroscope according to the desired moving distance.

[0099] Specifically, S121 and S122 in this embodiment can be applied to the surgical instrument tip intelligent tracking module in the scope holding robot. By executing S121 and S122 through this module, the image coordinate system can be transformed to the coordinate system of the end of the arthroscope. First, the center coordinate s of the desired prediction frame is known. des =[w u / 2 h v / 2] T , i.e. the center of the arthroscopic field of view, w u 、h v are the width and height of the arthroscopic image respectively. Thus, the expected moving distance d in the image coordinate system in this frame is obtained. exp =||s des -s tip ||.

[0100] Therefore, the desired speed corresponding to the arthroscopic camera coordinate system can be obtained by proportional-differential control:

[0101]

[0102] Image Jacobian:

[0103]

[0104]

[0105] Among them, f u 、f v , u0, v0 are the calibrated arthroscopic internal parameters, is a constant.

[0106] Thus, the expected velocity corresponding to the arthroscopic {ka} coordinate system (with its axial direction as the z-axis) is obtained, namely:

[0107]

[0108] Due to the high precision of the joint surgery environment, the expected speed of the arthroscopy is constrained, namely:

[0109]

[0110] Considering that the center point of the instrument tip is already at or close to the center of the arthroscopic field of view, the robotic arm in this embodiment can be fixed to facilitate fine manipulation.

[0111]

[0112] where d res The minimum expected distance for the robot to start moving.

[0113] S123: Determine the angular velocities of each joint of the robotic arm according to the expected speed.

[0114] Optionally, in this embodiment, S123 may include two implementation modes. The first implementation mode may include S1231 to S1232, and the second implementation mode may include S1233. The details are as follows:

[0115] S1231: Determine the end velocity of the robot arm according to the desired velocity under the remote motion center constraint.

[0116] Specifically, S1231 in this embodiment can be applied to the arthroscopic constraint module and the position-level inverse kinematics solution module of the mirror-holding robot.

[0117] Among them, in this embodiment, the arthroscope can be constrained by the arthroscope motion constraint module, so that the arthroscope has only a certain direction of speed at the insertion point, so the speed level RCM (Remote Center of Motion) constraint is introduced here. The linear velocity and angular velocity of the RCM point are The linear velocity and angular velocity of the arthroscope tip are The linear velocity and angular velocity of the arthroscopic connector are

[0118] In this embodiment, the connection member can be used to model the corresponding {con} coordinate system, and the specific formula is as follows:

[0119]

[0120] We can get:

[0121]

[0122] where μ in Denotes the insertion depth ratio of the arthroscopy, d ka Indicates the length of the arthroscope.

[0123] Since the x-axis and y-axis of the RCM position are constrained here, that is, the velocity perpendicular to the axis is 0, we can also get:

[0124]

[0125] Because the arthroscope and the connector can be regarded as the same rigid body, the angular velocity of the end of the arthroscope and the connector is equal, that is:

[0126]

[0127] Based on this, it can be and as well as and The linear velocity and angular velocity of the connecting part can be expressed by the linear velocity and angular velocity of the end of the arthroscope:

[0128]

[0129] In this way, the speed level relationship between the end of the arthroscope and the connecting part can be established, where

[0130]

[0131]

[0132] Since during arthroscopic planning, the insertion depth μ in There will be certain changes, resulting in certain errors in the long-term planning of the robot arm. The reason is that when the insertion depth changes, the RCM constraint point will also be displaced. Therefore, it needs to be constrained. For arthroscopic surgery, there should be a certain threshold for the change in the axial planning of the robot arm in the arthroscopy, so that the insertion depth of the arthroscopy works within the safe area. Therefore, after adding constraints, the expected insertion depth can be expressed as:

[0133]

[0134] Where K p,μ , K d,μ All are normal numbers.

[0135] Then, the joint kinematics of the robot arm of the mirror holding robot are solved by the position-level inverse kinematics solving module of the mirror holding robot. Specifically, the corresponding DH (Denavit-Hartenberg) parameters are obtained according to the initial coordinate system of the robot arm. DH parameters are a method for describing the geometric relationship between robot joints and links. They represent the length, torsion angle, link offset and joint angle of the link between joint i-1 and joint i. Here, they can be equivalent to a translation and rotation along the x-axis and z-axis respectively, thus obtaining the translation and rotation along the x-axis:

[0136]

[0137] Translation and rotation along the z-axis:

[0138]

[0139] in, is the rotation angle of the corresponding joint at a certain moment.

[0140] Based on this, the homogeneous transformation matrix corresponding to each connecting rod can be obtained as follows:

[0141]

[0142] The overall modeling is based on the end coordinate system of the robot arm to obtain the corresponding relationship between the coordinate systems based on the end coordinate system, that is:

[0143]

[0144] in They respectively represent the direction vectors of the i coordinate system along the x, y, and z axes based on the end coordinate system.

[0145] Since the robot arm used is 6-DOF and all of them are rotational joints, the Jacobian matrix J can be obtained e =[J1J2...J6] The column vector is:

[0146]

[0147] Based on this, we can get:

[0148]

[0149] Then, the end rotation S of the robot arm can be realized according to the above formula. e =[v e ω e ] T To joint angular velocity conversion.

[0150] The example diagram of mirror robot system construction and RCM constraints can be found in Figure 3 .

[0151] S1232: Solve the joint angular velocity corresponding to the terminal velocity based on the Jacobian matrix.

[0152] Further, S1232 may include:

[0153] Solve the joint angular velocity corresponding to the terminal velocity according to the first calculation formula;

[0154] The first calculation formula is:

[0155]

[0156] in, represents the joint angular velocity, J b represents the Jacobian matrix, represents the rotation matrix of the end of the robot arm, S con represents the terminal velocity.

[0157] Specifically, in this embodiment, the transformation from the end of the connector to the joint angle of the robotic arm can be realized through the speed level inverse kinematics solution module of the mirror holding robot. The rotation amount of the connector obtained by the arthroscopic constraint module is used to obtain the angle that the corresponding joint angle of the robotic arm should rotate, which is used to control the movement of the mirror holding robot. The solution formula is as follows:

[0158]

[0159] If the multi-joint medical device autonomous identification and positioning module does not detect the corresponding prediction frame, that is, the device is out of the field of view, the robot arm will stop immediately. The flowchart of solving the expected joint angle based on the mirror-holding robot speed level inverse kinematics solution module can be referred to Figure 4 .

[0160] S1233: Solve the joint angular velocity corresponding to the expected velocity based on the second recurrent neural network.

[0161] Further, S1233 may include:

[0162] Inputting the expected speed into the second recurrent neural network to obtain the joint angular speed output by the second recurrent neural network;

[0163] The second recurrent neural network is:

[0164]

[0165] Among them, μ is the Lagrangian operator, is the derivative of the Lagrangian operator, γ is a preset constant scalar and γ>0, t is the solution time, Γ is the activation function of the second recurrent neural network, Ψ is the projection function, S is the expected speed, U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW -1 ), A=JW -1 J T , W and J matrices are both full rank and positive definite matrices, I is the identity matrix, x = US-Vμ, x is the output of the second recurrent neural network, represents the joint angular velocity.

[0166] Specifically, this embodiment can complete the solution of the arthroscopic end speed to the joint angular velocity of the mirror holding robot based on the recurrent neural network inverse kinematics solution module. The module input is the expected speed planned by the vision-based arthroscopic The output is the angular velocity of the mirror robot joints And sent by the host computer to the mirror holding robot, where the rotation matrix is:

[0167]

[0168] where d con It is the axial length of the connecting part at the end of the robot arm.

[0169] It should be clear that the speed-level arthroscopy planning can be achieved based on the intelligent tracking module at the tip of the surgical instrument to obtain the desired speed. Finally, the RCM constraint points and other information are converted into So as to directly calculate the end speed of the robot arm

[0170] This module describes the inverse kinematics from the end velocity of the mirror-holding robot to the joint angular velocity as:

[0171]

[0172]

[0173] θ - ≤θ≤θ +

[0174]

[0175] The above inverse kinematics based on quadratic programming is established at the velocity level, where θ represents the angles of each joint of the mirror-holding robot. Represents the angular velocity of the mirror-holding robot joint. v end =[v x ,v y ,v z ,w x ,w y ,w z ] T represents the control speed of the end of the mirror holding robot, J acob Represents the Jacobian matrix from the joint angle space to the end of the mirror-holding robot.

[0176] After unifying the constraints on joint angles and angular velocities, we can obtain:

[0177]

[0178] in:

[0179] Therefore, the speed calculation model based on quadratic programming of speed level can be described as:

[0180] min x T Wx / 2

[0181] stJx=S

[0182] x - ≤x≤x +

[0183] in The subscripts of each matrix are cancelled to facilitate further calculation. dt is the communication time interval between the host computer and the robot arm. W is defined as the weight coefficient of each joint, and both W and J matrices are full rank and positive definite matrices.

[0184] According to the Lagrange multiplier method, the Lagrange equation can be expressed as:

[0185] min x T Wx+λ(Jx-S)+μ(Ψ(x+μ)-x)

[0186] According to the KKT condition, the QP problem is equivalent to the following equation:

[0187]

[0188] Where λ and μ are Lagrangian operators. Ψ(·) is the projection function, which is specifically expressed as:

[0189]

[0190] Definition A = JW -1 J T , then we can get:

[0191] λ=-A-1 (S+JW -1 μ)

[0192] Substitute λ into the original formula:

[0193] x=US-Vμ

[0194] where U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW -1 ), I is the unit matrix. To ensure Ψ(x+μ)-x=0 and solve μ, the quadratic programming solution structure based on recurrent neural network can be proposed:

[0195]

[0196] Where γ is the constant scalar of the design and γ>0, and t is the solution time. When t→∞, we can get (γ+t γ )→∞, together they constitute the neural network convergence coefficient. Γ(·) is the activation function of the neural network. The selection of the activation function can effectively change the convergence speed of the recurrent neural network. Therefore, based on the typical activation functions in the past, the power function is conducive to the convergence of the recurrent neural network in a limited time, and the linear function is conducive to accelerating the convergence speed. This patent selects the following activation functions:

[0197] Γ(x)=x+|x| 2 tanh(x)+|x| 0.5 tanh(x)

[0198] Tanh(x) is the hyperbolic tangent function, and its specific expression is:

[0199]

[0200] To sum up, the constructed recurrent neural network model is as follows:

[0201]

[0202] where U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW -1 ). And the output vector can be obtained by the following formula:

[0203] x=US-Vμ

[0204] Transformed into:

[0205]

[0206] According to multiple iterations of the recurrent neural network, we can finally obtain That is, the angular velocity of each joint of the robot arm. And through the upper computer interface of the mirror holding robot, the angular velocity of each joint is sent to the mirror holding robot, so that the arthroscopic lens can track the medical device in real time. Among them, the structural diagram of the second recurrent neural network can refer to Figure 5 The schematic diagram of the process of solving the desired joint angle of the mirror robot velocity level inverse kinematics solution module based on recurrent neural network can be referred to Figure 6 .

[0207] S124: Utilizing the first recurrent neural network to solve the angular velocities of the joints corresponding to the robotic arm according to the coordinates of the tip center of the instrument.

[0208] Further, S124 may include:

[0209] Iteratively solving the second calculation formula using the first recurrent neural network to obtain the angular velocity of each joint;

[0210] The second calculation formula is:

[0211]

[0212] Among them, J b is the Jacobian matrix of the mirror-holding robot, R is the rotation matrix, is the image Jacobian matrix, is the desired speed of the arthroscope, is the joint angular velocity; the expected velocity is determined according to the coordinates of the tip center.

[0213] Specifically, in this embodiment, the multi-constrained mirror-holding robot solving module based on the recurrent neural network can process the information obtained by the multi-joint medical device autonomous identification and positioning module, while considering the RCM constraints to realize the solution of the joint angular velocity of the mirror-holding robot.

[0214] The input of this module is the information obtained by the multi-joint medical device autonomous identification and positioning module, and the output is the angular velocity of the joints of the mirror-holding robot.

[0215] Specifically, the input is the information obtained by the multi-joint medical device autonomous identification and positioning module, that is, the center coordinates of the expected prediction box s des =[w u / 2 h v / 2] T , which is the center of the arthroscopic field of view. Thus, the expected moving distance d in the image coordinate system is obtained in this frame. exp =||s des -stip ||.

[0216] Therefore, proportional-differential control can be used to obtain the corresponding desired speed in the arthroscopic camera.

[0217]

[0218] According to the solution results of the position-level inverse kinematics solution module of the mirror-holding robot, the joint angular velocity is realized To the end of the robot arm rotation S e =[v e ω e ] T The conversion, that is:

[0219]

[0220] The image Jacobian is set as u = c u -u0,v=c v -v0, you can get:

[0221]

[0222] According to forward kinematics, we can obtain:

[0223]

[0224] Among them J b is the Jacobian matrix of the mirror-holding robot, R is the corresponding rotation matrix, is the image Jacobian matrix, The corresponding expected speed in the arthroscopic camera, is the joint angular velocity of the mirror-holding robot.

[0225] Secondly, according to the RCM constraints in the arthroscopic constraint module, we can get:

[0226]

[0227]

[0228] Derivative of both sides of the formula with respect to time:

[0229]

[0230] According to Section 3.4, the kinematics of the mirror-holding robot We can get:

[0231]

[0232] Among them J ka , J con ∈R3×6 Velocity portion of the Jacobian matrix for the arthroscope and the connector. Direction vectors belonging to the arthroscope, connector and RCM point in the base coordinate system. Represents the RCM point velocity. Due to the existence of RCM point constraints, is a zero vector. Combining the above equations, we get:

[0233]

[0234] make You can get:

[0235] Jx=v

[0236] The quadratic programming problem constructed according to the above embodiment is:

[0237] min x T Wx / 2

[0238] stjx=v

[0239] x - ≤x≤x +

[0240] Finally, according to multiple iterations of the recurrent neural network, we can finally obtain That is, the angular velocity of each joint of the robotic arm.

[0241] The flowchart of the multi-constraint mirror robot solution module based on recurrent neural network to solve the desired joint angle can be referred to Figure 7 .

[0242] S130: Controlling the robotic arm to drive the arthroscopy to move according to the angular velocities of each joint.

[0243] Specifically, this embodiment can be applied to a mirror-holding robot motion module in a mirror-holding robot, through which the input desired angles of each joint of the robotic arm can be transmitted to the real robotic arm, and the robotic arm can be controlled to make corresponding movements.

[0244] Reference Figure 8 The embodiment of the present application further provides a mirror-holding robot visual servo system, which can implement the above-mentioned visual servo method, and the system includes:

[0245] An image acquisition module, used to acquire images taken by the arthroscopic robot;

[0246] a coordinate determination module for determining the coordinates of the tip center of the instrument in the image;

[0247] A velocity solving module, used for solving the angular velocity of each joint corresponding to the mechanical arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position;

[0248] The motion control module is used to control the robot arm to drive the arthroscopy to move according to the angular velocity of each joint.

[0249] Optionally, the coordinate determination module includes:

[0250] A prediction frame detection unit, used to detect one or more instrument prediction frames in the image using a pre-trained target detection model, wherein the geometric center of each instrument prediction frame corresponds to a tip center of the instrument;

[0251] A coordinate determination unit, used to determine the coordinates of the geometric center of each of the instrument prediction frames as the coordinates of the tip center of the corresponding instrument;

[0252] The coordinates of the center of the tip of the instrument are:

[0253]

[0254] Among them, c u , c v are the horizontal and vertical coordinates of the tip center of the instrument after adjustment, are the horizontal and vertical coordinates of the tip center of the instrument before adjustment, Specifically:

[0255]

[0256] in, represents the horizontal coordinate of the i-th device prediction frame, represents the ordinate of the i-th device prediction frame, represents the width of the prediction box of the i-th device, represents the height of the i-th device prediction box, represents the predicted probability of the type of the instrument in the i-th instrument prediction frame, K is the total number of the instrument prediction frames, and λ is a positive constant.

[0257] Optionally, the speed solving module includes:

[0258] A first speed solving unit is used to determine an expected moving distance according to the coordinates of the tip center of the instrument and the coordinates of the target position; determine an expected speed of the end of the arthroscope according to the expected moving distance; and determine the angular speed of each joint corresponding to the robotic arm according to the expected speed;

[0259] The second speed solving unit is used to use the first recurrent neural network to solve the angular velocities of each joint corresponding to the robotic arm according to the coordinates of the tip center of the instrument.

[0260] Optionally, the first speed solving unit includes:

[0261] A Jacobian matrix solving unit, used for determining the end velocity of the manipulator according to the desired velocity under the remote motion center constraint; solving the joint angular velocity corresponding to the end velocity based on the Jacobian matrix;

[0262] The recurrent neural network solving unit is used to solve the joint angular velocity corresponding to the expected velocity based on the second recurrent neural network.

[0263] Optionally, the Jacobian matrix solving unit includes:

[0264] A Jacobian matrix solving subunit, used for solving the joint angular velocity corresponding to the terminal velocity according to the first calculation formula;

[0265] The first calculation formula is:

[0266]

[0267] in, represents the joint angular velocity, J b represents the Jacobian matrix, represents the rotation matrix of the end of the robot arm, S con represents the terminal velocity.

[0268] Optionally, the recurrent neural network solving unit includes:

[0269] A recurrent neural network solving subunit, used for inputting the expected speed into the second recurrent neural network to obtain the joint angular speed output by the second recurrent neural network;

[0270] The second recurrent neural network is:

[0271]

[0272] Among them, μ is the Lagrangian operator, is the derivative of the Lagrangian operator, γ is a preset constant scalar and γ>0, t is the solution time, Γ is the activation function of the second recurrent neural network, Ψ is the projection function, S is the expected speed, U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW-1 ), A=JW -1 J T , W and J matrices are both full rank and positive definite matrices, I is the identity matrix, x = US-Vμ, x is the output of the second recurrent neural network, represents the joint angular velocity.

[0273] Optionally, the motion control module includes:

[0274] A motion control unit, used for iteratively solving a second calculation formula using the first recurrent neural network to obtain the angular velocity of each joint;

[0275] The second calculation formula is:

[0276]

[0277] Among them, J b is the Jacobian matrix of the mirror-holding robot, R is the rotation matrix, is the image Jacobian matrix, is the desired speed of the arthroscope, is the joint angular velocity; the expected velocity is determined according to the coordinates of the tip center.

[0278] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0279] In view of the fact that the existing mirror-holding robot cannot consider the constraints caused by the working environment and its own structure of the redundant robot, and the low intelligence in the complex working environment, the solution of the embodiment of the present application will be introduced and explained in detail in combination with specific application examples:

[0280] Reference Fig. 9 , this embodiment provides a structural block diagram of a visual servo system for a mirror-holding robot. Fig.10 , this embodiment provides an application scenario diagram of a mirror-holding robot visual servo system.

[0281] Specifically, the mirror-holding robot of this embodiment may include a mirror-holding robot body (i.e., a robotic arm), an arthroscopic connector on the end of the robot body, an arthroscope, and a visual servo system. The visual servo system may include a multi-joint medical instrument autonomous identification and positioning module, which is used to realize real-time acquisition of the positions of several surgical instruments and obtain the centers of several surgical instrument tips; a mirror-holding robot kinematics solution module, which is used to perform inverse kinematics solution of the joints of the mirror-holding robotic arm body; a surgical instrument tip intelligent tracking module, which is used to realize the desired speed conversion from the image coordinate system to the coordinate system of the end of the arthroscope; an arthroscopic constraint module, which is used to realize the speed conversion from the connector coordinate system to the coordinate system of the end of the arthroscope after adding constraints; the inverse kinematics module of the mirror-holding robot may include an inverse kinematics solution module based on a Jacobian matrix and an inverse kinematics solution module based on a recurrent neural network. The above two modules can be used to realize the speed conversion between the end of the arthroscope and the connector under the constraint of the remote center of motion RCM (Remote Center of Motion).

[0282] The multi-constraint mirror-holding robot solving module based on recurrent neural network is used to realize the information processing obtained by the multi-joint medical device autonomous identification and positioning module, and at the same time consider the RCM constraints to realize the solution of the joint angular velocity of the mirror-holding robot. This module can replace the surgical instrument tip intelligent tracking module, arthroscopic constraint module and mirror-holding robot inverse kinematics module in this embodiment; the mirror-holding robot motion module is used to control the motion of the robotic arm body.

[0283] The mirror-holding robot visual servo system of this embodiment can provide two structures and three different methods for planning the robot, and can realize precise control of the mirror-holding robot.

[0284] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned visual servoing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0285] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0286] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0287] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0288] The memory 1102 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 is used to call and execute the visual servoing method of the embodiment of this application;

[0289] Input / output interface 1103, used to implement information input and output;

[0290] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0291] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0292] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0293] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned visual servoing method is implemented.

[0294] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0295] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0296] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0297] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0298] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0299] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0300] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0301] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0302] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0303] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0304] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0305] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0306] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A visual servo method for a mirror-holding robot, characterized in that: The method comprises: Acquire images captured by the arthroscope of the scope-holding robot; determining coordinates of a tip center of the instrument in the image; Calculating the angular velocities of each joint of the robot arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position; Controlling the robotic arm to drive the arthroscopy to move according to the angular velocity of each joint; Determining the coordinates of the tip center of the instrument in the image includes: Detecting one or more device prediction frames in the image using a pre-trained object detection model, wherein the geometric center of each device prediction frame corresponds to a tip center of the device; Determining the coordinates of the geometric center of each of the instrument prediction frames as the coordinates of the tip center of the corresponding instrument; The coordinates of the center of the tip of the instrument are: Among them, c u , c v are the horizontal and vertical coordinates of the tip center of the instrument after adjustment, are the horizontal and vertical coordinates of the tip center of the instrument before adjustment, Specifically: in, represents the horizontal coordinate of the i-th device prediction frame, represents the ordinate of the i-th device prediction frame, represents the width of the prediction box of the i-th device, represents the height of the i-th device prediction box, represents the predicted probability of the type of the instrument in the i-th instrument prediction frame, K is the total number of the instrument prediction frames, and λ is a positive constant.

2. A mirror-holding robot visual servo method according to claim 1, characterized in that: The step of solving the angular velocities of the joints of the robot arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position includes: Determine the expected moving distance according to the coordinates of the center of the tip of the instrument and the coordinates of the target position; determine the expected speed of the end of the arthroscope according to the expected moving distance; determine the angular speed of each joint corresponding to the robotic arm according to the expected speed; Alternatively, the first recurrent neural network is used to solve the angular velocities of each joint corresponding to the robotic arm according to the coordinates of the tip center of the instrument.

3. A mirror-holding robot visual servo method according to claim 2, characterized in that: Determining the angular velocities of each joint corresponding to the robotic arm according to the expected speed includes: Determine the end velocity of the manipulator according to the desired velocity under the remote motion center constraint; solve the joint angular velocity corresponding to the end velocity based on the Jacobian matrix; Alternatively, the joint angular velocity corresponding to the expected velocity is solved based on a second recurrent neural network.

4. A mirror-holding robot visual servo method according to claim 3, characterized in that: The step of solving the joint angular velocity corresponding to the terminal velocity based on the Jacobian matrix includes: Solve the joint angular velocity corresponding to the terminal velocity according to the first calculation formula; The first calculation formula is: in, represents the joint angular velocity, J b represents the Jacobian matrix, represents the rotation matrix of the end of the robot arm, S con represents the terminal speed.

5. The visual servo method of a mirror-holding robot according to claim 3, characterized in that: The step of solving the joint angular velocity corresponding to the expected velocity based on the second recurrent neural network includes: Inputting the expected speed into the second recurrent neural network to obtain the joint angular speed output by the second recurrent neural network; The second recurrent neural network is: Among them, μ is the Lagrangian operator, is the derivative of the Lagrangian operator, γ is a preset constant scalar and γ>0, t is the solution time, Γ is the activation function of the second recurrent neural network, Ψ is the projection function, S is the expected speed, U=W -1 J T A -1 , V=W -1 (IJ T A -1 JW -1 ), A=JW -1 J T , W and J matrices are both full rank and positive definite matrices, I is the identity matrix, x = US-Vμ, x is the output of the second recurrent neural network, represents the joint angular velocity.

6. The visual servo method of a mirror-holding robot according to claim 2, characterized in that: The method of using the first recurrent neural network to solve the angular velocities of the joints corresponding to the robotic arm according to the coordinates of the tip center of the instrument includes: Iteratively solving the second calculation formula using the first recurrent neural network to obtain the angular velocity of each joint; The second calculation formula is: Among them, J b is the Jacobian matrix of the mirror-holding robot, R is the rotation matrix, is the image Jacobian matrix, is the desired speed of the arthroscope, is the joint angular velocity; the expected velocity is determined according to the coordinates of the tip center.

7. A mirror-holding robot visual servo system, characterized in that: The system comprises: An image acquisition module, used to acquire images taken by the arthroscopic robot; a coordinate determination module for determining the coordinates of the tip center of the instrument in the image; A velocity solving module, used for solving the angular velocity of each joint corresponding to the mechanical arm of the mirror holding robot according to the coordinates of the center of the tip and the coordinates of the target position; A motion control module, used for controlling the mechanical arm to drive the arthroscopy to move according to the angular velocity of each joint; Determining the coordinates of the tip center of the instrument in the image includes: Detecting one or more device prediction frames in the image using a pre-trained object detection model, wherein the geometric center of each device prediction frame corresponds to a tip center of the device; Determining the coordinates of the geometric center of each of the instrument prediction frames as the coordinates of the tip center of the corresponding instrument; The coordinates of the center of the tip of the instrument are: Among them, c u , c v are the horizontal and vertical coordinates of the tip center of the instrument after adjustment, are the horizontal and vertical coordinates of the tip center of the instrument before adjustment, Specifically: in, represents the horizontal coordinate of the i-th device prediction frame, represents the ordinate of the i-th device prediction frame, represents the width of the prediction box of the i-th device, represents the height of the i-th device prediction box, represents the predicted probability of the type of the instrument in the i-th instrument prediction frame, K is the total number of the instrument prediction frames, and λ is a positive constant.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images