Multi-dimensional interaction method, system and storage medium of laparoscopic surgical robot

Abstract

The present invention relates to the field of robot control technology, and specifically to a multi-dimensional interaction method, system, and storage medium for a laparoscopic surgical robot, comprising the following steps: in a standard laparoscopic simulated surgery process, marking the robot's standard motion information in the visual information recorded by a camera at each process node, and marking the instrument's standard force information in the force information recorded by a force sensor at each process node; training a two-branch neural network based on a data set to construct a multi-dimensional interaction model for interactively controlling the laparoscopic surgical robot through visual force information; and utilizing the real-time robot motion information and instrument real-time force information predicted by the multi-dimensional interaction model to perform real-time interactive control of the laparoscopic surgical robot. The present invention realizes automated model analysis of the robot's control parameters, improves the objectivity and efficiency of the control analysis, and ultimately achieves standardized control during the surgical operation to ensure the effectiveness of the surgical operation.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robot control, and in particular to a multi-dimensional interaction method, system and storage medium for a laparoscopic surgical robot. Background Art

[0002] Robotic laparoscopic surgery is a type of laparoscopic surgery performed with the assistance of robotic technology. The system primarily consists of a control console and a manipulator arm. The surgeon, seated at the console, uses a touchscreen monitor to adjust the robot's joint angles, position coordinates, instrument movement amplitude, opening angle, and whether the instrument is locked after closing.

[0003] In the existing technology, the operation and control of laparoscopic surgical robots are all obtained by the operator through judgment and analysis based on the visual information fed back by the monitor. Therefore, the information that the operator can grasp for analyzing the operation and control is only the visual information fed back during the operation. The limited amount of information in a single dimension will inevitably lead to limited analysis performance. The operation and control may not be able to ensure the optimal state, which will affect the surgical operation process and ultimately affect the surgical effect. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-dimensional interaction method for a laparoscopic surgical robot to solve the technical problem in the prior art that the limited amount of information in a single dimension will limit the control analysis effect and cannot achieve optimal control.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A multi-dimensional interaction method for a laparoscopic surgical robot comprises the following steps:

[0007] In the standard laparoscopic simulation surgery process, the robot's standard motion information is marked in the visual information recorded by the camera at each process node, and the instrument's standard force information is marked in the force information recorded by the force sensor at each process node;

[0008] The visual information, force information, robot standard motion information, and instrument standard force information at each process node are combined into a data set. A two-branch neural network is trained based on the data set to construct a multi-dimensional interaction model for controlling the laparoscopic surgical robot through visual and force information interaction.

[0009] The real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interaction model are used to perform real-time interactive control of the laparoscopic surgical robot.

[0010] As a preferred solution of the present invention, the motion information includes the robot joint angles and position coordinates, and the force information includes: the depth, amplitude, speed and strength of the tissue pulling and cutting operations of the surgical instruments loaded on the robot.

[0011] As a preferred solution of the present invention, the method for constructing the multidimensional interaction model includes:

[0012] Using visual information as an input item of the first branch neural network and using instrument standard force information as an output item of the first branch neural network;

[0013] The force information is used as the input item of the second branch neural network, and the standard motion information of the robot is used as the output item of the second branch neural network;

[0014] Using the prediction loss and the reconstruction loss, the first branch neural network and the second branch neural network are trained to obtain the multidimensional interaction model;

[0015] The multi-dimensional interaction model is:

[0016] ;

[0017] ;

[0018] Where, is the instrument standard force information output by the first branch neural network, is the robot standard motion information output by the second branch neural network, For visual information, is force information, CNN1 is the first branch neural network, and CNN2 is the second branch neural network.

[0019] As a preferred solution of the present invention, the predicted loss is:

[0020] ;

[0021] Where, To predict losses, is the instrument standard force information at the t-th process node output by the first branch neural network, is the true value of the instrument standard force information at the t-th process node in the data set, is the standard motion information of the robot at the tth process node output by the second branch neural network, is the true value of the robot standard motion information at the t-th process node in the dataset, N is the total number of process nodes in the laparoscopic surgery standard process, and Both are L2 norm forms.

[0022] As a preferred solution of the present invention, the reconstruction loss is:

[0023] ;

[0024] Where, is the reconstruction loss, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, for the reason The converted force information, for the reason The converted visual information, exchange is the conversion network, N is the total number of process nodes in the standard laparoscopic surgery process, and All are L2 norm forms;

[0025] Among them, the conversion network exchange of force information and visual information is:

[0026] ;

[0027] ;

[0028] Where, is the force information at the tth process node output by the force information conversion network, is the instrument standard force information at the t-th process node output by the first branch neural network, is the visual information at the t-th process node output by the visual information conversion network, It is the standard motion information of the robot at the t-th process node output by the second branch neural network. Both CNN3 and CNN4 are convolutional neural networks.

[0029] As a preferred embodiment of the present invention, a method for real-time interactive control of a laparoscopic surgical robot using real-time robot motion information and instrument force information predicted by a multi-dimensional interactive model includes:

[0030] The real-time visual information fed back by the camera and the real-time force information fed back by the force sensor are input into the multi-dimensional interaction model, and the first branch neural network in the multi-dimensional interaction model outputs the real-time motion information of the robot, and the second branch neural network in the multi-dimensional interaction model outputs the real-time force information of the instrument;

[0031] The laparoscopic surgical robot is controlled to perform surgical operations based on the robot's real-time motion information and the instrument's real-time force information.

[0032] As a preferred solution of the present invention, the loss function of the exchange network for converting force information and visual information is:

[0033] ;

[0034] Where, is the prediction loss of the conversion network, is the force information at the tth process node output by the force information conversion network, is the visual information at the t-th process node output by the visual information conversion network, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, N is the total number of process nodes in the standard laparoscopic surgery process, and Both are L2 norm forms.

[0035] As a preferred solution of the present invention, the visual information at each process node is normalized, and the force information at each process node is normalized.

[0036] As a preferred embodiment of the present invention, the present invention provides a multi-dimensional interactive system for a laparoscopic surgical robot, which is applied to a multi-dimensional interactive method for a laparoscopic surgical robot. The system includes:

[0037] A data acquisition unit is used to mark the robot's standard motion information in the visual information recorded by the camera at each process node in the standard laparoscopic simulation surgery process, and to mark the instrument's standard force information in the force information recorded by the force sensor at each process node;

[0038] A model building unit is used to combine visual information, force information, robot standard motion information, and instrument standard force information at each process node into a data set, and train a two-branch neural network based on the data set to build a multi-dimensional interaction model for interactively controlling the laparoscopic surgical robot through visual and force information;

[0039] The interactive control unit is used to perform real-time interactive control of the laparoscopic surgical robot using the real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interactive model.

[0040] As a preferred embodiment of the present invention, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, a multi-dimensional interaction method such as a laparoscopic surgical robot is implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention uses visual information and force information to construct a multi-dimensional interactive model for controlling the laparoscopic surgical robot, thereby analyzing control parameters based on multi-dimensional information interaction and improving control accuracy. Moreover, the multi-dimensional interactive model can realize automatic analysis of the robot's control parameters, improve the objectivity and efficiency of the control analysis, and ultimately achieve standardized control during the surgical operation to ensure the effectiveness of the surgical operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0044] Figure 1 A flowchart of a multi-dimensional interaction method for a laparoscopic surgical robot provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of the multi-dimensional interactive system of the laparoscopic surgical robot provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a multi-dimensional interaction model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, the present invention provides a multi-dimensional interaction method for a laparoscopic surgical robot, comprising the following steps:

[0049] In the standard laparoscopic simulation surgery process, the robot's standard motion information is marked in the visual information recorded by the camera at each process node, and the instrument's standard force information is marked in the force information recorded by the force sensor at each process node;

[0050] The visual information, force information, robot standard motion information, and instrument standard force information at each process node are combined into a data set. A two-branch neural network is trained based on the data set to construct a multi-dimensional interaction model for controlling the laparoscopic surgical robot through visual and force information interaction.

[0051] The real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interaction model are used to perform real-time interactive control of the laparoscopic surgical robot.

[0052] When analyzing the control parameters of the laparoscopic surgical robot, the present invention utilizes information from two dimensions, namely vision and force perception, to achieve multi-dimensional control analysis and ensure the interactive unity of multi-dimensional information, thereby obtaining more accurate control parameters and thus more accurately controlling the robot surgically.

[0053] In the standard process of laparoscopic simulation surgery, the present invention collects visual information and force information to form a data set, which can ensure the interactive uniformity of visual information and force information, that is, the robot motion information and instrument force information reflected by the visual information and force information at each process node are standardized, and the best surgical effect can be obtained. At the same time, the robot motion information and instrument force information reflected by the visual information and force information at the same process node are one-to-one matched, ensuring the smoothness and accuracy of the cooperation between the surgical operation robot and the surgical instrument.

[0054] After constructing a standardized surgical control data set, the present invention uses it as a data sample for training a multidimensional interaction model, so that the multidimensional interaction model can analyze multidimensional information (that is, visual information and force information) to perform real-time and precise control of the laparoscopic surgical robot.

[0055] The multi-dimensional interactive model constructed by the present invention includes two neural network branch structures, which are used to realize interactive supervision analysis of visual information and force information. Among them, the first neural network branch is used to establish a mapping relationship between visual information and instrument standard force information, so as to output instrument standard force information according to visual information, so as to achieve the purpose of matching the visual information with the most suitable instrument force information. The second neural network branch is used to establish a mapping relationship between force information and robot standard motion information, so as to output robot standard motion information according to force information, so as to achieve the purpose of matching the force information with the most suitable robot motion information. The two neural network branches each establish an interactive mapping relationship so that the prediction results output by each have corresponding matching, thereby ensuring the unity of robot motion information and instrument force information at the same process node or at the same time.

[0056] When training two neural network branch structures, the present invention adopts a self-supervision method to ensure that the robot motion information and the instrument force information are unified and standardized. First, the two neural network branches are trained with the difference between the instrument standard force information output by the first branch neural network and the true value of the instrument standard force information in the data set, and the difference between the robot standard motion information output by the second branch neural network and the true value of the robot standard motion information in the data set (i.e., prediction loss). In this way, while making the prediction results output by each branch have corresponding matching, the prediction results output by each branch are closest to the true results, and the higher the prediction accuracy performance, the true value of the instrument standard force information in the data set and the true value between the robot standard motion information in the data set are used as label values for training the two neural network branches, thereby realizing mutual self-supervision training of the two neural network branch structures at the instrument force information and robot motion information levels.

[0057] Secondly, the two neural network branches are trained based on the difference between the force information obtained by inverting the instrument standard force information output by the first branch neural network and the original force information, and the difference between the visual information obtained by inverting the robot standard motion information output by the second branch neural network and the original visual information (i.e., reconstruction loss). This can further ensure that the instrument standard force information predicted and output by the first branch neural network based on the original visual information is consistent with the original force information, and that there is temporal consistency between the original visual information and the original force information.

[0058] Therefore, it is ensured that the first branch neural network can output the machine force information that best matches the visual information. Similarly, it is ensured that the robot motion information predicted and output by the second branch neural network based on the original force information is consistent with the original force information. There is temporal uniformity between the original visual information and the original force information. Therefore, it is ensured that the second branch neural network can output the robot motion information that best matches the force information. The use of reconstruction loss ensures that the machine force information and robot motion information output by the two neural network branch structures inherit the temporal uniformity characteristics between the original visual information and the original force information. The original visual information and the original force information are still used in the reconstruction loss as label values for training the two neural network branches, thereby realizing mutual self-supervised training of the two neural network branch structures at the visual information and force information levels.

[0059] In order to realize the force information obtained by inverse mapping the standard force information of the instrument and the visual information obtained by inverse mapping the standard motion information of the robot, the present invention constructs a conversion network, establishes a mapping relationship between the standard force information of the instrument and the force information, and a mapping relationship between the standard motion information of the robot and the visual information, thereby respectively realizing the force information predicted according to the standard force information of the instrument and the visual information predicted according to the standard motion information of the robot.

[0060] The motion information includes the robot's joint angles and position coordinates, and the force information includes the depth, amplitude, speed, and strength of the robot's surgical instruments pulling and cutting tissues.

[0061] In the standard process of laparoscopic simulation surgery, the present invention collects visual information and force information to form a data set, which can ensure the interactive uniformity of visual information and force information. That is, the robot motion information and instrument force information reflected by the visual information and force information at each process node are standardized, which can achieve the best surgical effect. At the same time, the robot motion information and instrument force information reflected by the visual information and force information at the same process node are one-to-one matched, ensuring the smoothness and accuracy of the cooperation between the surgical operation robot and the surgical instrument. The details are as follows:

[0062] like Figure 3 As shown in Figure 2, the method for constructing a multidimensional interaction model includes:

[0063] Using visual information as an input item of the first branch neural network and using instrument standard force information as an output item of the first branch neural network;

[0064] The force information is used as the input item of the second branch neural network, and the standard motion information of the robot is used as the output item of the second branch neural network;

[0065] Using prediction loss and reconstruction loss, the first branch neural network and the second branch neural network are trained to obtain a multi-dimensional interaction model;

[0066] The multidimensional interaction model is:

[0067] ;

[0068] ;

[0069] Where, is the instrument standard force information output by the first branch neural network, is the robot standard motion information output by the second branch neural network, For visual information, is force information, CNN1 is the first branch neural network, and CNN2 is the second branch neural network.

[0070] The multi-dimensional interactive model constructed by the present invention includes two neural network branch structures, which are used to realize interactive supervision analysis of visual information and force information. Among them, the first neural network branch is used to establish a mapping relationship between visual information and instrument standard force information, so as to output instrument standard force information according to visual information, so as to achieve the purpose of matching the visual information with the most suitable instrument force information. The second neural network branch is used to establish a mapping relationship between force information and robot standard motion information, so as to output robot standard motion information according to force information, so as to achieve the purpose of matching the force information with the most suitable robot motion information. The two neural network branches each establish an interactive mapping relationship so that the prediction results output by each have corresponding matching, thereby ensuring the unity of robot motion information and instrument force information at the same process node or at the same time.

[0071] The prediction loss is:

[0072] ;

[0073] Where, To predict losses, The standard force information of the instrument at the t-th process node output by the first branch neural network (corresponding to Figure 3 in ), is the true value of the instrument standard force information at the t-th process node in the data set (corresponding to Figure 3 in ), The standard motion information of the robot at the tth process node output by the second branch neural network (corresponding to Figure 3 in ), is the true value of the robot standard motion information at the tth process node in the dataset (corresponding to Figure 3 in ), N is the total number of process nodes in the standard laparoscopic surgery process, and Both are L2 norm forms.

[0074] When training two neural network branch structures, the present invention adopts a self-supervision method to ensure that the robot motion information and the instrument force information are unified and standardized. First, the two neural network branches are trained with the difference between the instrument standard force information output by the first branch neural network and the true value of the instrument standard force information in the data set, and the difference between the robot standard motion information output by the second branch neural network and the true value of the robot standard motion information in the data set (i.e., prediction loss). In this way, while making the prediction results output by each branch have corresponding matching, the prediction results output by each branch are closest to the true results, and the higher the prediction accuracy performance, the true value of the instrument standard force information in the data set and the true value between the robot standard motion information in the data set are used as label values for training the two neural network branches, thereby realizing mutual self-supervision training of the two neural network branch structures at the instrument force information and robot motion information levels.

[0075] The reconstruction loss is:

[0076] ;

[0077] Where, is the reconstruction loss, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, for the reason The converted force information, for the reason The converted visual information, exchange is the conversion network, N is the total number of process nodes in the standard laparoscopic surgery process, and All are L2 norm forms;

[0078] Among them, the conversion network exchange of force information and visual information is:

[0079] ;

[0080] ;

[0081] Where, The force information at the tth process node output by the force information conversion network (corresponding to Figure 3 in ), is the instrument standard force information at the t-th process node output by the first branch neural network, The visual information at the t-th process node output by the visual information conversion network (corresponding to Figure 3 in ), It is the standard motion information of the robot at the t-th process node output by the second branch neural network. Both CNN3 and CNN4 are convolutional neural networks.

[0082] Secondly, the two neural network branches are trained based on the difference between the force information obtained by inverting the instrument standard force information output by the first branch neural network and the original force information, and the difference between the visual information obtained by inverting the robot standard motion information output by the second branch neural network and the original visual information (i.e., reconstruction loss). This can further ensure that the instrument standard force information predicted and output by the first branch neural network based on the original visual information is consistent with the original force information, and that there is temporal consistency between the original visual information and the original force information.

[0083] Therefore, it is ensured that the first branch neural network can output the machine force information that best matches the visual information. Similarly, it is ensured that the robot motion information predicted and output by the second branch neural network based on the original force information is consistent with the original force information. There is temporal uniformity between the original visual information and the original force information. Therefore, it is ensured that the second branch neural network can output the robot motion information that best matches the force information. The use of reconstruction loss ensures that the machine force information and robot motion information output by the two neural network branch structures inherit the temporal uniformity characteristics between the original visual information and the original force information. The original visual information and the original force information are still used in the reconstruction loss as label values for training the two neural network branches, thereby realizing mutual self-supervised training of the two neural network branch structures at the visual information and force information levels.

[0084] The method for real-time interactive control of a laparoscopic surgical robot using the real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interactive model includes:

[0085] The real-time visual information fed back by the camera and the real-time force information fed back by the force sensor are input into the multi-dimensional interaction model, and the first branch neural network in the multi-dimensional interaction model outputs the real-time motion information of the robot, and the second branch neural network in the multi-dimensional interaction model outputs the real-time force information of the instrument;

[0086] The laparoscopic surgical robot is controlled to perform surgical operations based on the robot's real-time motion information and the instrument's real-time force information.

[0087] The loss function of the exchange network between force information and visual information is:

[0088] ;

[0089] Where, is the prediction loss of the conversion network, is the force information at the tth process node output by the force information conversion network, is the visual information at the t-th process node output by the visual information conversion network, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, N is the total number of process nodes in the standard laparoscopic surgery process, and Both are L2 norm forms.

[0090] In order to realize the force information obtained by inverse mapping the standard force information of the instrument and the visual information obtained by inverse mapping the standard motion information of the robot, the present invention constructs a conversion network, establishes a mapping relationship between the standard force information of the instrument and the force information, and a mapping relationship between the standard motion information of the robot and the visual information, thereby respectively realizing the force information predicted according to the standard force information of the instrument and the visual information predicted according to the standard motion information of the robot.

[0091] The visual information at each process node is normalized, and the force information at each process node is normalized.

[0092] like Figure 2 As shown, the present invention provides a multi-dimensional interactive system for a laparoscopic surgical robot, which is applied to a multi-dimensional interactive method for a laparoscopic surgical robot. The system includes:

[0093] A data acquisition unit is used to mark the robot's standard motion information in the visual information recorded by the camera at each process node in the standard laparoscopic simulation surgery process, and to mark the instrument's standard force information in the force information recorded by the force sensor at each process node;

[0094] A model building unit is used to combine visual information, force information, robot standard motion information, and instrument standard force information at each process node into a data set, and train a two-branch neural network based on the data set to build a multi-dimensional interaction model for interactively controlling the laparoscopic surgical robot through visual and force information;

[0095] The interactive control unit is used to perform real-time interactive control of the laparoscopic surgical robot using the real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interactive model.

[0096] The present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a multi-dimensional interaction method such as a laparoscopic surgical robot is implemented.

[0097] The present invention uses visual information and force information to construct a multi-dimensional interactive model for controlling the laparoscopic surgical robot, thereby analyzing control parameters based on multi-dimensional information interaction and improving control accuracy. Moreover, the multi-dimensional interactive model can realize automatic analysis of the robot's control parameters, improve the objectivity and efficiency of the control analysis, and ultimately achieve standardized control during the surgical operation to ensure the effectiveness of the surgical operation.

[0098] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A multi-dimensional interactive system for a laparoscopic surgical robot, characterized in that: include: A data acquisition unit is used to mark the robot's standard motion information in the visual information recorded by the camera at each process node in the standard laparoscopic simulation surgery process, and to mark the instrument's standard force information in the force information recorded by the force sensor at each process node; The model building unit is used to combine the visual information, force information, robot standard motion information, and instrument standard force information at each process node into a data set, and train a two-branch neural network based on the data set to build a multi-dimensional interaction model for controlling the laparoscopic surgical robot through visual and force information interaction, including: Using visual information as an input item of the first branch neural network and using instrument standard force information as an output item of the first branch neural network; The force information is used as the input item of the second branch neural network, and the standard motion information of the robot is used as the output item of the second branch neural network; Using prediction loss and reconstruction loss, the first branch neural network and the second branch neural network are trained to obtain a multi-dimensional interaction model; The multidimensional interaction model is: ； ； is the instrument standard force information output by the first branch neural network, is the robot standard motion information output by the second branch neural network, For visual information, is force information, CNN1 is the first branch neural network, and CNN2 is the second branch neural network; Predicted losses for: ； is the instrument standard force information at the t-th process node output by the first branch neural network, is the true value of the instrument standard force information at the t-th process node in the data set, is the standard motion information of the robot at the tth process node output by the second branch neural network, is the true value of the robot standard motion information at the t-th process node in the dataset, N is the total number of process nodes in the laparoscopic surgery standard process, and All are L2 norm forms; Reconstruction loss for: ； is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, for the reason The converted force information, for the reason The converted visual information is exchanged as a conversion network. and All are L2 norm forms; The conversion network exchange of force information and visual information is: ； ； is the force information at the tth process node output by the force information conversion network, The visual information at the t-th process node is output by the visual information conversion network. Both CNN3 and CNN4 are convolutional neural networks. The interactive control unit is used to perform real-time interactive control of the laparoscopic surgical robot using the real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interactive model.

2. The multi-dimensional interactive system of a laparoscopic surgical robot according to claim 1, characterized in that: The robot's standard motion information and real-time motion information both include the robot's joint angles and position coordinates; the instrument's standard force information and real-time force information both include: the depth, amplitude, speed, and strength of the robot's loaded surgical instruments pulling and cutting tissues.

3. The multi-dimensional interactive system of a laparoscopic surgical robot according to claim 1, characterized in that: The method for the interactive control unit to interactively control the laparoscopic surgical robot in real time using the robot's real-time motion information and the instrument's real-time force information predicted by the multi-dimensional interactive model includes: The real-time visual information fed back by the camera and the real-time force information fed back by the force sensor are input into the multi-dimensional interaction model, and the first branch neural network in the multi-dimensional interaction model outputs the real-time motion information of the robot, and the second branch neural network in the multi-dimensional interaction model outputs the real-time force information of the instrument; The laparoscopic surgical robot is controlled to perform surgical operations based on the robot's real-time motion information and the instrument's real-time force information.

4. The multi-dimensional interactive system of a laparoscopic surgical robot according to claim 1, characterized in that: Loss function of the exchange network between force information and visual information for: ； is the force information at the tth process node output by the force information conversion network, is the visual information at the t-th process node output by the visual information conversion network, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, N is the total number of process nodes in the standard laparoscopic surgery process, and Both are L2 norm forms.

5. The multi-dimensional interactive system of a laparoscopic surgical robot according to claim 1, characterized in that: The model building unit normalizes the visual information at each process node and normalizes the force information at each process node.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, a multi-dimensional interaction method for a laparoscopic surgical robot is implemented, including the following steps: In the standard laparoscopic simulation surgery process, the robot's standard motion information is marked in the visual information recorded by the camera at each process node, and the instrument's standard force information is marked in the force information recorded by the force sensor at each process node; The visual information, force information, robot standard motion information, and instrument standard force information at each process node are combined into a dataset. A two-branch neural network is trained based on the dataset to construct a multi-dimensional interaction model for controlling the laparoscopic surgical robot through visual and force information interaction, including: Using visual information as an input item of the first branch neural network and using instrument standard force information as an output item of the first branch neural network; The force information is used as the input item of the second branch neural network, and the standard motion information of the robot is used as the output item of the second branch neural network; Using prediction loss and reconstruction loss, the first branch neural network and the second branch neural network are trained to obtain a multi-dimensional interaction model; The multidimensional interaction model is: ； ； is the instrument standard force information output by the first branch neural network, is the robot standard motion information output by the second branch neural network, For visual information, is force information, CNN1 is the first branch neural network, and CNN2 is the second branch neural network; Predicted losses for: ； is the instrument standard force information at the t-th process node output by the first branch neural network, is the true value of the instrument standard force information at the t-th process node in the data set, is the standard motion information of the robot at the tth process node output by the second branch neural network, is the true value of the robot standard motion information at the t-th process node in the dataset, N is the total number of process nodes in the laparoscopic surgery standard process, and All are L2 norm forms; Reconstruction loss for: ； is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, for the reason The converted force information, for the reason The converted visual information is exchanged as a conversion network. and All are L2 norm forms; The conversion network exchange of force information and visual information is: ； ； is the force information at the tth process node output by the force information conversion network, The visual information at the t-th process node is output by the visual information conversion network. Both CNN3 and CNN4 are convolutional neural networks. The real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interaction model are used to perform real-time interactive control of the laparoscopic surgical robot.

7. The computer-readable storage medium according to claim 6, wherein: The robot's standard motion information and real-time motion information both include the robot's joint angles and position coordinates; the instrument's standard force information and real-time force information both include: the depth, amplitude, speed, and strength of the robot's loaded surgical instruments pulling and cutting tissues.

8. The computer-readable storage medium according to claim 6, wherein: The method for real-time interactive control of a laparoscopic surgical robot using the real-time motion information of the robot and the real-time force information of the instrument predicted by the multi-dimensional interactive model includes: The real-time visual information fed back by the camera and the real-time force information fed back by the force sensor are input into the multi-dimensional interaction model, and the first branch neural network in the multi-dimensional interaction model outputs the real-time motion information of the robot, and the second branch neural network in the multi-dimensional interaction model outputs the real-time force information of the instrument; The laparoscopic surgical robot is controlled to perform surgical operations based on the robot's real-time motion information and the instrument's real-time force information.

9. The computer-readable storage medium according to claim 6, wherein: Loss function of the exchange network between force information and visual information for: ； is the force information at the tth process node output by the force information conversion network, is the visual information at the t-th process node output by the visual information conversion network, is the force information at the tth process node in the data set, is the visual information at the t-th process node in the dataset, N is the total number of process nodes in the standard laparoscopic surgery process, and Both are L2 norm forms.

10. The computer-readable storage medium according to claim 6, wherein: The visual information at each process node is normalized, and the force information at each process node is normalized.

Images