Intelligent lens holding system

Abstract

The invention relates to the technical field of medical operation detection, and discloses an intelligent endoscope holding system which comprises a mechanical arm, an endoscope, a checkerboard, a controller and an upper computer. The mechanical arm is used for clamping an endoscope; the endoscope is used for observing and displaying the inner cavity environment in real time; the checkerboard is used for calibrating the endoscope; the upper computer is used for receiving and analyzing a voice instruction of a user and image information observed by the endoscope, generating a target control instruction and sending the target control instruction to the controller; and the controller is used for controlling the mechanical arm to move according to the target control instruction. According to the method, a full-closed-loop management and control system of environmental perception, intelligent decision-making, precise execution and man-machine interaction can be formed, and collaborative optimization of millimeter-level operation precision and sub-second-level response speed in an operation scene is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical surgical detection, and particularly to an intelligent endoscope holding system. Background Art

[0002] In minimally invasive surgeries such as laparoscopy and thoracoscopy, the endoscope holding robotic arm undertakes the core task of providing real-time and stable intraoperative field images for doctors, and its performance directly affects surgical safety and operation efficiency. Existing endoscope holding systems mostly rely on doctors to manually control the endoscope holding arm to adjust the viewing angle. However, limited by human physiological factors, long-term operation is likely to cause operator fatigue, resulting in a decline in image stability, and extremely high requirements are imposed on doctors' spatial manipulation ability.

[0003] In recent years, although voice control technology has been tried in the intelligent control of endoscope holding systems, it can only execute preset single instructions. Existing voice control systems mostly adopt an open-loop control architecture of "voice command - mechanical response", and the types of commands are limited to discrete operations, lacking the ability to parse voice descriptions of continuous motion trajectories. This makes it difficult to meet doctors' refined operation requirements for the endoscope holding arm during surgery. Moreover, when facing dynamic scenarios such as tissue elastic deformation and intraoperative bleeding, the traditional PID control algorithm needs to rely on doctors to manually adjust control parameters, which not only increases the operation difficulty for doctors but also may affect the stability of the surgical field due to untimely or inaccurate parameter adjustment. Although some research has tried to introduce visual feedback, limited by image processing delay, it is difficult to achieve sub-millimeter-level real-time precision control and cannot meet the strict requirements of minimally invasive surgery for high-precision vision control.

[0004] In addition, currently, most intelligent endoscope holding systems adopt an independent modular design, lacking deep coupling between subsystems such as voice recognition, motion control, and visual feedback, resulting in low efficiency of multi-modal data fusion and the inability of each subsystem to work efficiently in coordination. For example, the commands recognized by the voice recognition system cannot be transmitted to the motion control subsystem in a timely and accurate manner, and the information obtained from visual feedback is also difficult to quickly integrate into the adjustment of motion control. These technical defects seriously restrict the application expansion of intelligent endoscope holding robots, and there is an urgent need to construct a closed-loop control system with multi-modal perception ability. Summary of the Invention

[0005] The present invention aims to provide an intelligent endoscope holding system, which can form a full closed-loop control system of "environmental perception - intelligent decision-making - precise execution - human-machine interaction" and realize the coordinated optimization of millimeter-level operation accuracy and sub-second response speed in the surgical scenario.

[0006] The basic solution provided by the present invention is: an intelligent endoscope holding system, including a robotic arm, an endoscope, a checkerboard, a controller, and a host computer; the robotic arm is used to hold the endoscope; the endoscope is used to observe and display the internal cavity environment in real time; the checkerboard is used to calibrate the endoscope;

[0007] The host computer is used to receive and analyze the voice commands of the user and the image information obtained by the endoscope observation, and generate a target control command to be sent to the controller; the controller is used to control the movement of the robotic arm according to the target control command.

[0008] When analyzing the image information, the host computer classifies the organ features in the image information based on a preset deep learning model, identifies the organ boundary and the best observation point, and generates a first control command based on the best observation point; when analyzing the voice command, the host computer converts the voice command into a text message, and based on the text message, on the basis of the first control command, confirms the incremental motion parameters, and forms a second control command as the target control command; when there is no voice command, the first control command is used as the target control command.

[0009] Furthermore, a voice control system built based on the Vosk voice recognition framework is provided in the host computer; the voice control system collects the voice commands of the user through an audio input module, extracts the audio key features therefrom based on a voice recognition model, and converts them into text messages.

[0010] Furthermore, the endoscope is arranged at the end of the robotic arm; when calibrating the endoscope, based on the pixel coordinate system corresponding to the pixel points in the image information, through the camera internal parameter matrix of the endoscope, the pixel coordinate system is converted to the camera coordinate system; then based on the rotation matrix and the translation vector, it is converted from the camera coordinate system to the world coordinate system; a robotic arm working coordinate system is constructed, and based on the world coordinate system and the robotic arm working coordinate system, the position coordinates of each pixel point in the robotic arm working coordinate system are confirmed.

[0011] Furthermore, the deep learning model is an improved YOLOv5 model; in the improved YOLOv5 model, there are an input layer, a backbone feature extraction network, a neck network, a prediction head and an output layer.

[0012] Furthermore, the backbone feature extraction network adopts CSPDarkNet, and only the middle layer and the bottom layer of CSPDarkNet are retained as the detection input; the neck network contains 2 1×1 convolutional kernels and 1 3×3 convolutional kernel arranged in a parallel structure; in the neck network, the features output by different convolutional kernels are integrated by channel dimension splicing to form a multi-scale fusion feature map; and this operation is repeated, and the multi-scale fusion feature map formed for the first time is used as the input for the second time.

[0013] Furthermore, a CBAM attention module is inserted before each CBS structure in CSPDarkNet to form a preprocessing structure; the CBS structure is a Conv-BN-SiLU network structure.

[0014] Furthermore, the improved YOLOv5 model uses the DIoU function as its loss function.

[0015] Furthermore, when generating the target control instruction, first perform path planning based on the DH parameter model used to describe the geometric transformation relationship between the coordinate systems of each joint of the robotic arm.

[0016] Furthermore, when performing path planning, use the improved RRT* path planning algorithm to perform path planning from the current observation point of the endoscope to the best observation point, or from the current observation point of the endoscope to the target point corresponding to the voice instruction.

[0017] Furthermore, in the improved RRT* path planning algorithm, a target bias strategy is incorporated; after the initial path is generated by the improved RRT* path planning algorithm, the initial path is further smoothed based on the cubic B-spline curve method to form the target path, and then the target control instruction is generated based on the target path.

[0018] The working principle and advantages of the present invention are as follows:

[0019] An intelligent endoscope holding system of the present invention can form a full closed-loop control system of "environmental perception - intelligent decision-making - precise execution - human-computer interaction", and realize the collaborative optimization of millimeter-level operation accuracy and sub-second response speed in the surgical scenario. The key points are as follows:

[0020] First, this solution designs an improved YOLOv5 model, which can achieve high detection efficiency and intracavitary environment detection accuracy. First, this solution reconstructs the lightweight feature extraction network. By abandoning the FPN+PAN multi-scale fusion structure and only retaining the middle layer (medium-scale semantic features) and the bottom layer (deep semantic features) of CSPDarkNet as the detection input, the number of parameters and the amount of calculation can be effectively reduced, and the analysis efficiency can be improved. At the same time, by adopting the parallel splicing structure of 1×1 convolution (dimensionality reduction / dimensionality increase) and 3×3 convolution (local feature extraction), not only multi-scale information (such as local details and global semantics) is retained, which can adapt to complex intracavitary environments and ensure accurate recognition, but also redundant calculations are reduced through channel dimension fusion, and a better balance can be achieved between real-time performance and accuracy. And in the channel dimension fusion, through two repeated convolution-splicing operations, a feature abstraction path similar to residual learning can be formed, enabling the network to capture higher-level context relevance, and thus achieving better detailed feature extraction capabilities.

[0021] Second, by introducing the CBAM attention mechanism and optimizing the loss function, this solution can promote the model to focus on key areas and improve the detection accuracy. Among them, by embedding CBAM into the front end of the CBS module, this solution can form a preprocessing process of "attention guidance → feature extraction", avoiding the feature deviation problem caused by noise interference in the original convolutional layer, enabling the subsequent convolutional kernels to directly act on the optimized feature space, and thus achieving a higher capture efficiency of the detail differences of tiny features. Moreover, by setting the DIoU function, the robustness of the model to features such as occlusion and deformation can be enhanced, further reducing the detection error and helping to improve the network convergence speed.

[0022] Second, the motion control efficiency of this solution is relatively high, and the motion line control is precise and smooth. Among them, through the DH parameter model, this solution can accurately describe the geometric transformation relationship between the coordinate systems of each joint of the robotic arm, and thus provide an accurate mathematical basis for subsequent path planning, enabling the motion control of the robotic arm to have strict kinematic constraint guarantees. In path planning, this solution further integrates the target bias strategy and performs trajectory smoothing and dynamic adaptation. Among them, the former can reduce the path planning time and optimize the path generation quality (such as shortening the path length, reducing the path curvature, which helps to reduce the occurrence of joint mutation faults in the robotic arm). The latter, based on the trajectory smoothing process of the cubic B-spline curve method, helps to optimize the actions of the robotic arm, reduce the motion impact, and achieve smooth and gentle motion.

[0023] Third, this solution can achieve multi-modal perception fusion, forming a full closed-loop control system of "environmental perception - intelligent decision-making - precise execution - human-machine interaction" to achieve efficient and intelligent collaborative work. Specifically, this solution takes deep learning visual perception as the core driving force and voice commands as the collaborative driving force to jointly form a multi-source fusion motion control route. Among them, based on the driving analysis of visual perception (corresponding to the first control command), it can autonomously locate to the best observation point to provide an intelligent surgical field of view; based on the driving analysis of voice commands (corresponding to the second control command), it can provide a targeted surgical field of view in real time according to specific command requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the system operation process of the first embodiment of an intelligent endoscope holding system of the present invention;

[0025] Figure 2 It is a schematic diagram of the operation process of the voice control system of the first embodiment of an intelligent endoscope holding system of the present invention;

[0026] Figure 3 It is a schematic diagram of the conversion process of coordinate calibration of the first embodiment of an intelligent endoscope holding system of the present invention;

[0027] Figure 4Schematic diagram of the improved YOLOv5 model structure of Embodiment 1 of an intelligent endoscope holding system according to the present invention. Detailed implementation manners

[0028] The following is a further detailed description through specific implementation manners:

[0029] Embodiment 1

[0030] The embodiment is basically as shown in the appendix Figure 1 shown: An intelligent endoscope holding system includes a robotic arm, an endoscope, a checkerboard, a controller, and a host computer.

[0031] The robotic arm is used to hold the endoscope; the endoscope is used to observe and display the internal cavity environment in real time; the checkerboard is used to calibrate the endoscope. In this embodiment, the robotic arm, the endoscope, and the checkerboard all use existing devices.

[0032] The host computer is used to receive and analyze the user's voice commands and the image information obtained by the endoscope observation, and generate a target control command to send to the controller; the controller is used to control the movement of the robotic arm according to the target control command.

[0033] When analyzing the image information, the host computer classifies the organ features in the image information based on a preset deep learning model, identifies the organ boundary and the best observation point, and generates a first control command based on the best observation point; when analyzing the voice command, the host computer converts the voice command into a text message, and based on the text message, on the basis of the first control command, confirms the incremental motion parameters, and forms a second control command as the target control command; when there is no voice command, the first control command is used as the target control command.

[0034] Specifically, a voice control system based on the Vosk speech recognition framework is provided in the host computer, which can recognize the speech of multiple languages in real time on the local device. The voice control system collects the user's voice commands through an audio input module, extracts the audio key features from them based on the speech recognition model, and converts them into text messages. In this embodiment, the audio input module is a voice input device such as a microphone. The voice control system captures the real-time audio stream through the pyaudio library, then transmits it to the speech recognition model of the pre-trained neural network architecture, and then the speech recognition model extracts the audio key features from it, and combines the acoustic model, the language model, and the pronunciation dictionary to recognize the voice command and convert it into a text message, as Figure 2 shown.

[0035] Preferably, after being converted into a text message, the text message is also preprocessed to filter out the noise data and irrelevant information in the text message.

[0036] The endoscope is arranged at the end of the robotic arm. When calibrating the endoscope, based on the pixel coordinate system corresponding to the pixel points in the image information, the pixel coordinate system is converted to the camera coordinate system through the camera internal parameter matrix of the endoscope; then, based on the rotation matrix and the translation vector, it is converted from the camera coordinate system to the world coordinate system; a robotic arm working coordinate system is constructed, and based on the world coordinate system and the robotic arm working coordinate system, the position coordinates of each pixel point in the robotic arm working coordinate system are confirmed.

[0037] Among them, the robotic arm working coordinate system (i.e., the robotic arm workpiece coordinate system) is also the endoscope coordinate system. Since the position transformation matrix from the robotic arm base coordinate system to the robotic arm workpiece coordinate system can be obtained by forward kinematics, the actual position of the pixel point in the robotic arm base coordinate system can be obtained based on forward kinematics here, as Figure 3 shown.

[0038] As Figure 4 shown, the deep learning model is an improved YOLOv5 model. In the improved YOLOv5 model, there are an input layer, a backbone feature extraction network, a neck network, a prediction head, and an output layer. Specifically, the backbone feature extraction network uses CSPDarkNet, and only the middle layer and the bottom layer of CSPDarkNet are retained as the detection input, which can effectively reduce the number of parameters and the amount of calculation and improve the analysis efficiency. A CBAM attention module is inserted before each CBS structure in CSPDarkNet to form a preprocessing structure; the CBS structure is a Conv-BN-SiLU network structure.

[0039] With this setting, the CBAM attention module can achieve adaptive optimization of the feature map by integrating channel and spatial attention in parallel. Among them, the channel attention branch uses dual-path aggregation of global average pooling and max pooling, which can capture the global dependencies between channels by combining a shared-weight network and dynamically calibrate the feature response intensity of each channel; the spatial attention branch can generate a spatial weight mask using a convolutional layer through the spatial feature statistics after channel compression, highlighting the activation values of the target significant regions. This dual-path cooperation mechanism enables the model to screen key semantic features (such as the spectral characteristics of organ tissues) in the channel dimension, while focusing on the anatomical structure region in the spatial dimension, effectively alleviating the problems of texture blur and noise interference caused by low illuminance and low contrast in endoscope images. Secondly, the pre-set attention injection setting enables the subsequent convolutional kernels to directly act on the enhanced feature space, which not only improves the directionality of feature extraction, can highlight the detail differences of small lesions, but also strengthens the discrimination ability for organs with similar textures. At the same time, by suppressing noise interference in non-critical regions, it can significantly improve the detection accuracy and robustness of the model in a complex cavity environment.

[0040] The neck network includes 2 1×1 convolutional kernels and 1 3×3 convolutional kernel arranged in a parallel structure. The convolutional strides of the three convolutional kernels are the same, and the spatial dimensions of the output feature maps are the same. In the neck network, the features output by different convolutional kernels are integrated by concatenation in the channel dimension to form a multi-scale fusion feature map; and this operation is repeated, with the multi-scale fusion feature map formed for the first time used as the input for the second time. Here, a parallel concatenation structure of 1×1 convolution (mainly used to change the number of channels of the feature map, achieve dimensionality reduction or increase, while increasing the non-linearity of the model, and with small computational complexity, without changing the spatial dimensions of the feature map) and 3×3 convolution (used to extract local features, capture the relationship between adjacent pixels with a certain receptive field, and obtain richer feature information) can enhance feature extraction, retain multi-scale information, and reduce redundant calculations through channel dimension fusion. Furthermore, it can achieve a balance between real-time performance and accuracy, and help enhance the feature expression ability of the network. Through repeated convolution and concatenation operations, the network can learn more advanced and abstract features, improving the object detection performance.

[0041] The improved YOLOv5 model uses the DIoU function as its loss function and is trained using a self-made training set. The self-made training set includes a publicly available dataset collected through the network (which contains multiple endoscopic environment images), and a clinically collected dataset self-collected (which includes endoscopic environment images collected clinically in the hospital). In this embodiment, there are a total of 200 endoscopic environment images in the self-made training set, and medical experts manually segment the organs to be observed in the endoscopic environment images. In this embodiment, the roLabelImg software is also used for image annotation. This software has the function of annotating rectangular boxes with rotation angles and can be used to set corresponding categories for the annotated objects. At the same time, when making the self-made training set, for the endoscopic environment images with the organ regions segmented, two points are selected on the organ to draw a line segment, and the center of the line segment is finely annotated following the doctor's habit of observing the organ and the principle of the best observation position. As for the direction of the line segment, it is uniformly marked according to the standard from left to right to ensure the consistency and standardization of the annotation. Training based on this self-made training set can provide sufficient learnable information for the improved YOLOv5 model.

[0042] Among them, the setting of the DIoU function, compared with the GIoU function used in traditional networks, the GIoU function often only focuses on the difference in the area of the bounding box, while the DIoU function can achieve a more intelligent bounding box calibration mechanism through two-dimensional constraints. When the predicted bounding box and the ground truth bounding box have no overlap, the penalty term of DIoU generates a strong gradient signal, driving the predicted bounding box to move quickly along the direction of the line connecting the center points, which can significantly shorten the number of iterations required for convergence. Especially in special spatial relationships such as horizontal / vertical arrangements, this optimization strategy based on vector distance can accurately correct the offset of the bounding box position, avoiding the path redundancy caused by simply expanding the bounding box in GIoU. Utilizing the above characteristics of the DIoU function can effectively enhance the convergence performance during model training. In addition, the DIoU function can also be applied to the non-maximum suppression process, which helps to further improve the overall efficiency of the model.

[0043] In specific applications, the improved YOLOv5 model is used to extract multi-scale features of endoscopic images, and the anchor box mechanism is used to predict the organ bounding box and class probability. The positioning of the best observation point is based on the geometric center of the line segment annotated by the doctor. Through coordinate transformation, the image pixel coordinates (u, v) of the best observation point of the required observed organ are converted into three-dimensional coordinates (x, y, z) in the base coordinate system of the robotic arm, and it is ensured that the spatial positioning error of the target point is less than 1 mm. After the robotic arm obtains the position coordinates, it moves to the corresponding point, thereby realizing the movement of the end position of the robotic arm based on vision.

[0044] When generating the target control command, first perform path planning based on the DH parameter model used to describe the geometric transformation relationship between the joint coordinate systems of the robotic arm. Specifically, by multiplying the homogeneous transformation matrices in series, the forward kinematics equation of the robotic arm is constructed, and the pose matrix of the end effector (endoscope) can be deduced step by step. The inverse kinematics solution uses the analytical method, and the angle values of each joint of the robotic arm are deduced inversely in combination with geometric constraint conditions to ensure that the end of the robotic arm can reach the target pose. The DH parameter model can provide an accurate mathematical basis for path planning, enabling the motion control of the robotic arm to have strict kinematic constraint guarantees.

[0045] When performing path planning, the improved RRT* path planning algorithm is used to perform path planning from the current observation point of the endoscope to the best observation point, or from the current observation point of the endoscope to the target point corresponding to the voice command.

[0046] In the improved RRT* path planning algorithm, a target bias strategy is incorporated. In the traditional RRT* path planning algorithm, when generating a random tree, the expansion direction of nodes is completely random. In this solution, by integrating the target bias strategy, this strategy probabilistically guides the growth direction of the random tree: with a certain probability, directly expand nodes towards the target point instead of being completely random. And it can retain the asymptotically optimal property. As the running time of the algorithm increases, the found path will gradually approach the optimal path. Preferably, in this improved RRT* path planning algorithm, the path is continuously optimized through a re-wiring mechanism. During the growth process of the random tree, the generated path is continuously adjusted. This mechanism can reduce the invalid search area through sub-sampling. For example, when it is found that certain areas are clearly unable to lead to the target, no more computational resources are wasted searching in these areas. In this way, the algorithm can focus on the areas where feasible paths may be generated faster, thus accelerating the generation of feasible paths. This improved RRT* path planning algorithm shows high search efficiency and path quality in a narrow cavity environment. The generated path is shorter, smoother, reduces unnecessary detours, and is more suitable for the requirements of actual application scenarios.

[0047] After generating the initial path by the improved RRT* path planning algorithm, based on the cubic B-spline curve method, the initial path is smoothed to form the target path, and then based on the target path, the target control instructions are generated.

[0048] Among them, the cubic B-spline curve is composed of piecewise cubic polynomials, and the curve shape is defined by control points. Its mathematical expression is:

[0049]

[0050] Where N i,3 is the cubic basis function, and P i is the control point.

[0051] Based on the cubic B-spline curve method for trajectory smoothing helps to optimize the movement of the robotic arm, reduce motion impact, and achieve smooth and gentle movement.

[0052] Furthermore, in this embodiment, a six-dimensional force sensor is also integrated at the end of the robotic arm. By measuring the three-dimensional forces Fx, Fy, Fz and the torques Mx, My, Mz when the endoscope body contacts the internal cavity tissue, a contact force vector space is constructed. This space can comprehensively describe the mechanical state when the endoscope body contacts the tissue and can provide an accurate mechanical information basis for subsequent control.

[0053] Then, an impedance control model is adopted to convert the force error (i.e., the error between the measured actual contact force and the desired force) into a joint torque command, dynamically adjusting the pose of the robotic arm; when the contact force exceeds the safety threshold (0.1 N), a reverse compensation movement is triggered (i.e., the robotic arm will move in the direction opposite to the current movement direction to reduce the contact force and bring it back within the safe range), thereby achieving millimeter-level accuracy and compliant control for the operation of the mirror body.

[0054] An intelligent mirror holding system provided in this embodiment can form a full closed-loop control system of "environmental perception-intelligent decision-making-precise execution-human-computer interaction", realizing the collaborative optimization of millimeter-level operation accuracy and sub-second response speed in the surgical scenario.

[0055] Embodiment 2

[0056] Based on Embodiment 1, this embodiment provides an application method for an intelligent mirror holding system, including the following steps:

[0057] The endoscope collects the intracavitary images in real time. In this embodiment, the collected intracavitary images are abdominal cavity images or thoracic cavity images, etc. The image information observed by the endoscope is input into the upper computer in real time. The improved YOLOv5 model in the upper computer accurately identifies the organ boundary and the best observation point (the positioning error < 1 mm) based on the image information through the backbone feature extraction network. Then, the pixel coordinates of the best observation point are dynamically mapped to the robotic arm base coordinate system through the calibration matrix, and combined with the contact force data (accuracy 0.1 N) fed back by the six-axis force sensor, a composite task instruction set including spatial constraints and force constraints is constructed (corresponding to the first control instruction).

[0058] The voice control system analyzes the doctor's voice commands and converts them into incremental motion parameters, which are jointly input into the decision-making layer with the visual target to form a motion control requirement of multi-source fusion (corresponding to the second control instruction). When there are voice commands, the second control instruction is used as the target control instruction; when there are no voice commands, the first control instruction is used as the target control instruction.

[0059] Among them, the improved RRT* path planning algorithm quickly searches for a collision-free path in the robotic arm configuration space, and uses the target bias strategy to improve the planning efficiency in the narrow surgical environment; and after generating the initial trajectory, the initial trajectory is smoothed and optimized based on the cubic B-spline curve to generate a continuous trajectory in the joint space as the target path. During the execution process, the visual servo system compensates for the organ motion error through feature point tracking, and the six-axis force sensor triggers impedance control to dynamically adjust the end pose, forming a real-time closed loop of "perception-planning-execution-correction".

[0060] After the doctor's voice commands (such as "Focus on the left kidney") are parsed, they are dynamically weighed against the priority of the automatically planned path to ensure that manual intervention takes precedence in emergency situations. A full closed-loop control system of "environmental perception-intelligent decision-making-precise execution-human-computer interaction" is formed to achieve the collaborative optimization of millimeter-level operation accuracy and sub-second response speed in the surgical scenario.

[0061] An application method of an intelligent endoscope holding system provided in this embodiment is driven by deep learning visual perception as the core and voice commands as the collaborative drive to jointly form a multi-source fusion motion control route. Among them, the drive analysis based on visual perception (corresponding to the first control command) can autonomously locate to the best observation point to provide an intelligent surgical field of view; the drive analysis based on voice commands (corresponding to the second control command) can provide a targeted surgical field of view in real time according to specific command requirements.

[0062] Embodiment III

[0063] An intelligent endoscope holding system has the following improvements based on Embodiment I.

[0064] Emergency arbitration commands are pre-stored in the host computer. In this embodiment, the emergency arbitration commands include "Emergency Stop" and "Emergency Restoration" (i.e., restore to the initial position where the organ is not touched). When the voice control system recognizes that the voice command contains an emergency arbitration command, the host computer directly activates the emergency arbitration mechanism and transmits a stop command or a restoration command to the controller in a timely manner to stop the robotic arm movement or restore the endoscope position.

[0065] In addition, in the host computer, differentiated motion parameter sets (speed, acceleration, force threshold) are preset for different surgical procedures (such as gastroscope biopsy or bronchoscope navigation) for easy direct invocation.

[0066] In the improved RRT* path planning algorithm, an adaptive RRT* parameter is additionally set. This adaptive RRT* parameter dynamically adjusts the target bias probability p according to the degree of cavity stenosis. In this embodiment, the value range of p is set to 0.1-0.5. Through this parameter setting, the planned path has a higher safety level.

[0067] The intelligent endoscope holding system provided in this embodiment has an emergency arbitration mechanism, has a certain degree of self-adaptability, and has a relatively high operating safety level.

[0068] The above are only embodiments of the present invention. Specific structures and common knowledge such as characteristics well-known in the art are not described in detail herein. Those of ordinary skill in the art know all the common general technical knowledge in the technical field to which the invention pertains before the filing date or the priority date, are able to learn all the prior art in this field, and have the ability to apply conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, complete and implement this solution in combination with their own abilities. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent.

Claims

1. An intelligent mirror holding system, characterized in that: It includes a mechanical arm, an endoscope, a chessboard, a controller and a host computer; the mechanical arm is used to clamp the endoscope; the endoscope is used to observe and display the inner cavity environment in real time; the chessboard is used to calibrate the endoscope; The host computer is used to receive and analyze the user's voice commands and the image information obtained by endoscope observation, and generate target control commands to send to the controller; The controller is used to control the movement of the robot arm according to the target control instruction; When analyzing image information, the host computer classifies organ features in the image information based on a preset deep learning model, identifies organ boundaries and optimal observation points, and generates a first control instruction based on the optimal observation points; when analyzing voice instructions, the host computer converts the voice instructions into text messages, and based on the text messages, on the basis of the first control instructions, confirms incremental motion parameters, and forms a second control instruction as a target control instruction; when there is no voice instruction, the first control instruction is used as the target control instruction.

2. The intelligent mirror holding system according to claim 1, characterized in that: The host computer is provided with a voice control system based on the Vosk voice recognition framework; the voice control system collects the user's voice commands through an audio input module, extracts key audio features from them based on a voice recognition model, and converts them into text messages.

3. The intelligent mirror holding system according to claim 1, characterized in that: The endoscope is arranged at the end of the robotic arm; when calibrating the endoscope, based on the pixel coordinate system corresponding to the pixel points in the image information, the pixel coordinate system is converted to the camera coordinate system through the camera intrinsic parameter matrix of the endoscope; then based on the rotation matrix and the translation vector, the camera coordinate system is converted to the world coordinate system; the robotic arm working coordinate system is constructed, and based on the world coordinate system and the robotic arm working coordinate system, the position coordinates of each pixel point in the robotic arm working coordinate system are confirmed.

4. The intelligent mirror holding system according to claim 1, characterized in that: The deep learning model is an improved YOLOv5 model; in the improved YOLOv5 model, an input layer, a backbone feature extraction network, a neck network, a prediction head and an output layer are provided.

5. The intelligent mirror holding system according to claim 4, characterized in that: The backbone feature extraction network adopts CSPDarkNet, and only the middle and bottom layers of CSPDarkNet are retained as detection input; the neck network includes two parallel structures. Convolution kernel and 1 Convolution kernel; in the neck network, the features of different convolution kernel outputs are integrated through channel dimension splicing to form a multi-scale fusion feature map; and this operation is repeated, with the multi-scale fusion feature map formed for the first time as the input for the second time.

6. The intelligent mirror holding system according to claim 5, characterized in that: A CBAM attention module is inserted before each CBS structure of CSPDarkNet to form a preprocessing structure; the CBS structure is a Conv-BN-SiLU network structure.

7. The intelligent mirror holding system according to claim 4, characterized in that: The improved YOLOv5 model uses the DIoU function as its loss function.

8. The intelligent mirror holding system according to claim 1, characterized in that: When generating the target control instruction, path planning is first performed based on the DH parameter model used to describe the geometric transformation relationship between the coordinate systems of the joints of the robot arm.

9. The intelligent mirror holding system according to claim 8, characterized in that: When performing path planning, the improved RRT* path planning algorithm is used to perform path planning from the current observation point of the endoscope to the optimal observation point, or from the current observation point of the endoscope to the target point corresponding to the voice command.

10. The intelligent mirror holding system according to claim 9, characterized in that: The improved RRT* path planning algorithm incorporates a target bias strategy; after the initial path is generated by the improved RRT* path planning algorithm, the initial path is smoothed based on the cubic B-spline curve method to form a target path, and then a target control instruction is generated based on the target path.

Images