Image feature extraction and analysis method and system for laparoscopic mirror skill assessment

Abstract

The application discloses a kind of image feature extraction and analysis method and system of laparoscopic mirror supporting skill evaluation, method includes S100: according to the task characteristic analysis of laparoscopic surgery Determine the describable feature and hidden feature for evaluation, obtain evaluation feature dataset, and the dataset is labeled;S200: based on YOLO_v5 algorithm Feature extraction of instrument, obtain the kind feature of instrument, according to Bounding Box Obtain the position of instrument tip, observation distance feature;S300: obtain the local magnification of instrument tip and make classless tip dataset, train the classless tip recognition network based on YOLO_v5, and classless tip dataset is sent into another YOLOv5 network Training, after training, obtain tip recognition dataset;S400: quantitative analysis is carried out to instrument kind feature, the position of instrument tip and observation distance feature, and tip recognition dataset.The method of the application provides objective function for laparoscopic autonomous mirror supporting robot optimization control.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fastening equipment, and more particularly relates to an image feature extraction and analysis method and system for laparoscopic mirror holding skill evaluation. BACKGROUND

[0002] Abdominal minimally invasive surgery has the advantages of small trauma to patients and short recovery time, and has been widely popularized in surgery. Abdominal minimally invasive surgery requires several small holes to be punched in the abdomen of a patient, and a laparoscope and surgical instruments are inserted into the vicinity of the affected area from the small holes, and a surgeon operates outside the body to complete the surgery. During the surgery, a mirror holder holds the laparoscope, and the main surgeon commands the mirror holder to move the laparoscope by means of oral instructions to provide the view that the main surgeon wants. However, in actual surgery, there are the following problems: (1) the efficiency of oral instructions is low, and at the same time, the psychological burden of the main surgeon is increased; (2) the surgery time is long, sometimes even up to 10 hours, which brings psychological and physiological burden to the mirror holder; (3) it takes a long time of actual surgery experience to cultivate excellent mirror holders, and the training cost of a hospital is high. Therefore, many researchers have developed a robotic laparoscope autonomous mirror holding system to improve the efficiency of laparoscopic surgery. Among them, automatic skill evaluation is of great significance in the training of a laparoscope autonomous mirror holding robot. A lot of research has been done in the evaluation of surgical skills, and various quantitative indicators have been proposed. However, in the field of surgery, there is no written quantitative indicator for laparoscope holding operation skills because of the variety of laparoscopic surgeries and complex scenes, and the real mirror holding operation is very dependent on the experience of the mirror holder and the tacit understanding with the surgeon.

[0003] In the field of mirror holding robot development, a large number of researches take the following accuracy as the only evaluation index. However, in real laparoscopic surgery, when performing a stage task, the mirror holder basically does not move after placing the lens at a good viewing angle through the left process, that is, the laparoscope does not continuously move following the instrument tip at all times, but makes intermittent movements according to the key tasks in the surgery. Considering the dependence of mirror holding operation on experience, the European Cognitive-Guided Surgery team disassembles the surgical process task by task, allowing the surgeon to intervene in the robot mirror holding operation for training, and evaluating the quality of the current picture (good / average / poor) by the surgeon's monitoring, with the expert's evaluation representing the experience-dependent part.

[0004] The above method has the following problems to be solved: (1) the timing evaluation is a momentary perspective, and the mirror holding is performed in the entire operation process, so the evaluation scheme cannot guarantee that the model with the minimum prediction error is also the best model for the entire mirror holding operation; (2) how to extract features in the image that can be applied to intraoperative evaluation, including the type, position and observation distance of the instrument in the image; (3) in order to prevent massive bleeding or bile outflow caused by the loosening of the ligature clip in cholecystectomy and other operations, the tip of the instrument must be exposed in the field of view of the doctor, so as to ensure that the artery or gallbladder duct is completely stripped by the dissection forceps and then clamped by the ligature clip, otherwise, the laparoscope needs to be deflected, and whether the tip of the target instrument can be observed in the picture is an important feature of the evaluation and is also the core extraction target of the image feature extraction algorithm. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides an image feature extraction and analysis method and system for laparoscopic mirror holding skill evaluation, wherein the describable features and hidden features used for evaluation are determined according to the task characteristic analysis of laparoscopic surgery, an evaluation feature dataset is obtained, and the dataset is labeled; the features of the instrument are extracted based on a YOLO_v5 algorithm, the type features of the instrument are obtained, the position and observation distance features of the instrument tip are obtained according to BoundingBox, a local magnified image of the instrument tip is obtained to make a classless tip dataset, a classless tip recognition network based on YOLO_v5 is trained, and the classless tip dataset is input into another YOLO_v5 network for training, and a tip recognition dataset is obtained after the training is completed; the instrument type features, the position and observation distance features of the instrument tip, and the tip recognition dataset are quantitatively analyzed, which on the one hand provides a target function for the optimization control of a laparoscopic autonomous mirror holding robot, and on the other hand provides a reference basis for formulating a mirror holding evaluation standard in the medical field.

[0006] To solve the above problems, according to a first aspect of the present application, an image feature extraction and analysis method for laparoscopic mirror holding skill evaluation is provided, comprising the following steps:

[0007] S100: describable features and hidden features used for evaluation are determined according to the task characteristic analysis of laparoscopic surgery, an evaluation feature dataset is obtained, and the dataset is labeled;

[0008] S200: the features of the instrument are extracted based on a YOLO_v5 algorithm, the type features of the instrument are obtained, and the position and observation distance features of the instrument tip are obtained according to BoundingBox;

[0009] S300: Obtain a local magnification of the instrument tip, make a non-category tip data set, train a non-category tip recognition network based on YOLO_v5, and input the non-category tip data set into another YOLO_v5 network for training. After the training is completed, a tip recognition data set is obtained.

[0010] S400: Quantitative analysis is performed on the instrument type features obtained in step S200, the position of the instrument tip, and the observation distance features, and the tip recognition data set obtained in step S300.

[0011] Further, in step S100, the describable features are one or more of the type, position, observation distance, or tip occlusion status of the instrument, and the hidden features are the experience of the doctor.

[0012] Further, step S200 includes:

[0013] S201: Image segmentation processing is performed on the laparoscopic surgery video.

[0014] S202: The labeled data set is used as a YOLO training sample for training. The ground truth of the instrument anchor box in the image is labeled using a data set labeling tool Labeling for each image in the training set. After YOLO training is completed, the anchor box of each instrument in the validation set image can be determined. According to the name of the anchor box, different instruments can be distinguished, and the type features of the instruments can be obtained.

[0015] S203: The position information of the anchor box in the image is read by YOLO, the center point c, length w, and width h of the anchor box are read, the upper left corner of the image is taken as the origin, the horizontal direction to the right is the positive direction of the x-axis, and the vertical direction downward is the positive direction of the y-axis. The image center point can be determined to obtain the position features of the instrument, and then the observation distance features of the instrument can be estimated by the area w x h of the instrument anchor box.

[0016] Further, step S200 includes:

[0017] S204: Obtain the hidden layer output of the backbone: the output after the neural network backbone is obtained, which is the tensor representation of the hidden layer.

[0018] S205: Obtain the global uniform pooling layer and the linear layer.

[0019] S206: Determine the classification loss function and the regression loss function of the target instrument feature extraction task, wherein the classification loss function is:

[0020]

[0021] Where I is an indicator function, which determines whether the center of obj falls in the grid, when the center of obj falls in the grid, I = 1, otherwise I = 0, p(c) is the probability distribution of each category detected, The theoretical probability distribution of each category, s represents dividing the image into SxS grids, and i is the i-th grid.

[0022] The regression loss function takes into account three important set factors: overlapping area, center point distance, and CIOU_Loss of aspect ratio:

[0023]

[0024] Where IOU is the intersection over union of the Prediction box (network predicted bounding box) and the Ground truth box (the true bounding box of the object), Distance_2 2 is the Euclidean distance between the two centers of the Prediction box and the Ground truth box, Distance_C 2 is the diagonal distance of the minimum enclosing box (the smallest bounding box that can contain the target to be detected) obtained by first taking the union of the Prediction box and the Ground truth box. V is a parameter that measures the consistency of the aspect ratio.

[0025] Further, v is:

[0026]

[0027] Where w gt , h gt is the width and height of the Ground truth box, w P , h P is the width and height of the Prediction box.

[0028] Further, the image segmentation processing in step S201 includes: the processing video frame rate is 33.73-50fps, every 0.5s takes a video and saves it as a picture, and each picture is named with the timestamp in the original video.

[0029] Further, step S300 includes:

[0030] S301: According to the anchor frame of the instrument, determine the position and size of the anchor frame in the image, crop the picture outside the anchor frame in the original picture, and only keep the internal picture, which is the positioning area picture;

[0031] S302: Mark the tip of the instrument inside the cropped positioning area map through the Labeling tool, regardless of which instrument the tip in the positioning area belongs to, only determine it as a tip in the tip identification process, and obtain a classless tip dataset;

[0032] S303: Send the classless tip dataset into another YOLOv5 network for training, and after the training is completed, classless tip identification can be performed.

[0033] According to a second aspect of the present application, an image feature extraction and analysis system for laparoscopic mirror holding skill evaluation is provided, comprising:

[0034] A feature evaluation and data labeling module is configured to determine describable features and hidden features for evaluation according to the task characteristics of laparoscopic surgery, obtain an evaluation feature dataset, and label the dataset;

[0035] A feature extraction module is configured to extract the features of the instrument based on a YOLO_v5 algorithm, obtain the category features of the instrument, and obtain the position and observation distance features of the instrument tip according to the BoundingBox;

[0036] An instrument tip identification module is configured to obtain a local magnified image of the instrument tip, make a classless tip dataset, train a classless tip identification network based on YOLO_v5, send the classless tip dataset into another YOLOv5 network for training, and obtain a tip identification dataset after the training is completed;

[0037] A quantitative analysis module is configured to perform quantitative analysis on the obtained instrument category features, instrument tip position and observation distance features, and tip identification dataset.

[0038] According to a third aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a laparoscopic mirror holding skill evaluation image feature extraction and analysis method program, which implements the steps of the laparoscopic mirror holding skill evaluation image feature extraction and analysis method when executed by a processor.

[0039] According to a fourth aspect of the present application, a terminal device is provided, which comprises a memory, a processor, and a laparoscopic mirror holding skill evaluation image feature extraction and analysis method program stored on the memory and executable on the processor, which configures the steps of the laparoscopic mirror holding skill evaluation image feature extraction and analysis method.

[0040] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0041] 1. The method of the present application determines the describable features and hidden features for evaluation according to the task characteristics of laparoscopic surgery, obtains the evaluation feature dataset, and labels the dataset; extracts the features of the instrument based on the YOLO_v5 algorithm, obtains the category features of the instrument, obtains the position and observation distance features of the instrument tip according to the BoundingBox, obtains the local magnification image of the instrument tip to make a classless tip dataset, trains a classless tip recognition network based on YOLO_v5, and inputs the classless tip dataset into another YOLO_v5 network for training, and obtains the tip recognition dataset after the training is completed; and quantitatively analyzes the instrument category features, the position and observation distance features of the instrument tip, and the tip recognition dataset, which provides a target function for the optimization control of the laparoscopic autonomous mirror lifting robot and provides a reference basis for formulating the mirror lifting evaluation standard in the medical field.

[0042] 2. The method of the present application introduces a feature extraction algorithm based on YOLO_v5, slices the whole video of the cholecystectomy surgery frame by frame, and equivalently obtains the categories, positions, observation distances and other explainable features of the instruments appearing in the operation in time sequence by calculating the position and size information of BoundingBox. The unexplainable features in the picture are read by extracting the information of the hidden layer of the target detection network, so as to efficiently extract four basic features that can be used for evaluation.

[0043] 3. The method of the present application obtains the local magnification image of the instrument tip to make a classless tip dataset and train a tip recognition network, so as to accurately judge the tip occlusion condition while reducing the complexity of the network, thereby efficiently obtaining the key features of the mirror lifting skill evaluation.

[0044] 4. The method of the present application verifies the rationality of the extracted features by quantitatively extracting the features and performing statistical analysis, which provides explainability for the laparoscopic optimization control and lays a foundation for the related research of autonomous surgery in hepatobiliary surgery. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a feature extraction workflow diagram of the laparoscopic surgery image of the embodiment of the present application;

[0046] Figure 2 The figure is a feature extraction flowchart based on YOLO_v5 of the embodiment of the present application;

[0047] Figure 3 The figure is a flowchart of training a classless tip recognition network to extract tip discrimination features based on the method of intercepting the target instrument of the embodiment of the present application;

[0048] Figure 4A schematic diagram of the statistical analysis process of the gallbladder resection surgery feature extraction of the embodiment of the present application is shown in the figure.

[0049] Figure 5 A schematic diagram of the feature extraction experimental effect of the laparoscopic surgery image of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application clearer and more understandable, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0051] As shown in Figure 1 The present embodiment provides an image feature extraction and analysis method for laparoscopic mirror holding skill evaluation, which includes the following steps:

[0052] Step one: determine the features and data set annotation for evaluation through task characteristic analysis of laparoscopic surgery.

[0053] In the task analysis process, the present patent traverses the full video of six types of surgeries in hepatobiliary surgery (EndoGIA hepatic pedicle and hepatic vein, placement of hepatic portal blocking band, separation of perihilar ligament, separation of coronary ligament and liver splitting, intrathecal separation of hepatic pedicle, and resection of gallbladder), and analyzes that the laparoscope basically does not move after finding a good view in the holding process. Existing research focuses on tracking the end of the instrument without analyzing the characteristics of the intraoperative mirror, without combining the experience of the doctor, introducing the interference factors brought by the complex operation in the operation, and without having autonomy. After analyzing with hepatobiliary surgery experts, the present application sets the features for evaluation as the describable features of the types, positions, observation distances, tip occlusion conditions of the instruments, and the indescribable features that the doctor highly depends on experience, i.e. hidden features. The surgical instruments are assigned different weights according to the observation priority of the evaluation, i.e. the seven types of instruments appearing in the operation. The evaluation feature table for feature extraction is as follows:

[0054]

[0055] Among them, the instrument detection condition (1 represents detecting the instrument, and 0 represents not detecting the instrument), and the tip occlusion condition (1 represents that the tip corresponding to the detected instrument is in the field of view, and 0 represents that the tip corresponding to the detected instrument is not in the field of view) are both Boolean types, and the other features are all numerical types.

[0056] Step two: obtain the type features of the instruments based on the feature extraction algorithm of YOLO_v5, and obtain the position and observation distance of the instrument tip according to BoundingBox, as shown in the figure.Figure 2 As shown in the embodiment of the present application, the steps specifically include the following steps:

[0057] (1) The processed video frame rate is 33.73-50 fps. The slicing process is to execute the python frame extraction program, take a video every 0.5 s and save it as a picture, and each picture is named with the timestamp in the original video.

[0058] (2) Based on YOLO_v5, the category features of the instrument are obtained, and the position features of the instrument are extracted. The instrument category features are obtained through supervised learning, that is, the YOLO training sample is marked by the surgeon. For each picture in the training set, the ground truth of the instrument anchor box in the picture is marked by the data set labeling tool Labeling, and the anchor box of each instrument is named by the category name of the instrument. According to the labeling result of the training set, YOLO can determine the anchor box of each instrument in the verification set picture after training, and according to the name of the anchor box, different instruments can be distinguished. YOLO provides an interface for reading the position information of the anchor box in the image, through which the center point c, length w and width h of the anchor box can be read. The coordinate system takes the upper left corner of the image as the origin, the horizontal direction to the right as the positive direction of the x axis, and the vertical direction downward as the positive direction of the y axis. The center point can determine the position of the instrument. The anchor box of the instrument can just contain the instrument, so the closer the instrument is to the laparoscope, the larger the instrument will be in the image, and the larger the anchor box will be. Therefore, the area w x h of the instrument anchor box can be used to roughly estimate the observation distance of the instrument, and the hidden features of the picture are the hidden layer outputs of the backbone.

[0059] (3) Obtain the hidden layer output of the backbone: the output of the neural network backbone is the tensor representation of the hidden layer, and the output of each layer of neural network can be directly read.

[0060] (4) Global average pooling layer and linear layer: Pytorch has a function GAP for global average pooling layer, and the linear layer can be normalized by using the softmax function.

[0061] (5) The loss function of the target detection task is generally composed of two parts: Classificition Loss (classification loss function) and Bounding Box Regeression Loss (regression loss function).

[0062]

[0063] where I is an indicator function that determines whether the center of the obj falls in the grid, I = 1 when the center of the obj falls in the grid, otherwise I = 0, p(c) is the probability distribution of each category detected, Theoretical probability distribution of each category.

[0064] In YOLO_v5, the Bounding Box Regeression Loss is CIOU_Loss which takes into account three important factors: overlapping area, center distance, and aspect ratio.

[0065]

[0066] where IOU is the intersection over union of the Prediction box and the Ground truth box, Distance_2 2 is the Euclidean distance of the two centers of the Prediction box and the Ground truth box, Distance_C 2 is the diagonal distance of the minimum bounding box of the union of the Prediction box and the Ground truth box. The parameter v measures the consistency of the aspect ratio, which can be defined as:

[0067]

[0068] where w gt , h gt are the width and height of the Ground truth box, w P , h P are the width and height of the Prediction box.

[0069] Step three: Obtain the local magnification of the instrument tip, make a classless tip dataset, and train the tip recognition network. For example, Figure 3As shown, the goal of this step is to detect the target instrument and determine whether the tip of the instrument appears in the field of view, which is a key feature in evaluating laparoscopic mirror skills. The basic process is as follows: for all instrument tips, the most tip part of the instrument is difficult to identify based on the characteristics of the type after magnification, and based on the target detection network of the previous step and the information obtained about the type of instrument, this patent no longer marks the tip according to the type of instrument, but only marks the tip as a feature, thereby training a YOLO_v5-based classless tip recognition network that can accurately determine whether the observed tip is in the field of view while reducing the complexity of the network model and the workload of manual labeling. The local magnification image is the anchor box in the image that determines the position and size of the anchor box in the image. The anchor box is obtained by a python program to crop the image outside the anchor box and only keep the internal image. This image is called a positioning area map. The training process is similar to the instrument recognition process. The data set for tip recognition is prepared, i.e. the tip of the instrument is marked in the positioning area map by the Labeling tool. The anchor box of the tip of each instrument is named "tip", i.e. in the tip recognition process, regardless of the type of tip in the positioning area, it is only determined as a tip. This classless tip data set is fed into another YOLOv5 network for training, and after training is completed, the classless tip recognition is performed.

[0070] Step four: quantifying the extracted features and performing statistical analysis. After obtaining the features, statistical analysis needs to be performed on the features to verify the correctness and regularity of the extracted features. The statistical analysis tool used in this invention is the professional statistical analysis tool package pandas and the visualization tools matplotlib and seaborn of python. The analysis process is as follows Figure 4 As shown, the statistical analysis process is to visualize the extracted features after data cleaning, such as drawing a scatter plot. Compare the information represented by the scatter plot with the operation in the video to intuitively analyze whether the extracted features are consistent with the features exhibited by the instrument in the video, thereby intuitively judging the accuracy and scientificity of the feature extraction method.

[0071] Example 1:

[0072] Step one: data set

[0073] As one of the most common surgical procedures in the world, cholecystectomy has surpassed its open surgery gold standard. Therefore, detecting sensitive actions in cholecystectomy, called keyframes, has important prospects for wide adoption in the clinical field. All surgical videos in this patent were provided by the Department of Hepatobiliary Surgery of a certain hospital. The videos that make up this dataset are monocular and were recorded with the assistant surgeon holding the camera. The sensitive data used by the Department of Hepatobiliary Surgery of a certain hospital were legally collected for research in this invention. Before starting the necessary annotation work, all surgical videos were anonymized. This dataset includes 945 minutes of video, with frame rates ranging from 22.73 to 50 fps in 10 cholecystectomy cases. The length of each video ranges from 30 to 150 minutes, depending on the duration of the surgery. The dataset used in this patent divides each case into 13-minute 12-second segments.

[0074]

[0075] According to the data segmentation in the above table, the network is trained on 8 videos, validated on 2 videos, and tested on 2 videos randomly selected from the dataset. Images for training are extracted from videos every 0.5 seconds, and LabelImg labels the class and tip of the instrument on the images. Due to the variation of the video dataset, there are three frame resolutions: 720x480, 1920x1080, and 3840x2160. This invention unifies these spatial dimensions by resizing them to 640x640.

[0076] Step two: setup and metrics

[0077] The proposed feature extraction network is trained in an end-to-end manner. All models are trained on a Windows 10 system with an Intel(R) 4 Core(TM) 2.30GHz i5-8300H CPU and a GeForce GTX 1050Ti GPU. During model training and detection, several key hyperparameters need to be modified, and the rest are default values. The parameter "Intersection over Union (IOU)" is used to represent the degree of overlap of two boxes. The larger the overlap area, the higher the IOU. The confidence threshold (conf-thres) is the minimum confidence (box confidence score) of the model in the detected object. The tip with ambiguous semantic features belongs to the local part of the instrument and is more identifiable than the entire instrument. The conf-thres for identifying the tip needs to be set smaller than when classifying the instrument, indicating that the confidence of finding the tip is higher, thereby improving the accuracy of tip detection.

[0078] Two YOLO_v5s models in the proposed network were trained to detect the category and tip of the instrument, respectively. The mean Average Precision (mAP) is an indicator used to evaluate the multi-feature extraction network evaluation, which measures the average precision (AP) of the model among all subjects. After training, the mAP of the class and tip recognition reached 0.9954 and 0.9948, respectively.

[0079] YOLO is a single-stage network with good real-time performance. Figure 5 The video clip lasting 33 minutes is displayed, and the execution time of each frame is extracted. Figure 5 The execution time of each frame shown is associated with two scenarios: when there is no instrument, the average detection time of each frame is less than 0.02s, and when the target appears, the average detection time of each frame is 0.035s. Because the multi-feature extraction time is shorter than the frame sampling interval 0.5s, the real-time performance of the method is satisfactory.

[0080] Those skilled in the art will readily understand that the above description is only preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for image feature extraction and analysis in laparoscopic laparoscopic skill assessment, characterized in that, Includes the following steps: S100: Based on the task characteristics analysis of laparoscopic surgery, determine the describable and hidden features for evaluation, obtain the evaluation feature dataset, and label the dataset; S200: Extract instrument features based on YOLO_v5 algorithm to obtain instrument type features, and obtain instrument end position and observation distance features based on BoundingBox; Step S200 includes: S201: Perform image segmentation processing on laparoscopic surgery videos; S202: YOLO is trained using the labeled dataset as training samples. For each image in the training set, the ground truth of the device anchor box in the above image is labeled using the dataset labeling tool Labeling. After YOLO training is completed, the anchor box of each device in the validation set image can be determined. Different devices can be distinguished based on the name of the anchor box, and the type features of the device can be obtained. S203: Use YOLO to read the position information of the anchor frame in the image, read the center point c, length w, and width h of the anchor frame, take the upper left corner of the image as the origin, the horizontal direction to the right is the positive direction of the x-axis, and the vertical direction downward is the positive direction of the y-axis. By obtaining the center point of the image, the position characteristics of the instrument can be determined. Then, the observation distance characteristics of the instrument can be estimated by the area w×h of the instrument anchor frame. S204: Obtain the hidden layer output of the backbone: The output of the neural network backbone is the tensor representation of the hidden layer; S205: Obtain the global uniform pooling layer and linear layer; S206: Determine the classification loss function and regression loss function for the target instrument feature extraction task; S300: Obtain a magnified local image of the instrument tip to create a classless tip dataset, train a classless tip recognition network based on YOLO_v5, and feed the classless tip dataset into another YOLO_v5 network for training. After training, the tip recognition dataset is obtained. S400: Perform quantitative analysis on the instrument type characteristics, instrument end position and observation distance characteristics obtained in step S200, and the tip identification dataset obtained in step S300.

2. The image feature extraction and analysis method for laparoscopic laparoscopic skill assessment according to claim 1, characterized in that, In step S100, the describable feature is one or more of the following: the type of instrument, its location, the observation distance, or the tip obstruction condition; and the hidden feature is the doctor's experience.

3. The image feature extraction and analysis method for laparoscopic laparoscopic skill assessment according to claim 1, characterized in that, The classification loss function is: Where I is an indicator function that determines whether the center of obj falls within the grid. I = 1 when the center of obj falls within the grid, and I = 0 otherwise. p(c) is the probability distribution of each detected category. The theoretical probability distribution for each category, where s represents dividing the image into S×S grids, and i is the i-th grid; The regression loss function takes into account three important set factors: overlap area, center point distance, and aspect ratio CIOU_Loss. Where IOU is the intersection-union ratio of the network-predicted bounding boxes and the ground truth bounding boxes of the objects, and Distance_2 2 Distance_C is the Euclidean distance between the center points of the network-predicted bounding box and the ground truth bounding box of the object. 2 First, find the union between the predicted bounding boxes of the network and the actual bounding boxes of the objects. Then, find the diagonal distance of the minimum bounding box of the union. ν is a parameter that measures the consistency of aspect ratio.

4. The image feature extraction and analysis method for laparoscopic laparoscopic skill assessment according to claim 3, characterized in that, ν is: Among them, w gt h gt These are the width and height of the ground truth box, respectively. P h P These represent the width and height of the bounding boxes predicted by the network.

5. The image feature extraction and analysis method for laparoscopic laparoscopic skill assessment according to claim 4, characterized in that, The image segmentation process in step S201 includes: processing a video frame rate of 33.73 to 50 fps, taking a segment of the video every 0.5 seconds and saving it as an image, with each image named after the timestamp in the original video.

6. A method for image feature extraction and analysis for laparoscopic laparoscopic skill assessment according to any one of claims 1-5, characterized in that, Step S300 includes: S301: Determine the position and size of the anchor frame in the image based on the anchor frame of the instrument, crop out the image outside the anchor frame in the original image, and only keep the internal image. This image is the positioning area map. S302: Label the instrument tip within the cropped positioning area map using the Labeling tool. During the tip identification process, regardless of which instrument the tip in the positioning area belongs to, it is only identified as a tip, thus obtaining a category-free tip dataset. S303: The classless tip dataset is fed into another YOLOv5 network for training. After training, classless tip recognition can be performed.

7. An image feature extraction and analysis system for assessing laparoscopic laparoscopic skills, characterized in that, include: Feature evaluation and data annotation module: used to analyze and determine the describable and hidden features for evaluation based on the task characteristics of laparoscopic surgery, obtain the evaluation feature dataset, and annotate the dataset; Feature extraction module: Extracts instrument features based on the YOLO_v5 algorithm, obtains instrument type features, and obtains the position and observation distance features of the instrument's end effector based on the BoundingBox. This module is used to perform the following steps: S201: Perform image segmentation processing on laparoscopic surgery videos; S202: YOLO is trained using the labeled dataset as training samples. For each image in the training set, the ground truth of the device anchor box in the above image is labeled using the dataset labeling tool Labeling. After YOLO training is completed, the anchor box of each device in the validation set image can be determined. Different devices can be distinguished based on the name of the anchor box, and the type features of the device can be obtained. S203: Use YOLO to read the position information of the anchor frame in the image, read the center point c, length w, and width h of the anchor frame, take the upper left corner of the image as the origin, the horizontal direction to the right is the positive direction of the x-axis, and the vertical direction downward is the positive direction of the y-axis. By obtaining the center point of the image, the position characteristics of the instrument can be determined. Then, the observation distance characteristics of the instrument can be estimated by the area w×h of the instrument anchor frame. S204: Obtain the hidden layer output of the backbone: The output of the neural network backbone is the tensor representation of the hidden layer; S205: Obtain the global uniform pooling layer and linear layer; S206: Determine the classification loss function and regression loss function for the target instrument feature extraction task; Instrument tip recognition module: used to obtain magnified local images of instrument tips to create a classless tip dataset, train a classless tip recognition network based on YOLO_v5, and feed the classless tip dataset into another YOLO_v5 network for training. After training, the tip recognition dataset is obtained. Quantitative Analysis Module: Used to perform quantitative analysis on the obtained datasets of instrument type characteristics, instrument end position and observation distance characteristics, and tip identification.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for image feature extraction and analysis of laparoscopic endoscopic skills assessment. When the program is executed by a processor, it implements the steps of the image feature extraction and analysis method for laparoscopic endoscopic skills assessment as described in any one of claims 1-6.

9. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and an image feature extraction and analysis method program for laparoscopic endoscopic skill assessment stored in the memory and executable on the processor. The image feature extraction and analysis method program for laparoscopic endoscopic skill assessment is configured to implement the steps of the image feature extraction and analysis method for laparoscopic endoscopic skill assessment as described in any one of claims 1-6.

Images