A method for identifying the surgical process based on robot-assisted nephrectomy

Abstract

The present invention provides a method for identifying the surgical procedure based on robot-assisted nephrectomy, including determining the initial number of video frames; performing initial frame processing on the obtained surgical video of robot-assisted nephrectomy based on the initial number of video frames to obtain a set of surgical images; performing comprehensive detection of the image quality of the set of surgical images to determine whether the comprehensive image quality is qualified: if the comprehensive image quality is unqualified, then screening the frame images with unqualified quality in the set of surgical images and performing image enhancement processing on the screened frame images with unqualified quality; if the comprehensive image quality is qualified, then obtaining the surgical procedure feature data set of each frame image in the set of surgical images; determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image, which solves the problem that it is difficult to identify and track the surgical procedure in the analysis of complex surgical videos.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical procedure recognition, and in particular to a surgical procedure recognition method based on robot-assisted nephrectomy. Background Art

[0002] Surgical procedure recognition includes segmenting and recognizing surgical stages from surgical videos, and surgical procedure recognition is of great significance for endoscopic surgery. During the implementation of a surgery, automatic recognition of the surgical procedure can remind the doctor to pay attention to the occurrence of complications, reduce the doctor's operating errors during the surgery, and can also provide information on the progress of the relevant surgery for clinical staff outside the operating room. Subsequently, through intelligent segmentation of expert surgical videos, the doctor's workload can be reduced, a surgical video learning resource library can be constructed, guidance can be provided for junior clinicians, and it can also play a reference role in the arrangement of subsequent relevant work for the surgery.

[0003] Chinese Patent No. CN112818959B discloses a surgical procedure recognition method, device, system and computer-readable storage medium, belonging to the field of computer technology. In the embodiments of the present application, the surgical stages are divided into multiple main stages, and there are main stages that include sub-stages. Based on this, first refer to the surgical stages experienced before the first image in the surgical video to determine the main surgical stage to which the first image belongs. If the recognized main surgical stage is further divided into sub-stages, then refer to the instruments present in the first image to determine the surgical sub-stage to which the first image belongs. It can be seen that this solution combines rough recognition and fine recognition, and takes advantage of the characteristics of rough recognition and fine recognition respectively, effectively improving the fineness and accuracy of surgical procedure recognition. If surgical procedure recognition is performed during a surgery, this solution can provide accurate information for the surgery, effectively reduce surgical errors, and improve the surgical success rate.

[0004] However, existing surgical procedure recognition methods have the problem that it is difficult to perform recognition based on multiple aspects of features, resulting in difficult accurate recognition and tracking of surgical procedures in the analysis of complex surgical videos. Summary of the Invention

[0005] The present invention provides a surgical procedure recognition method based on robot-assisted nephrectomy, which solves the problem of difficult accurate recognition and tracking of surgical procedures in the analysis of complex surgical videos.

[0006] To achieve the above-mentioned invention objectives, the present invention provides the following technical solutions:

[0007] A surgical procedure recognition method based on robot-assisted nephrectomy, comprising the following steps: determining the initial number of video frames; performing initial frame processing on the obtained surgical video of robot-assisted nephrectomy based on the initial number of video frames to obtain a set of surgical images; performing comprehensive detection of the image quality of the set of surgical images to determine whether the comprehensive image quality is qualified: if the comprehensive image quality is unqualified, screening out the frame images with unqualified quality in the set of surgical images, and performing image enhancement processing on the screened frame images with unqualified quality; if the comprehensive image quality is qualified, obtaining a surgical procedure feature data set for each frame image in the set of surgical images, the surgical procedure feature data set including instrument shape data, wound surface shape, relative distance data between the instrument center and the frame image center, and included angle data between the instrument center axis and the horizontal line; determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image.

[0008] Optionally, the determining the initial number of video frames includes the following steps: obtaining the surgical video of robot-assisted nephrectomy and the brightness data of the resection site during the operation; determining the surgical video duration and the basic resection procedure; determining the average intraoperative ambient brightness based on the brightness data of the resection site during the operation; determining the initial number of video frames based on the surgical video duration, the basic resection procedure, and the average intraoperative ambient brightness.

[0009] Optionally, the determining the initial number of video frames based on the surgical video duration, the basic resection procedure, and the average intraoperative ambient brightness includes the following steps: comparing the basic resection procedure with each basic resection procedure label stored in the database to determine the basic resection procedure label corresponding to the basic resection procedure; obtaining the frame number matching data set stored in the database corresponding to the basic resection procedure label, the frame number matching data set including a plurality of frame number matching data, the frame number matching data including the surgical video matching duration and the average intraoperative ambient matching brightness; comparing the surgical video duration and the average intraoperative ambient brightness with each frame number matching data in the frame number matching data set one by one to obtain a comparison coefficient; obtaining the initial number of video frames stored in the database corresponding to the frame number matching data corresponding to the smallest comparison coefficient.

[0010] Optionally, the method for obtaining the comparison coefficient is as follows:

[0011]

[0012] where bx is the comparison coefficient, St is the surgical video duration, Sd is the average intraoperative ambient brightness, a is the number of the frame number matching data, and Ct a is the surgical video matching duration of the a-th frame number matching data, and Cd aThe intraoperative average ambient matching luminance for the a-th sub-frame quantity matching data, where e is the natural constant.

[0013] Optionally, the comprehensive detection of the image quality of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: obtaining the image detection data of each frame image in the surgical image set, where the image detection data includes the luminance range, image uniformity, and wound surface color accuracy; comparing the image detection data of each frame image with the specified image detection data of the frame images stored in the database to obtain an image quality comparison coefficient, where the specified image detection data includes the specified luminance range, specified image uniformity, and specified wound surface color accuracy; and determining whether the comprehensive image quality is qualified based on the image quality comparison coefficient.

[0014] Optionally, the determination of whether the comprehensive image quality is qualified based on the image quality comparison coefficient includes the following steps: obtaining the mean square deviation of the image quality comparison coefficient; determining whether the mean square deviation is greater than the specified mean square deviation value stored in the database: if the mean square deviation is greater than the specified mean square deviation value stored in the database, then the comprehensive image quality is unqualified; if the mean square deviation is not greater than the specified mean square deviation value stored in the database.

[0015] Optionally, the determination of the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image includes the following steps: obtaining the allowable error data of the surgical procedure features of each frame image in the surgical image set; obtaining the matching data sets of the surgical procedure features stored in the database; comprehensively analyzing the allowable error data of the surgical procedure features of each frame image, the matching data sets of the surgical procedure features, and the surgical procedure feature data set of each frame image to obtain a process matching coefficient for each frame image; obtaining the minimum process matching coefficient corresponding to each frame image; and determining the surgical procedure label stored in the database corresponding to the matching data set of the surgical procedure features corresponding to each minimum process matching coefficient as the surgical procedure corresponding to the frame image.

[0016] Optionally, the allowable error data of the surgical procedure features includes the allowable error of the instrument shape matching, the allowable error of the wound surface shape matching, the allowable error of the relative distance matching between the instrument center and the frame image center, and the allowable error of the included angle matching between the instrument center axis and the horizontal line, and the matching data sets of the surgical procedure features include the instrument shape matching data, the wound surface matching shape, the relative distance matching data between the instrument center and the frame image center, and the included angle matching data between the instrument center axis and the horizontal line.

[0017] Optionally, obtaining the allowable error data of the surgical process features of each frame image in the surgical image set includes the following steps: obtaining the brightness of the resection site corresponding to each frame image; obtaining the image quality comparison coefficient corresponding to each frame image; combining the brightness of the resection site corresponding to each frame image with the image quality comparison coefficient into an error matching data group; comparing the error matching data group with each matching data group stored in the database one by one to obtain an error matching coefficient, where the matching data group includes the matching brightness of the resection site and the image quality matching comparison coefficient; obtaining the allowable error data of the surgical process features of the frame image corresponding to the smallest error matching coefficient stored in the database corresponding to the matching data group.

[0018] Optionally, obtaining the brightness of the resection site corresponding to each frame image includes the following steps: obtaining the time data corresponding to each frame image; obtaining the brightness data of the resection site during the operation; and obtaining the brightness of the resection site corresponding to each frame image from the brightness data of the resection site during the operation based on the time data corresponding to each frame image.

[0019] The above technical solution has at least the following beneficial effects compared with the prior art:

[0020] The above solution, based on the surgical process recognition method for robot-assisted nephrectomy, through initial frame processing and comprehensive image quality detection of the surgical video, ensures that the image quality meets the standard, and uses image enhancement processing to improve the frame images with unqualified quality, so as to accurately extract the surgical process feature data set. This makes the analysis of the surgical video more accurate, helps doctors better understand the surgical process, improves surgical safety and efficiency, and solves the problem that it is difficult to identify and track the surgical process in the analysis of complex surgical videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of the surgical process recognition method for robot-assisted nephrectomy according to the present invention;

[0023] Figure 2 It is a surgical process splitting framework for robot-assisted partial nephrectomy according to the present invention;

[0024] Figure 3 It is a schematic diagram of the surgical stage provided by the present invention;

[0025] Figure 4The detailed introduction diagram of the surgical stage provided by the present invention;

[0026] Figure 5 The actual surgical scene diagram provided by the present invention;

[0027] Figure 6 The surgical marking diagram provided by the present invention. Specific implementation manners

[0028] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0029] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains. The "first", "second" and similar terms used in the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. Similarly, the terms such as "a", "an" or "the" do not denote a quantity limitation, but mean that there is at least one. The terms such as "include" or "comprise" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connect" or "be connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0030] It should be noted that the "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to represent relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0031] Aiming at the problem that it is difficult to identify and track the surgical process in the analysis of complex surgical videos, the present invention provides a method for identifying the surgical process based on robot-assisted nephrectomy. By performing initial frame division processing and comprehensive image quality detection on the surgical video, the image quality is ensured to meet the standard, and the frame images with unqualified quality are enhanced by image enhancement processing, so as to accurately extract the surgical process feature dataset. This makes the analysis of surgical videos more accurate, helps doctors better understand the surgical process, and improves surgical safety and efficiency.

[0032] Such as Figure 1As shown in the figure, an embodiment of the present invention provides a surgical process recognition method based on robot-assisted nephrectomy, including the following steps: determining the initial number of video frames; performing initial frame processing on the obtained surgical video of robot-assisted nephrectomy based on the initial number of video frames to obtain a set of surgical images; performing comprehensive image quality detection on the set of surgical images to determine whether the comprehensive image quality is qualified: if the comprehensive image quality is unqualified, screening out the frame images with unqualified quality in the set of surgical images, and performing image enhancement processing on the screened frame images with unqualified quality; if the comprehensive image quality is qualified, obtaining a surgical process feature data set for each frame image in the set of surgical images, where the surgical process feature data set includes instrument shape data, wound surface shape, relative distance data between the instrument center and the center of the frame image, and angle data between the axis of the instrument center and the horizontal line; determining the surgical process corresponding to each frame image based on the surgical process feature data set of each frame image, where the "set of surgical images" refers to the data set of surgical images that has not undergone image enhancement processing.

[0033] As Figure 2 shown, it is the surgical process splitting framework of the robot-assisted partial nephrectomy of the present invention. From Figure 2 this, it can be seen that the robot-assisted partial nephrectomy includes six stages, namely preparing pneumoperitoneum and placing trocars (if any), cleaning retroperitoneal fat, exposing the kidney and tumor, separating the renal hilum, clamping the renal artery, tumor resection, kidney reconstruction, removing the specimen and placing a drainage tube (if any), as Figure 3 shown, it is a simple diagram of the surgical stage provided by the present invention, which introduces the entry and exit points and marked positions of each step. As Figure 4 shown, it is a detailed introduction diagram of the surgical stage provided by the present invention, which details the surgical steps, start, and end features of the surgical stage. As Figure 5 shown, it is a real scene diagram of the surgery provided by the present invention. As Figure 6 shown, it is a surgical marking diagram provided by the present invention. The method of manually annotating the real scene diagram can help doctors obtain more accurate and detailed surgical area segmentation results, including clear boundaries of various tissues and organs such as blood vessels, nerves, and muscles.

[0034] By determining the initial number of frames for the video, the surgical video is decomposed into individual frame images, which lays the foundation for subsequent image analysis and processing. Comprehensive quality detection is performed on each frame image to ensure that the image clarity and quality meet the analysis requirements. The key to this step is to filter or correct images of low quality to avoid affecting the accuracy of feature extraction. Enhance the images of unqualified quality to improve the image quality and ensure that all frame images can clearly display the key information during the surgery. Extract key surgical procedure feature data from the qualified images, such as the shape of the instrument, the shape of the wound surface, the position and angle of the instrument, etc. These data are crucial for understanding and tracking the surgical process. Based on the extracted feature dataset, identify and determine the surgical procedure stage corresponding to each frame image, thereby achieving continuous tracking and analysis of the entire surgical process.

[0035] Among them, the various parameters in the surgical procedure feature dataset can be obtained through the following methods:

[0036] The instrument shape data can use image processing techniques, such as convolutional neural networks (CNNs) in deep learning for image segmentation, to accurately separate surgical instruments from surgical images, and determine the contour of the instrument through edge detection algorithms (such as the Canny edge detection algorithm), thereby obtaining the shape of the instrument.

[0037] The wound surface shape also uses image segmentation techniques to identify and analyze the shape of the surgical wound surface, focusing on identifying and tracking the tissue cutting and modification areas during the surgery, and extracting geometric characteristics of the wound surface area, such as size, shape descriptors, etc.

[0038] The relative distance data between the center of the instrument and the center of the frame image needs to first determine the geometric center of the entire image (the shape of the instrument identified through image segmentation, and calculate the geometric center of the instrument through methods such as the image moment method or the contour method), and use a simple geometric calculation formula to calculate the Euclidean distance from the center of the instrument to the center of the image.

[0039] The angle data between the central axis of the instrument and the horizontal line uses image processing techniques to determine the main axis of the instrument, which can be obtained by fitting the minimum circumscribed rectangle of the instrument shape, or using principal component analysis (PCA) to estimate the direction, and using trigonometric functions to calculate the angle between the main axis of the instrument and the horizontal direction of the image (usually defined as the x-axis of the image).

[0040] The determination of the initial number of frames for the video includes the following steps: Obtain the surgical video of robot-assisted nephrectomy and the brightness data of the resection site during the operation; Determine the duration of the surgical video and the basic process of the resection; Determine the average ambient brightness during the operation based on the brightness data of the resection site during the operation; Determine the initial number of frames for the video based on the duration of the surgical video, the basic process of the resection, and the average ambient brightness during the operation.

[0041] First, obtain the surgical video of robot-assisted nephrectomy and the brightness data of the resection site during the operation. This is the basis of the whole process, as the video and brightness data will directly affect subsequent analysis and processing. Determine the total duration of the operation and the basic process of each stage (the basic process can be determined according to the prior surgical plan). This helps to understand the duration and importance of each stage of the operation and provides a time reference for frame division. According to the collected brightness data (which can be obtained through light sensors), calculate the average brightness of the intraoperative environment. The brightness level affects the clarity and visibility of video frames and is crucial for ensuring image quality.

[0042] Taking the operation duration, basic process, and average brightness as parameters, comprehensively consider and determine the initial number of frames for the video. For example, a darker environment may require a higher frame rate to capture more details, and the critical steps of the operation may require denser frame division to ensure that all important events are recorded.

[0043] By adjusting the frame rate according to the brightness, the image quality can be optimized to ensure that each frame is clear enough for subsequent analysis. By adjusting the number of frames, data overload can be avoided, the data volume can be reasonably controlled, and the storage and processing burdens can be reduced. Precise frame division helps to focus on the critical steps of the operation and improve the efficiency and accuracy of surgical process analysis, especially in automated video analysis and machine learning applications.

[0044] Determining the initial number of frames for the video based on the surgical video duration, the basic process of resection, and the average intraoperative ambient brightness includes the following steps: Compare the basic process of resection with each resection basic process label stored in the database to determine the resection basic process label corresponding to the resection basic process; Obtain the frame number matching data set stored in the database corresponding to the resection basic process label. The frame number matching data set includes multiple frame number matching data, and the frame number matching data includes the matching duration of the surgical video and the matching brightness of the intraoperative average environment; Compare the surgical video duration and the intraoperative average environment brightness with each frame number matching data in the frame number matching data set one by one to obtain the comparison coefficient; Obtain the video initial frame number stored in the database corresponding to the frame number matching data with the smallest comparison coefficient.

[0045] By comparing the basic process of the actual operation with each resection process label pre-stored in the database, determine the process label that best matches the current surgical process. This step ensures that the subsequent selection of the number of frames is closely related to the type of operation. According to the determined basic process label, extract the corresponding frame number matching data set from the database. This data set contains the recommended number of frames in different situations, and each situation is associated with a specific matching duration of the surgical video and the intraoperative average environment brightness.

[0046] Compare the actual surgical video duration and the average intraoperative ambient brightness with the data of each frame number matching data in the dataset, and calculate the comparison coefficient. This coefficient reflects the matching degree between the actual surgical situation and the standard settings recorded in the database. Select the frame number matching data with the smallest comparison coefficient, which represents the best matching result, so as to determine the most suitable initial frame number of the video. This ensures that the selected frame number can most accurately reflect the surgical process, especially the critical surgical stages.

[0047] By optimizing the number of frames, ensure that every detail of the important surgical stages is captured, thereby improving the accuracy and comprehensiveness of the surgical record. An appropriate number of frames helps to avoid too much or too little data, reasonably control the storage requirements and processing burden, and optimize resource utilization. The precisely matched frame settings can improve the efficiency and quality of surgical video analysis, making subsequent automated processing, feature extraction, and learning algorithms more accurate. By matching the surgical situation with the standard data in the database, it supports personalized processing of surgical types while maintaining a certain degree of standardization, which is beneficial to maintaining and improving the quality of medical services.

[0048] The method for obtaining the comparison coefficient is as follows:

[0049]

[0050] In the formula, bx is the comparison coefficient, St is the surgical video duration, Sd is the average intraoperative ambient brightness, a is the number of the frame number matching data, Ct a is the surgical video matching duration of the a-th frame number matching data, Cd a is the average intraoperative ambient matching brightness of the a-th frame number matching data, and e is the natural constant.

[0051] This comparison coefficient calculation formula is mainly used to evaluate the similarity between the actual surgical video duration and the average intraoperative ambient brightness and the preset matching data in the database. The |St - Ct a | calculates the absolute difference between the actual surgical video duration and the surgical video matching duration of a certain matching data stored in the database, which reflects the matching degree in the time dimension. |Sd - Cd a | calculates the absolute difference between the actual average intraoperative ambient brightness and the average intraoperative ambient matching brightness of the corresponding matching data, reflecting the brightness matching degree. The exponential function of the natural constant is used to weight the sum of the above differences. The use of the exponential function enhances the sensitivity to larger differences. Even small differences may cause large changes in the comparison coefficient, thereby more precisely distinguishing different matching data.

[0052] The comparison coefficient in exponential form is very sensitive to differences and can effectively amplify the tiny differences between the actual parameters and the database parameters, helping to more accurately select the number of frames closest to the actual surgical conditions. This formula can ensure that the selected number of frames is most suitable for the current surgical video and ambient brightness conditions, thereby improving the quality of the surgical video and the accuracy of analysis. Through automated comparison and calculation, the most matching parameters can be quickly found from a large amount of pre-stored data, reducing the time and error of manual selection and increasing the efficiency of the entire system. Using an accurate mathematical model to guide the selection of the number of frames improves the data-driven and scientific nature of decision-making, and helps to make more reasonable technical decisions in complex surgical scenarios.

[0053] The comprehensive detection of the image quality of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: obtaining the image detection data of each frame image in the surgical image set, where the image detection data includes the brightness range, image uniformity, and wound surface color accuracy; comparing the image detection data of each frame image with the specified image detection data of the frame images stored in the database to obtain an image quality comparison coefficient, where the specified image detection data includes the specified brightness range, specified image uniformity, and specified wound surface color accuracy; obtaining the mean square deviation of the image quality comparison coefficient; determining whether the mean square deviation is greater than the specified mean square deviation value stored in the database: if the mean square deviation is greater than the specified mean square deviation value stored in the database, the comprehensive image quality is unqualified; if the mean square deviation is not greater than the specified mean square deviation value stored in the database, the comprehensive image quality is qualified.

[0054] The comprehensive image quality detection method is based on the evaluation of the key quality indicators (brightness range, image uniformity, wound surface color accuracy) of each frame image. For each frame image, its brightness range, image uniformity, and wound surface color accuracy are measured and recorded respectively. These indicators reflect the visual quality and technical quality of the image. The detection data of each frame image is compared with the specified image detection data (standard value) preset in the database. This step evaluates the deviation between the actual image data and the expected standard. Calculate the mean square deviation of all frame image quality comparison coefficients. This indicator helps to quantify the consistency of the image quality in the image set and the overall degree of deviation from the standard. If the calculated mean square deviation exceeds the threshold set in the database, the quality of the image set is considered unqualified; if it does not exceed, the image quality is considered qualified. This judgment criterion ensures that the overall quality of the image set meets the expected medical standards.

[0055] The brightness range describes the brightness difference between the darkest and brightest regions in an image. The method to obtain this parameter can be pixel value statistics. By scanning all the pixels in the image, the maximum and minimum brightness values can be directly obtained, thereby determining the brightness range. In a surgical environment, the ability to clearly distinguish tissue boundaries and identify fine structures often depends on appropriate brightness adjustment. An overly wide brightness range may cause overexposure or underexposure of the image, resulting in loss of details; while an overly narrow range may make the image appear dull, lacking sufficient contrast to distinguish different tissues or structures.

[0056] Image uniformity is an indicator that measures the degree of evenness of color distribution in an image. It is usually used to evaluate whether there are overexposed or underexposed regions in the image. By calculating the gray variance, that is, calculating the variance of the gray values of the image, a smaller variance indicates that the image is more uniform. An uneven image may have shadows or bright spots in certain areas, which may mask or create false lesions in surgical images, affecting the doctor's accurate assessment of the surgical area. Optimizing image uniformity ensures that there are no locally overexposed or underexposed regions in the image, making the overall image more realistic and useful.

[0057] The color accuracy of the wound surface refers to the degree of matching between the color representing the wound surface in the image and the actual color of the wound surface. Compare the color output of the imaging device with a known wound surface color standard (which can be represented by the difference in gray values) to evaluate the color accuracy. Color accuracy is crucial for medical images, especially during surgical procedures that require precise color judgment, such as judging the blood supply of tissues or identifying specific types of tissues and organs. Accurate color representation can help doctors make more accurate diagnoses and surgical decisions. Insufficient color accuracy of the wound surface may lead to misdiagnosis or missed important clinical information.

[0058] Ensuring that all images meet certain quality standards provides high-quality data for the detailed recording and subsequent analysis of the surgical process. By comparing with the standard values in the database, the standardized evaluation of image quality is realized, which helps to maintain the consistency and reliability of medical image analysis. Qualified image quality reduces errors and repetitive work in subsequent image processing and analysis, improving the overall processing efficiency.

[0059] The method for obtaining the image quality comparison coefficient is as follows:

[0060]

[0061] In the formula, b is the number of the frame image, and α b is the image quality comparison coefficient of the b-th frame image, SLm v is the maximum brightness of the brightness range of the b-th frame image, SLm b is the minimum brightness of the brightness range of the b-th frame image, Sy bis the image uniformity of the b-th frame image, Smy b is the wound surface color accuracy of the b-th frame image, CLm is the maximum brightness of the brightness reference range, CLm is the minimum brightness of the brightness reference range, Cy is the image reference uniformity, Cmy is the wound surface reference color accuracy, and e is the natural constant.

[0062] In the formula The calculated average deviation of the maximum and minimum brightness of the frame image from the reference brightness reflects whether the image brightness configuration meets the preset standard. By calculating the square root of the difference between the actual image uniformity and the standard uniformity, it is measured whether the uniformity of the image meets the requirements. Using the square root can reduce the sensitivity of the deviation and make the evaluation smoother. By calculating the exponent of the deviation of the wound surface color accuracy, the importance of color accuracy in image quality is emphasized. The exponential function can amplify the color deviation, so that even small color errors are significantly reflected in the comparison coefficient.

[0063] Brightness directly affects color performance. Improper brightness settings may lead to color distortion, affecting the doctor's judgment of the wound surface state. The uneven distribution of brightness may cause the loss of details in some areas of the image, thereby affecting the overall image quality evaluation. The unevenness of color may indicate problems with the image capture or display device, or may be due to improper parameter settings during the image processing process.

[0064] By independent evaluation and synthesis of different dimensions (brightness, uniformity, color accuracy), specific problems of image quality can be more accurately located, providing a direction for further image optimization. By direct comparison with the preset standard, the standardization of image quality control is achieved, ensuring that all surgical images meet certain quality standards.

[0065] Determining the surgical procedure corresponding to each frame image based on the surgical procedure feature dataset of each frame image includes the following steps: obtaining the allowable error data of the surgical procedure features of each frame image in the surgical image set, where the allowable error data of the surgical procedure features includes the allowable error of instrument shape matching, the allowable error of wound surface shape matching, the allowable error of the relative distance matching between the instrument center and the frame image center, and the allowable error of the angle matching between the instrument center axis and the horizontal line; obtaining the surgical procedure feature matching datasets stored in the database, where the surgical procedure feature matching datasets include instrument shape matching data, wound surface matching shapes, relative distance matching data between the instrument center and the frame image center, and angle matching data between the instrument center axis and the horizontal line; comprehensively analyzing the allowable error data of the surgical procedure features of each frame image, each surgical procedure feature matching dataset, and the surgical procedure feature dataset of each frame image to obtain the process matching coefficient of each frame image; obtaining the minimum process matching coefficient corresponding to each frame image; determining the surgical procedure feature matching dataset corresponding to each minimum process matching coefficient and storing it as the surgical procedure corresponding to this frame image in the surgical procedure label in the database.

[0066] Collect the allowable error ranges of the instrument shape, wound surface shape, relative position of the instrument and the image center, and the angle between the instrument center axis and the horizontal line in each frame image. These error data are preset according to the precise requirements of the surgery and the possible deviations in actual operation. Extract the pre-defined feature matching datasets corresponding to each surgical procedure from the database. These datasets define the ideal states of the key features such as the instrument shape and position expected in each surgical stage. Compare the actual surgical procedure features of each frame image with the matching datasets in the database, and calculate the process matching coefficient according to the preset allowable error range. This coefficient measures the degree of closeness between the actual image and the expected state. For each frame image, select the surgical procedure with the minimum matching coefficient as the marked procedure for this frame. This means that this procedure has the smallest difference from the actual image among all possible surgical stages.

[0067] The method for obtaining the process matching coefficient is as follows:

[0068]

[0069] In the formula, b is the number of the frame image, and β b is the process matching coefficient of the b-th frame image, A b is the instrument shape data of the b-th frame image, B b is the wound surface shape of the b-th frame image, C b is the relative distance data between the instrument center and the frame image center of the b-th frame image, D bThe angle data between the instrument central axis of the v-th frame image and the horizontal line, c is the number of the surgical procedure feature matching data set, A c is the instrument shape matching data of the c-th surgical procedure feature matching data set, B c is the wound surface matching shape of the c-th surgical procedure feature matching data set, C c is the instrument center of the c-th surgical procedure feature matching data set, D c is the relative distance matching data and the angle matching data between the instrument central axis and the horizontal line of the frame image center of the c-th surgical procedure feature matching data set, σ(A b , A c ) is the similarity function of A b and A c , σ(B b , B c ) is the similarity function of B b and B c , σ(C b , C c ) is the similarity function of C b and C c , σ(D b , D c ) is the similarity function of D b and D c . The similarity function includes but is not limited to the Euclidean distance, Manhattan distance, etc.

[0070] For each feature (instrument shape, wound surface shape, relative distance, angle position), calculate the similarity between the current frame image and the surgical procedure feature matching data set stored in the database. The summation part in the formula accumulates the similarities of all features to obtain a comprehensive similarity score. Each feature contributes an equal weight. Use the hyperbolic tangent function tanh to normalize the total similarity score, restricting the matching coefficient between 0 and 1. This transformation helps to handle extreme values and makes the matching coefficient more stable and interpretable.

[0071] The shape of the instrument determines its function and use. Different surgical stages may require instruments of different shapes. The instrument shape affects the operation method and the specific implementation of the surgery. Correctly identifying the shape is crucial for determining the surgical procedure. The shape of the wound surface can reflect the progress and current stage of the surgery. For example, the size and shape of the incision may indicate the specific steps of the surgery. The morphological changes of the wound surface are directly related to the success or failure of the surgery and are an intuitive indicator for evaluating the surgical progress. The relative distance shows the positional relationship between the instrument and the operation field (i.e., the center of the frame image), which is necessary for ensuring surgical accuracy and avoiding misoperation. Precise distance control can reduce surgical risks and improve the accuracy of the operation. The included angle position indicates the operation angle of the instrument. For some delicate operations, such as suturing or cutting, the correct angle is the key to success. The precise adjustment of the instrument angle directly affects the effect and safety of the surgical cutting or treatment.

[0072] These four features act together in every specific step of the surgery and are highly correlated with each other. The shape of the instrument is closely related to the used angle and position. Instruments of different shapes require specific angle and position adjustments when used at specific surgical sites. The change in the shape of the wound surface is often caused by the instrument, and vice versa. The operation method of the instrument (including shape and angle) needs to be adjusted according to the condition of the wound surface. The comprehensive consideration of these four features makes the determination of the surgical procedure more comprehensive and accurate.

[0073] The surgical procedure feature allowable error data for obtaining each frame image in the surgical image set includes the following steps: obtaining the time data corresponding to each frame image; obtaining the brightness data of the resection site during the operation; obtaining the brightness of the resection site corresponding to each frame image from the brightness data of the resection site during the operation based on the time data corresponding to each frame image; obtaining the image quality comparison coefficient corresponding to each frame image; combining the brightness of the resection site corresponding to each frame image with the image quality comparison coefficient into an error matching data group; comparing the error matching data group with each matching data group stored in the database one by one to obtain an error matching coefficient, where the matching data group includes the matching brightness of the resection site and the image quality matching comparison coefficient; obtaining the surgical procedure feature allowable error data of the frame image corresponding to the matching data group with the smallest error matching coefficient stored in the database.

[0074] The logic of this process is to accurately match and determine the surgical procedure characteristics and allowable errors of each frame image by analyzing in detail the brightness data of the resection site in the frame image and the image quality comparison coefficient. Each frame image has a corresponding timestamp. Based on this time data, the brightness information of the corresponding frame image is extracted from the brightness data of the intraoperative resection site, the quality of each frame image is analyzed, and an image quality comparison coefficient is calculated. This coefficient reflects the degree of compliance of the image with the preset quality standard. The brightness of the resection site and the image quality comparison coefficient are combined into an error matching data group to establish a comprehensive feature description including brightness and quality for each frame image. The error matching data group is compared with the matching data groups stored in the database, and the matching item with the smallest difference is found by comparison. The matching data group with the smallest error matching coefficient is selected, and the allowable error data of the surgical procedure characteristics in this data group is determined to provide the surgical team with the required brightness and image quality standards for the current frame image.

[0075] The calculation formula for the error matching coefficient is as follows:

[0076]

[0077] In the formula, b is the number of the frame image, γ b is the image quality comparison coefficient corresponding to the b-th frame image, α b is the brightness of the resection site corresponding to the b-th frame image, SL b is the image quality comparison coefficient corresponding to the b-th frame image, j is the number of the matching data group, α j is the image quality matching comparison coefficient in the j-th matching data group, CL j is the matching brightness of the resection site in the j-th matching data group.

[0078] |α b - α j | calculates the absolute difference between the brightness of the resection site of the frame image and the preset brightness standard in the matching data group. |SL b - CL j | calculates the absolute difference between the image quality comparison coefficient of the frame image and the image quality standard in the matching data group. Adding the above two difference values, this summation reflects the overall deviation degree of the image from the standard in the two dimensions of brightness and image quality. The square root processing and natural logarithm transformation are performed on the weighted sum of the total differences. Such a transformation helps to stabilize the influence of extreme values and ensure that the growth of the error matching coefficient is smoother, which is especially useful when dealing with data with extremely large differences.

[0079] Using square root and logarithmic transformations helps to mitigate the impact of extreme difference values on the final result, making the result more robust. By combining the differences in brightness and image quality, the formula can provide a more comprehensive error assessment, which helps to comprehensively understand the degree of deviation of the image from the preset standard.

[0080] The image quality comparison coefficient is usually a comprehensive index calculated based on multiple image attributes (such as sharpness, contrast, noise level, etc.), which reflects the overall quality and applicability of the image. The brightness of the resection site refers to the illumination level of the surgical area, which is crucial for the visibility and sharpness of the image. The overall quality of the image is directly affected by the brightness level. Inappropriate brightness can lead to overexposure or underexposure of the image, affecting the image quality comparison coefficient. For example, low brightness may cause loss of image details, while high brightness may cause overexposure of the image and the details are "washed out". These two factors together determine whether the image is clear enough, thus affecting the execution of the surgery. In summary, the image quality comparison coefficient and the brightness of the resection site are key factors for evaluating the quality of surgical images. They jointly support the visual needs of the surgical process and have a direct impact on improving the surgical precision and success rate.

[0081] The following points need to be explained:

[0082] (1) The accompanying drawings of the embodiments of the present invention only relate to the structures involved in the embodiments of the present invention, and other structures can refer to the general design.

[0083] (2) For clarity, in the accompanying drawings used to describe the embodiments of the present invention, the thickness of the layer or region is enlarged or reduced, that is, these drawings are not drawn according to the actual scale. It can be understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element can be "directly" on or under the other element or there can be intermediate elements.

[0084] (3) Without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0085] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A surgical procedure recognition method based on robot-assisted nephrectomy, characterized in that: The following steps are involved: Determine the number of initial video frames; Performing initial frame segmentation processing on the acquired robot-assisted nephrectomy surgical video based on the number of initial video segmentation frames to obtain a surgical image set; Perform comprehensive image quality testing on the surgical image set to determine whether the comprehensive image quality is qualified: If the overall image quality is unqualified, the frame images with unqualified quality in the surgical image set are screened, and image enhancement processing is performed on the screened frame images with unqualified quality; If the comprehensive image quality is qualified, then obtain the surgical process feature data set of each frame image in the surgical image set, wherein the surgical process feature data set includes instrument shape data, wound surface shape, relative distance data between the instrument center and the frame image center, and angle data between the instrument center axis and the horizontal line; Determine the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image; The step of determining the number of initial video frames comprises the following steps: To obtain surgical videos and intraoperative brightness data of the resection site during robot-assisted nephrectomy; Determine the duration of the surgical video and the basic process of resection; Determine the average intraoperative ambient brightness based on the intraoperative resection site brightness data; The number of initial video frames is determined based on the duration of the surgical video, the basic process of resection, and the average ambient brightness during the operation.

2. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 1, characterized in that: The method of determining the number of initial video frames based on the duration of the surgical video, the basic process of the resection, and the average ambient brightness during the operation includes the following steps: Comparing the basic resection procedure with each basic resection procedure label stored in the database to determine the basic resection procedure label corresponding to the basic resection procedure; Acquire a frame quantity matching data set corresponding to a basic process label of resection surgery stored in a database, wherein the frame quantity matching data set includes a plurality of frame quantity matching data, and the frame quantity matching data includes a surgical video matching duration and an average environmental matching brightness during surgery; Compare the duration of the surgical video, the average ambient brightness during the operation, and each frame number matching data in the frame number matching data set one by one to obtain the comparison coefficient; The number of sub-frames corresponding to the minimum comparison coefficient is obtained, and the matching data corresponds to the number of initial sub-frames of the video stored in the database.

3. The surgical procedure identification method based on robot-assisted nephrectomy according to claim 2, characterized in that: The method for obtaining the comparison coefficient is as follows: Where bx is the comparison coefficient, St is the duration of the surgical video, Sd is the average ambient brightness during the operation, a is the number of frames matching the data, and Ct a Cd is the duration of the surgical video matching data for the ath frame number, a is the intraoperative average environment matching brightness of the a-th frame matching data, and e is a natural constant.

4. The surgical procedure identification method based on robot-assisted nephrectomy according to claim 1, characterized in that: The comprehensive image quality detection of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: Acquire image detection data of each frame image in the surgical image set, wherein the image detection data includes brightness range, image uniformity, and wound surface color accuracy; The image detection data of each frame image is compared with the reference image detection data of the frame image stored in the database to obtain an image quality comparison coefficient, wherein the reference image detection data includes a brightness reference range, image reference uniformity, and wound surface reference color accuracy; The image quality comparison coefficient is used to determine whether the overall image quality is qualified.

5. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 4, characterized in that: The method of judging whether the comprehensive image quality is qualified based on the image quality comparison coefficient comprises the following steps: Obtain the mean square error of the image quality comparison coefficient; Determine whether the mean square error is greater than the mean square error parameter value stored in the database: If the mean square error is greater than the mean square error parameter value stored in the database, the overall image quality is unqualified; If the mean square error is not greater than the mean square error parameter value stored in the database, the overall image quality is qualified.

6. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 5, characterized in that: The step of determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image comprises the following steps: Acquire surgical process feature allowable error data of each frame image in the surgical image set; Acquire each surgical procedure feature matching data set stored in the database; Comprehensively analyzing the surgical process feature allowable error data of each frame image, each surgical process feature matching data set, and the surgical process feature data set of each frame image to obtain the process matching coefficient of each frame image; Obtain the minimum process matching coefficient corresponding to each frame image; The surgical process feature matching data sets corresponding to each minimum process matching coefficient correspond to the surgical process label stored in the database and are determined as the surgical process corresponding to the frame image.

7. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 6, characterized in that: The surgical process feature allowable error data includes the instrument shape matching allowable error, the wound surface shape matching allowable error, the relative distance matching allowable error between the instrument center and the frame image center, and the angle matching allowable error between the instrument center axis and the horizontal line. The surgical process feature matching data set includes instrument shape matching data, wound surface matching shape, relative distance matching data between the instrument center and the frame image center, and angle matching data between the instrument center axis and the horizontal line.

8. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 6, characterized in that: The step of obtaining the surgical process feature allowable error data of each frame image in the surgical image set comprises the following steps: Acquire the brightness of the resection site corresponding to each frame image; Obtaining image quality comparison coefficients corresponding to each frame image; Combining the brightness of the resection site and the image quality comparison coefficient corresponding to each frame image into an error matching data set; Comparing the error matching data set with each matching data set stored in the database one by one to obtain an error matching coefficient, wherein the matching data set includes a resection site matching brightness and an image quality matching comparison coefficient; The matching data group corresponding to the minimum error matching coefficient is obtained, which corresponds to the surgical procedure feature allowable error data of the frame image stored in the database.

9. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 8, characterized in that: The step of obtaining the brightness of the resection site corresponding to each frame image comprises the following steps: Obtaining time data corresponding to each frame image; Acquire intraoperative resection site brightness data; The resection site brightness corresponding to each frame image is acquired from the resection site brightness data during the operation based on the time data corresponding to each frame image.

Images