Surgical procedure identification method based on robot-assisted renal site resection

Abstract

The invention provides a surgical procedure identification method based on robot-assisted renal part resection. The method comprises the following steps: determining the number of initial frames of a video; based on the video initial framing number, performing initial framing processing on the obtained operation video of the robot-assisted renal region resection operation to obtain an operation image set; image quality comprehensive detection is carried out on the operation image set, whether the image comprehensive quality is qualified or not is judged, if the image comprehensive quality is not qualified, unqualified frame images in the operation image set are screened, and image enhancement processing is carried out on the screened unqualified frame images; if the comprehensive quality of the image is qualified, acquiring an operation process feature data set of each frame image in the operation image set; and determining the operation process corresponding to each frame image based on the operation process feature data set of each frame image. The problem that the operation process is difficult to identify and track in the existing complex operation video analysis is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical procedure recognition, and in particular to a surgical procedure recognition method based on robot-assisted nephrectomy. Background Art

[0002] Surgical process recognition includes segmenting and identifying surgical stages of surgical videos. Surgical process recognition is of great significance to endoscopic surgery. During the operation, automatic recognition of surgical process can remind doctors to pay attention to the occurrence of complications, reduce doctors' operational errors during the operation, and provide relevant information on the progress of the operation to clinical staff outside the operating room. Subsequently, through intelligent segmentation of expert surgical videos, it can reduce doctors' workload, build a surgical video learning resource library, provide guidance for junior clinical physicians, and also serve as a reference for the arrangement of subsequent related work of the operation.

[0003] The Chinese invention patent with announcement number CN112818959B discloses a surgical process identification method, device, system and computer-readable storage medium, which belongs to the field of computer technology. In the embodiment of the present application, the surgical stage is divided into multiple main stages, and there is a main stage including sub-stages. Based on this, the surgical stage that has been experienced before the first image in the surgical video is first referred to to determine the main surgical stage to which the first image belongs. If the identified main surgical stage is further subdivided into sub-stages, the surgical sub-stage to which the first image belongs is determined by referring to the instruments present in the first image. It can be seen that this scheme combines coarse recognition and fine recognition, and is beneficial to the respective characteristics of coarse recognition and fine recognition, thereby effectively improving the precision and accuracy of surgical process identification. If surgical process identification is performed during the operation, this scheme can provide accurate information for the operation, effectively reduce surgical errors, and improve the success rate of the operation.

[0004] However, existing surgical process recognition methods have difficulty in identifying based on multiple features, resulting in difficulty in accurately identifying and tracking surgical processes in complex surgical video analysis. Summary of the invention

[0005] The present invention provides a surgical procedure recognition method based on robot-assisted nephrectomy, which solves the existing problem that it is difficult to accurately identify and track the surgical procedure in complex surgical video analysis.

[0006] In order to solve the above-mentioned invention object, the present invention provides the following technical solutions:

[0007] A method for identifying a surgical procedure based on robot-assisted nephrectomy comprises the following steps: determining the number of initial video frame divisions; performing initial frame division processing on an acquired surgical video of robot-assisted nephrectomy based on the number of initial video frame divisions to obtain a surgical image set; performing comprehensive image quality detection on the surgical image set to determine whether the comprehensive image quality is qualified: if the comprehensive image quality is unqualified, screening out frame images with unqualified quality in the surgical image set, and performing image enhancement processing on the screened out frame images with unqualified quality; if the comprehensive image quality is qualified, obtaining a surgical procedure feature data set of each frame image in the surgical image set, the surgical procedure feature data set comprising instrument shape data, wound surface shape, relative distance data between the instrument center and the frame image center, and angle data between the instrument center axis and the horizontal line; determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image.

[0008] Optionally, the method of determining the initial number of video frames includes the following steps: obtaining a surgical video of robot-assisted nephrectomy and intraoperative resection site brightness data; determining the duration of the surgical video and the basic process of resection; determining the intraoperative average ambient brightness based on the intraoperative resection site brightness data; determining the initial number of video frames based on the duration of the surgical video, the basic process of resection and the intraoperative average ambient brightness.

[0009] Optionally, the method of determining the initial number of video frames based on the duration of the surgical video, the basic process of resection and the average ambient brightness during the operation includes the following steps: comparing the basic process of resection with each basic process label of resection stored in a database to determine the basic process label of resection corresponding to the basic process of resection; obtaining a frame number matching data set corresponding to the basic process label of resection stored in the database, wherein the frame number matching data set includes multiple frame number matching data, and the frame number matching data includes the matching duration of the surgical video and the average ambient brightness during the operation; comparing the duration of the surgical video, the average ambient brightness during the operation and each frame number matching data in the frame number matching data set one by one to obtain a comparison coefficient; obtaining the frame number matching data corresponding to the minimum comparison coefficient corresponds to the initial number of video frames stored in the database.

[0010] Optionally, the comparison coefficient is obtained as follows:

[0011]

[0012] Where bx is the comparison coefficient, St is the duration of the surgical video, Sd is the average ambient brightness during the operation, a is the number of frames matching the data, and Ct a Cd is the duration of the surgical video matching data for the ath frame number, ais the intraoperative average environment matching brightness of the a-th frame matching data, and e is a natural constant.

[0013] Optionally, the comprehensive image quality detection of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: obtaining image detection data of each frame image in the surgical image set, the image detection data including brightness range, image uniformity and wound color accuracy; comparing the image detection data of each frame image with the parameter image detection data of the frame image stored in the database to obtain an image quality comparison coefficient, the parameter image detection data including brightness parameter range, image parameter uniformity and wound color accuracy; and determining whether the comprehensive image quality is qualified based on the image quality comparison coefficient.

[0014] Optionally, judging whether the comprehensive image quality is qualified based on the image quality comparison coefficient includes the following steps: obtaining the mean square error of the image quality comparison coefficient; judging whether the mean square error is greater than the mean square error parameter value stored in the database: if the mean square error is greater than the mean square error parameter value stored in the database, the comprehensive image quality is unqualified; if the mean square error is not greater than the mean square error parameter value stored in the database.

[0015] Optionally, the method of determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image includes the following steps: obtaining surgical procedure feature allowable error data of each frame image in the surgical image set; obtaining each surgical procedure feature matching data set stored in a database; comprehensively analyzing the surgical procedure feature allowable error data of each frame image, each surgical procedure feature matching data set and the surgical procedure feature data set of each frame image to obtain a process matching coefficient of each frame image; obtaining the minimum process matching coefficient corresponding to each frame image; and determining the surgical procedure corresponding to the frame image by corresponding to the surgical procedure label stored in the database to the surgical procedure feature matching data set corresponding to each minimum process matching coefficient.

[0016] Optionally, the surgical process feature allowable error data includes the instrument shape matching allowable error, the wound surface shape matching allowable error, the relative distance matching allowable error between the instrument center and the frame image center, and the angle matching allowable error between the instrument center axis and the horizontal line, and the surgical process feature matching data set includes instrument shape matching data, wound surface matching shape, relative distance matching data between the instrument center and the frame image center, and angle matching data between the instrument center axis and the horizontal line.

[0017] Optionally, the method of obtaining the allowable error data of surgical process characteristics of each frame image in the surgical image set includes the following steps: obtaining the brightness of the resection site corresponding to each frame image; obtaining the image quality comparison coefficient corresponding to each frame image; combining the brightness of the resection site and the image quality comparison coefficient corresponding to each frame image into an error matching data group; comparing the error matching data group with each matching data group stored in the database one by one to obtain the error matching coefficient, wherein the matching data group includes the matching brightness of the resection site and the matching comparison coefficient of the image quality; obtaining the allowable error data of surgical process characteristics of the frame image stored in the database corresponding to the matching data group corresponding to the minimum error matching coefficient.

[0018] Optionally, the method of obtaining the brightness of the resection site corresponding to each frame image includes the following steps: obtaining time data corresponding to each frame image; obtaining brightness data of the resection site during the operation; and obtaining the brightness of the resection site corresponding to each frame image from the brightness data of the resection site during the operation based on the time data corresponding to each frame image.

[0019] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0020] The above solution is based on the surgical process recognition method for robot-assisted nephrectomy. It performs initial frame processing and comprehensive image quality detection on the surgical video to ensure that the image quality meets the standard, and uses image enhancement processing to improve the frame images with unqualified quality, so as to accurately extract the surgical process feature data set. This makes the analysis of surgical videos more accurate, helps doctors better understand the surgical process, improves the safety and efficiency of surgery, and solves the problem of difficulty in identifying and tracking surgical processes in the existing complex surgical video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a surgical procedure identification method based on robot-assisted nephrectomy according to the present invention;

[0023] Figure 2 The robot-assisted partial nephrectomy surgical procedure splitting framework of the present invention;

[0024] Figure 3 A simplified diagram of the surgical stages provided by the present invention;

[0025] Figure 4A detailed diagram of the surgical stages provided by the present invention;

[0026] Figure 5 A real-life picture of the surgery provided by the present invention;

[0027] Figure 6 This is a surgical marking diagram provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0030] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0031] In view of the problem that it is difficult to identify and track the surgical process in the existing complex surgical video analysis, the present invention provides a surgical process identification method based on robot-assisted nephrectomy. The surgical video is subjected to initial frame processing and comprehensive image quality detection to ensure that the image quality meets the standard, and the frame images with unqualified quality are improved by image enhancement processing, so as to accurately extract the surgical process feature data set. This makes the analysis of surgical videos more accurate, helps doctors better understand the surgical process, and improves the safety and efficiency of surgery.

[0032] like Figure 1As shown, an embodiment of the present invention provides a surgical process identification method based on robot-assisted nephrectomy, comprising the following steps: determining the number of initial video frame divisions; performing initial frame division processing on the acquired robot-assisted nephrectomy surgical video based on the number of initial video frame divisions to obtain a surgical image set; performing comprehensive image quality detection on the surgical image set to determine whether the comprehensive image quality is qualified: if the comprehensive image quality is unqualified, screening out frame images with unqualified quality in the surgical image set, and performing image enhancement processing on the screened out frame images with unqualified quality; if the comprehensive image quality is qualified, obtaining a surgical process feature data set of each frame image in the surgical image set, the surgical process feature data set including instrument shape data, wound surface shape, relative distance data between the center of the instrument and the center of the frame image, and angle data between the center axis of the instrument and the horizontal line; determining the surgical process corresponding to each frame image based on the surgical process feature data set of each frame image, wherein the "surgical image set" refers to a surgical image data set that has not been subjected to image enhancement processing.

[0033] like Figure 2 As shown in FIG. 1 , the robot-assisted partial nephrectomy surgical procedure splitting framework of the present invention is shown. Figure 2 It is known that robot-assisted partial nephrectomy includes six stages, namely, preparing pneumoperitoneum and placing cannula (if any), cleaning retroperitoneal fat, exposing kidney and tumor, separating renal hilum, clamping renal artery, tumor removal, kidney reconstruction, removing specimen and placing drainage tube (if any), Figure 3 As shown in FIG. 1 , a simplified diagram of the surgical stages provided by the present invention is provided, wherein the entry and exit points and marking points of each step are introduced. Figure 4 As shown in FIG. 1 , a detailed description of the surgical stages provided by the present invention is provided, wherein the surgical steps, the start and end features of the surgical stages are described in detail. Figure 5 As shown in FIG. 1 , it is a real-life operation diagram provided by the present invention. Figure 6 The figure shows the surgical marking diagram provided by the present invention. Manually marking the real scene diagram can help doctors obtain more accurate and detailed surgical area segmentation results, including clear boundaries of various tissues and organs such as blood vessels, nerves, and muscles.

[0034] By determining the initial number of video frames, the surgical video is decomposed into separate frame images, which lays the foundation for subsequent image analysis and processing. Comprehensive quality detection is performed on each frame image to ensure that the image clarity and quality meet the analysis requirements. The key to this step is to filter or correct low-quality images to avoid affecting the accuracy of feature extraction. Enhancement processing is performed on unqualified images to improve image quality and ensure that all frame images can clearly display key information during the operation. Extract key surgical process feature data from qualified images, such as instrument shape, wound shape, instrument position and angle, etc., which are crucial for understanding and tracking the surgical process. Based on the extracted feature data set, identify and determine the surgical process stage corresponding to each frame image, so as to achieve continuous tracking and analysis of the entire surgical process.

[0035] The various parameters in the surgical process feature data set can be obtained in the following ways:

[0036] The instrument shape data can be processed using image processing techniques, such as convolutional neural networks (CNN) in deep learning for image segmentation, to accurately separate surgical instruments from surgical images, and to determine the contours of the instruments through edge detection algorithms (such as the Canny edge detection algorithm) to obtain the shape of the instruments.

[0037] The wound shape also uses image segmentation technology to identify and analyze the shape of the surgical wound, focusing on identifying and tracking the tissue cutting and modification areas during the operation, and extracting the geometric characteristics of the wound area, such as size and shape descriptors.

[0038] The relative distance data between the center of the device and the center of the frame image requires first determining the geometric center of the entire image (the shape of the device is identified through image segmentation, and the geometric center of the device is calculated by image moment method or contour method, etc.), and using a simple geometric calculation formula to calculate the Euclidean distance from the center of the device to the center of the image.

[0039] The angle data between the central axis of the device and the horizontal line uses image processing technology to determine the main axis of the device. It can be obtained by fitting the minimum circumscribed rectangle of the device shape, or by using principal component analysis (PCA) to estimate the direction and using trigonometric functions to calculate the angle between the main axis of the device and the horizontal direction of the image (usually defined as the x-axis of the image).

[0040] The method of determining the initial number of video frames comprises the following steps: obtaining a surgical video of robot-assisted nephrectomy and intraoperative resection site brightness data; determining the duration of the surgical video and the basic procedure of the resection; determining the intraoperative average ambient brightness based on the intraoperative resection site brightness data; and determining the initial number of video frames based on the duration of the surgical video, the basic procedure of the resection, and the intraoperative average ambient brightness.

[0041] First, obtain the surgical video of the robot-assisted nephrectomy and the brightness data of the resection site during the operation. This is the basis of the entire process, because the video and brightness data will directly affect the subsequent analysis and processing. Determine the total duration of the operation and the basic flow of each stage (the basic flow can be determined according to the prior surgical plan). This helps to understand the duration and importance of each stage of the operation and provide a time reference for frame division. Based on the collected brightness data (which can be obtained through light sensors), calculate the average brightness of the intraoperative environment. Brightness levels affect the clarity and visibility of video frames and are critical to ensuring image quality.

[0042] The duration of the surgery, basic procedures, and average brightness are used as parameters to determine the initial number of video frames. For example, a darker environment may require a higher frame rate to capture more details, while key steps of the surgery may require more dense frames to ensure that all important events are recorded.

[0043] By adjusting the frame rate according to brightness, the image quality can be optimized to ensure that each frame is clear enough for subsequent analysis. By adjusting the number of frames, data overload can be avoided, the amount of data can be reasonably controlled, and the burden of storage and processing can be reduced. Accurate framing helps focus on key surgical steps and improves the efficiency and accuracy of surgical process analysis, especially in automated video analysis and machine learning applications.

[0044] The method of determining the initial number of video frames based on the duration of the surgical video, the basic process of resection and the average ambient brightness during the operation includes the following steps: comparing the basic process of resection with each basic process label of resection stored in a database to determine the basic process label of resection corresponding to the basic process of resection; obtaining a frame number matching data set corresponding to the basic process label of resection stored in the database, wherein the frame number matching data set includes multiple frame number matching data, and the frame number matching data includes the matching duration of the surgical video and the average ambient brightness during the operation; comparing the surgical video duration, the average ambient brightness during the operation with each frame number matching data in the frame number matching data set one by one to obtain a comparison coefficient; obtaining the frame number matching data corresponding to the minimum comparison coefficient corresponding to the initial number of video frames stored in the database.

[0045] By comparing the basic process of the actual surgery with the pre-stored labels of each resection process in the database, the process label that best matches the current surgical process is determined. This step ensures that the subsequent selection of the number of sub-frames is closely related to the type of surgery. According to the determined basic process labels, the corresponding sub-frame number matching dataset is extracted from the database. This dataset contains the recommended number of sub-frames in different situations, each of which is associated with a specific surgical video matching duration and the average ambient brightness during the operation.

[0046] The actual duration of the surgical video and the average ambient brightness during the operation are compared with each frame number matching data in the dataset to calculate the comparison coefficient. This coefficient reflects the degree of match between the actual surgical situation and the standard settings recorded in the database. The frame number matching data with the smallest comparison coefficient is selected, which represents the best matching result, to determine the most suitable initial video frame number. This ensures that the selected frame number can most accurately reflect the surgical process, especially the critical surgical stage.

[0047] By optimizing the number of frames, we can ensure that every detail of the important surgical stage is captured, thereby improving the accuracy and thoroughness of surgical records. The appropriate number of frames helps to avoid too much or too little data, reasonably control storage requirements and processing burden, and optimize resource use. Accurately matched frame settings can improve the efficiency and quality of surgical video analysis, making subsequent automated processing, feature extraction, and learning algorithms more accurate. By matching surgical conditions with standard data in the database, it supports personalized processing of surgical types while maintaining a certain degree of standardization, which is conducive to maintaining and improving the quality of medical services.

[0048] The method for obtaining the comparison coefficient is as follows:

[0049]

[0050] Where bx is the comparison coefficient, St is the duration of the surgical video, Sd is the average ambient brightness during the operation, a is the number of frames matching the data, and Ct a Cd is the duration of the surgical video matching data for the ath frame number, a is the intraoperative average environment matching brightness of the a-th frame matching data, and e is a natural constant.

[0051] The comparison coefficient calculation formula is mainly used to evaluate the similarity between the actual surgical video duration and the average ambient brightness during surgery and the preset matching data in the database. a |The absolute difference between the actual duration of the surgical video and the duration of the surgical video matching data stored in the database is calculated, which reflects the degree of matching in the time dimension. |Sd-Cd a |The absolute difference between the actual intraoperative average ambient brightness and the intraoperative average ambient matching brightness of the corresponding matching data is calculated, reflecting the degree of brightness matching. The sum of the above differences is weighted using an exponential function of a natural constant. The use of an exponential function enhances the sensitivity to large differences, and even small differences may lead to large changes in the comparison coefficient, thereby more accurately distinguishing different matching data.

[0052] The comparison coefficient in exponential form is very sensitive to differences and can effectively amplify small differences between actual parameters and database parameters, helping to more accurately select the number of frames that are closest to the actual surgical conditions. This formula can ensure that the selected number of frames is most suitable for the current surgical video and environmental brightness conditions, thereby improving the quality of the surgical video and the accuracy of the analysis. Through automated comparison and calculation, the most matching parameters can be quickly found from a large amount of pre-stored data, reducing the time and error of manual selection and increasing the efficiency of the entire system. Using precise mathematical models to guide the selection of the number of frames improves the data-driven and scientific nature of decision-making, and helps make more reasonable technical decisions in complex surgical scenarios.

[0053] The comprehensive image quality detection of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: obtaining image detection data of each frame image in the surgical image set, the image detection data including brightness range, image uniformity and wound color accuracy; comparing the image detection data of each frame image with the reference image detection data of the frame image stored in a database to obtain an image quality comparison coefficient, the reference image detection data including brightness reference range, image reference uniformity and wound color accuracy; obtaining the mean square error of the image quality comparison coefficient; determining whether the mean square error is greater than the mean square error reference value stored in the database: if the mean square error is greater than the mean square error reference value stored in the database, the comprehensive image quality is unqualified; if the mean square error is not greater than the mean square error reference value stored in the database, the comprehensive image quality is qualified.

[0054] The comprehensive image quality detection method is based on the evaluation of key quality indicators (brightness range, image uniformity, and wound color accuracy) for each frame of the image. For each frame of the image, its brightness range, image uniformity, and wound color accuracy are measured and recorded respectively. These indicators reflect the visual quality and technical quality of the image. The detection data of each frame of the image is compared with the preset reference image detection data (standard value) in the database. This step evaluates the deviation between the actual image data and the expected standard. The mean square error of the image quality comparison coefficients of all frames is calculated. This indicator helps quantify the consistency of image quality in the image set and the degree of overall deviation from the standard. If the calculated mean square error exceeds the threshold set in the database, the quality of the image set is considered to be unqualified; if it does not exceed, the image quality is considered to be qualified. This judgment standard ensures that the overall quality of the image set meets the expected medical standards.

[0055] The brightness range describes the brightness difference between the darkest and brightest areas in the image. This parameter can be obtained by pixel value statistics. By scanning all pixels in the image, the maximum and minimum brightness values ​​can be directly obtained to determine the brightness range. In the surgical environment, the ability to clearly distinguish tissue boundaries and identify small structures often depends on proper brightness adjustment. Too wide a brightness range may cause the image to be overexposed or underexposed, with loss of details; too narrow a range may make the image appear bland and lack sufficient contrast to distinguish different tissues or structures.

[0056] Image uniformity is an indicator that measures the degree of color balance in an image. It is usually used to evaluate whether there are overexposed or underexposed areas in the image. The variance of the grayscale values ​​of the image is calculated through grayscale variance calculation. A smaller variance indicates a more uniform image. Uneven images may have shadows or bright spots in certain areas, which may mask or false lesions in surgical images and affect the doctor's accurate assessment of the surgical area. The optimization of image uniformity ensures that there are no local overexposed or underexposed areas in the image, making the overall image more realistic and useful.

[0057] Wound color accuracy refers to the degree of match between the color of the wound surface represented in the image and the actual wound surface color. The color output of the camera device is compared with the known wound surface color standard (which can be represented by the difference in grayscale values) to evaluate color accuracy. Color accuracy is critical for medical images, especially in surgical procedures that require precise color judgment, such as determining the blood supply of tissues or identifying specific types of tissues and organs. Accurate expression of color can help doctors make more accurate diagnoses and surgical decisions. Insufficient accuracy of wound surface color may lead to misdiagnosis or missing important clinical information.

[0058] Ensure that all images meet certain quality standards, provide high-quality data for detailed records of the surgical process and subsequent analysis, and achieve standardized assessment of image quality by comparing with standard values ​​in the database, which helps maintain the consistency and reliability of medical image analysis. Qualified image quality reduces errors and duplication of work in subsequent image processing and analysis, and improves overall processing efficiency.

[0059] The method for obtaining the image quality comparison coefficient is:

[0060]

[0061] In the formula, b is the frame number, α b is the image quality comparison coefficient of the b-th frame image, SLm v is the maximum brightness of the brightness range of the b-th frame image, SLm b is the minimum brightness of the brightness range of the b-th frame image, Sy bis the image uniformity of the b-th frame image, Smy b is the wound color accuracy of the b-th frame image, CLm is the maximum brightness of the brightness parameter range, CLm is the minimum brightness of the brightness parameter range, Cy is the image parameter uniformity, Cmy is the wound color accuracy, and e is a natural constant.

[0062] In the formula What is calculated is the average deviation between the maximum and minimum brightness of the frame image and the reference brightness, which partially reflects whether the image brightness configuration meets the preset standard. By calculating the square root of the difference between the actual image uniformity and the standard uniformity, we can measure whether the image uniformity meets the requirements. Using the square root can reduce the sensitivity of the deviation and make the evaluation smoother. By calculating the exponential of the deviation of the color accuracy of the wound surface, the importance of color accuracy in image quality is emphasized. The exponential function can amplify the color deviation so that even small color errors are clearly reflected in the comparison coefficient.

[0063] Brightness directly affects color performance. Improper brightness settings may cause color distortion and affect the doctor's judgment of the wound status. Uneven brightness distribution may cause loss of details in certain areas of the image, thereby affecting the overall image quality assessment. Color unevenness may indicate problems with the image capture or display device, or may be due to improper parameter settings during image processing.

[0064] By independently evaluating and integrating different dimensions (brightness, uniformity, color accuracy), specific problems with image quality can be more accurately located, providing direction for further image optimization. By directly comparing with preset standards, standardization of image quality control is achieved, ensuring that all surgical images meet certain quality standards.

[0065] The method of determining the surgical process corresponding to each frame image based on the surgical process feature data set of each frame image includes the following steps: obtaining surgical process feature allowable error data of each frame image in the surgical image set, wherein the surgical process feature allowable error data includes the instrument shape matching allowable error, the wound surface shape matching allowable error, the relative distance matching allowable error between the instrument center and the frame image center, and the angle matching allowable error between the instrument center axis and the horizontal line; obtaining each surgical process feature matching data set stored in a database, wherein the surgical process feature matching data set includes the instrument shape matching data, the wound surface matching shape, the relative distance matching data between the instrument center and the frame image center, and the angle matching data between the instrument center axis and the horizontal line; comprehensively analyzing the surgical process feature allowable error data of each frame image, each surgical process feature matching data set, and the surgical process feature data set of each frame image to obtain the process matching coefficient of each frame image; obtaining the minimum process matching coefficient corresponding to each frame image; and determining the surgical process label stored in the database corresponding to the surgical process feature matching data set corresponding to each minimum process matching coefficient as the surgical process corresponding to the frame image.

[0066] The error tolerance range of the instrument shape, wound surface shape, relative position of the instrument to the image center, and the angle between the instrument center axis and the horizontal line is collected in each frame image. These error data are preset according to the precise requirements of the surgery and the deviations that may occur in actual operation. The pre-defined feature matching data sets corresponding to each surgical process are extracted from the database. These data sets define the ideal state of key features such as instrument shape and position expected at each surgical stage. The actual surgical process features of each frame image are compared with the matching data sets in the database, and the process matching coefficient is calculated according to the preset allowable error range. This coefficient measures the degree of closeness between the actual image and the expected state. For each frame image, the surgical process with the smallest matching coefficient is selected as the labeled process of the frame. This means that this process has the smallest difference with the actual image in all possible surgical stages.

[0067] The method for obtaining the process matching coefficient is:

[0068]

[0069] Where b is the frame number, β b is the process matching coefficient of the b-th frame image, A b is the device shape data of the b-th frame image, B b is the wound shape of the b-th frame image, C b is the relative distance data between the center of the instrument and the center of the frame image of the bth frame image, D bis the angle data between the central axis of the instrument and the horizontal line of the vth frame image, c is the number of the surgical process feature matching dataset, A c is the instrument shape matching data of the cth surgical process feature matching dataset, B c is the wound surface matching shape of the cth surgical process feature matching dataset, C c is the instrument center of the cth surgical procedure feature matching dataset, D c is the relative distance matching data of the frame image center and the angle matching data between the instrument center axis and the horizontal line of the cth surgical process feature matching data set, σ(A b , A c ) is A b With A c The similarity function, σ(B b , B c ) is B b With B c The similarity function, σ(C b , C c ) is C b With C c The similarity function, σ(D b , D c ) is D b With D c Similarity function. The similarity function includes but is not limited to Euclidean distance, Manhattan distance, etc.

[0070] For each feature (instrument shape, wound shape, relative distance, angle position), calculate the similarity between the current frame image and the surgical process feature matching dataset stored in the database. The summation part in the formula accumulates the similarities of all features to obtain a comprehensive similarity score. Each feature contributes equal weight. The total similarity score is normalized using the hyperbolic tangent function tanh, limiting the matching coefficient to between 0 and 1. This transformation helps to deal with extreme values ​​and makes the matching coefficient more stable and interpretable.

[0071] The shape of the instrument determines its function and purpose. Different surgical stages may require instruments of different shapes. The shape of the instrument affects the operation mode and specific implementation of the operation. Correctly identifying the shape is crucial to determining the surgical process. The shape of the wound can reflect the progress and current stage of the operation. For example, the size and shape of the incision may indicate the specific steps of the operation. The morphological changes of the wound are directly related to the success or failure of the operation and are an intuitive indicator for evaluating the progress of the operation. The relative distance shows the positional relationship between the instrument and the operating field of view (i.e., the center of the frame image), which is necessary to ensure surgical accuracy and avoid misoperation. Accurate distance control can reduce surgical risks and improve the accuracy of the operation. The angle position indicates the operating angle of the instrument. For some delicate operations, such as suturing or cutting, the correct angle is the key to success. The precise adjustment of the instrument angle directly affects the effect and safety of surgical cutting or treatment.

[0072] These four characteristics work together in every specific step of the operation and are highly correlated with each other. The shape of the instrument and the angle and position of use are closely related. The use of instruments of different shapes in specific surgical sites requires specific angle and position adjustments. Changes in the shape of the wound are often caused by the instrument, and vice versa. The operating method of the instrument (including shape and angle) needs to be adjusted according to the condition of the wound. The comprehensive consideration of these four characteristics makes the judgment of the surgical process more comprehensive and accurate.

[0073] The method of obtaining the surgical process feature allowable error data of each frame image in the surgical image set includes the following steps: obtaining the time data corresponding to each frame image; obtaining the brightness data of the resection site during the operation; obtaining the brightness of the resection site corresponding to each frame image from the brightness data of the resection site during the operation based on the time data corresponding to each frame image; obtaining the image quality comparison coefficient corresponding to each frame image; combining the brightness of the resection site corresponding to each frame image and the image quality comparison coefficient into an error matching data group; comparing the error matching data group with each matching data group stored in the database one by one to obtain the error matching coefficient, wherein the matching data group includes the matching brightness of the resection site and the matching comparison coefficient of the image quality; obtaining the surgical process feature allowable error data of the frame image stored in the database corresponding to the matching data group corresponding to the minimum error matching coefficient.

[0074] The logic of this process is to accurately match and determine the surgical process characteristics and allowable errors of each frame image by analyzing the brightness data and image quality comparison coefficient of the resection site in the frame image in detail. Each frame image has a corresponding timestamp. Based on these time data, the brightness information of the corresponding frame image is extracted from the brightness data of the resection site during the operation, the quality of each frame image is analyzed, and an image quality comparison coefficient is calculated. This coefficient reflects the degree of compliance of the image with the preset quality standard. The brightness of the resection site and the image quality comparison coefficient are combined into an error matching data group, and a comprehensive feature description including brightness and quality is established for each frame image. The error matching data group is compared with the matching data group stored in the database, and the matching item with the smallest difference is found by comparison. The matching data group with the smallest error matching coefficient is selected, and the allowable error data of the surgical process characteristics in the data group is determined, so as to provide the surgical team with the brightness and image quality standards that the current frame image should have.

[0075] The calculation formula of the error matching coefficient is as follows:

[0076]

[0077] In the formula, b is the frame number, γ b is the image quality comparison coefficient corresponding to the b-th frame image, α b is the brightness of the resection site corresponding to the b-th frame image, SL b is the image quality comparison coefficient corresponding to the b-th frame image, j is the number of the matching data group, α j is the image quality matching comparison coefficient in the jth matching data set, CL j Match the brightness for the resection site in the jth matching data set.

[0078] |α b -α j |The absolute difference between the brightness of the resection site in the frame image and the preset brightness standard in the matching data set is calculated. |SL b -CL j |The absolute difference between the image quality comparison coefficient of the frame image and the image quality standard in the matching data set is calculated. The two difference values ​​are added together, and this sum reflects the overall deviation of the image from the standard in terms of brightness and image quality. The weighted sum of the total difference is square root processed and natural logarithmic transformed. Such a transformation helps stabilize the influence of extreme values ​​and ensures that the growth of the error matching coefficient is smoother, which is especially useful when dealing with data with extremely large differences.

[0079] Using square roots and logarithmic transformations helps mitigate the impact of extreme difference values ​​on the final result, making the result more robust. By incorporating differences in brightness and image quality, the formula can provide a more comprehensive error assessment, helping to fully understand how much the image deviates from the preset standard.

[0080] The image quality comparison coefficient is usually a comprehensive indicator calculated based on multiple image attributes (such as clarity, contrast, noise level, etc.), which reflects the overall quality and applicability of the image. The brightness of the resection site refers to the light level of the surgical area, which is crucial for the visibility and clarity of the image. The overall quality of the image is directly affected by the brightness level. Inappropriate brightness can cause the image to be overexposed or underexposed, affecting the image quality comparison coefficient. For example, low brightness may cause the image details to be lost, while high brightness may cause the image to be overexposed and the details to be "washed out". These two factors together determine whether the image is clear enough, thus affecting the execution of the operation. In summary, the image quality comparison coefficient and the brightness of the resection site are key factors in evaluating the quality of surgical images. Together, they support the visual needs of the surgical process and have a direct impact on improving surgical accuracy and success rate.

[0081] There are a few points to note:

[0082] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0083] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0084] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0085] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A surgical procedure recognition method based on robot-assisted nephrectomy, characterized in that: The following steps are involved: Determine the initial number of video frames; Performing initial frame segmentation processing on the acquired robot-assisted nephrectomy surgical video based on the number of initial video segmentation frames to obtain a surgical image set; Perform comprehensive image quality testing on the surgical image set to determine whether the comprehensive image quality is qualified: If the overall image quality is unqualified, the frame images with unqualified quality in the surgical image set are screened, and image enhancement processing is performed on the screened frame images with unqualified quality; If the comprehensive image quality is qualified, then obtain the surgical process feature data set of each frame image in the surgical image set, wherein the surgical process feature data set includes instrument shape data, wound surface shape, relative distance data between the instrument center and the frame image center, and angle data between the instrument center axis and the horizontal line; The surgical procedure corresponding to each frame image is determined based on the surgical procedure feature data set of each frame image.

2. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 1, characterized in that: Determining the number of initial video frames includes the following steps: To obtain surgical videos and intraoperative brightness data of the resection site during robot-assisted nephrectomy; Determine the duration of the surgical video and the basic process of resection; Determine the average intraoperative ambient brightness based on the intraoperative resection site brightness data; The number of initial video frames is determined based on the duration of the surgical video, the basic process of resection, and the average ambient brightness during the operation.

3. The surgical procedure identification method based on robot-assisted nephrectomy according to claim 2, characterized in that: The method of determining the number of initial video frames based on the duration of the surgical video, the basic process of the resection, and the average ambient brightness during the operation includes the following steps: Comparing the basic resection procedure with each basic resection procedure label stored in the database to determine the basic resection procedure label corresponding to the basic resection procedure; Acquire a frame quantity matching data set corresponding to a basic process label of resection surgery stored in a database, wherein the frame quantity matching data set includes a plurality of frame quantity matching data, and the frame quantity matching data includes a surgical video matching duration and an average environmental matching brightness during surgery; Compare the duration of the surgical video, the average ambient brightness during the operation, and each frame number matching data in the frame number matching data set one by one to obtain the comparison coefficient; The number of sub-frames corresponding to the minimum comparison coefficient is obtained, and the matching data corresponds to the number of initial sub-frames of the video stored in the database.

4. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 3, characterized in that: The method for obtaining the comparison coefficient is as follows: Where bx is the comparison coefficient, St is the duration of the surgical video, Sd is the average ambient brightness during the operation, a is the number of frames matching the data, and Ct a Cd is the duration of the surgical video matching data for the ath frame number, a is the intraoperative average environment matching brightness of the a-th frame matching data, and e is a natural constant.

5. The surgical procedure identification method based on robot-assisted nephrectomy according to claim 1, characterized in that: The comprehensive image quality detection of the surgical image set to determine whether the comprehensive image quality is qualified includes the following steps: Acquire image detection data of each frame image in the surgical image set, wherein the image detection data includes brightness range, image uniformity, and wound surface color accuracy; The image detection data of each frame image is compared with the reference image detection data of the frame image stored in the database to obtain an image quality comparison coefficient, wherein the reference image detection data includes a brightness reference range, image reference uniformity, and wound surface reference color accuracy; The image quality comparison coefficient is used to determine whether the overall image quality is qualified.

6. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 5, characterized in that: The method of judging whether the comprehensive image quality is qualified based on the image quality comparison coefficient comprises the following steps: Obtain the mean square error of the image quality comparison coefficient; Determine whether the mean square error is greater than the mean square error parameter value stored in the database: If the mean square error is greater than the mean square error parameter value stored in the database, the overall image quality is unqualified; If the mean square error is not greater than the mean square error parameter value stored in the database.

7. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 6, characterized in that: The step of determining the surgical procedure corresponding to each frame image based on the surgical procedure feature data set of each frame image comprises the following steps: Acquire surgical process feature allowable error data of each frame image in the surgical image set; Acquire each surgical procedure feature matching data set stored in the database; Comprehensively analyzing the surgical process feature allowable error data of each frame image, each surgical process feature matching data set, and the surgical process feature data set of each frame image to obtain the process matching coefficient of each frame image; Obtain the minimum process matching coefficient corresponding to each frame image; The surgical process feature matching data sets corresponding to each minimum process matching coefficient correspond to the surgical process label stored in the database and are determined as the surgical process corresponding to the frame image.

8. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 7, characterized in that: The surgical process feature allowable error data includes the instrument shape matching allowable error, the wound surface shape matching allowable error, the relative distance matching allowable error between the instrument center and the frame image center, and the angle matching allowable error between the instrument center axis and the horizontal line. The surgical process feature matching data set includes instrument shape matching data, wound surface matching shape, relative distance matching data between the instrument center and the frame image center, and angle matching data between the instrument center axis and the horizontal line.

9. The surgical procedure identification method based on robot-assisted nephrectomy according to claim 7, characterized in that: The step of obtaining the surgical process feature allowable error data of each frame image in the surgical image set comprises the following steps: Acquire the brightness of the resection site corresponding to each frame image; Obtaining image quality comparison coefficients corresponding to each frame image; Combining the brightness of the resection site and the image quality comparison coefficient corresponding to each frame image into an error matching data set; Comparing the error matching data set with each matching data set stored in the database one by one to obtain an error matching coefficient, wherein the matching data set includes a resection site matching brightness and an image quality matching comparison coefficient; The matching data group corresponding to the minimum error matching coefficient is obtained, which corresponds to the surgical procedure feature allowable error data of the frame image stored in the database.

10. The method for identifying surgical procedures based on robot-assisted nephrectomy according to claim 9, characterized in that: The step of obtaining the brightness of the resection site corresponding to each frame image comprises the following steps: Obtaining time data corresponding to each frame image; Acquire intraoperative resection site brightness data; The resection site brightness corresponding to each frame image is acquired from the resection site brightness data during the operation based on the time data corresponding to each frame image.

Images