Surgical stage prompting method, device, equipment, medium and product

Abstract

Embodiments of the present application provide a surgical stage prompting method, device, equipment, medium and product. The method first acquires a to-be-processed surgery video, wherein the to-be-processed surgery video includes at least one to-be-processed video frame, then for each to-be-processed video frame, the to-be-processed video frame is input into a surgery stage recognition model to obtain stage matching information for representing matching probability of the to-be-processed video frame and each surgery stage, then according to a time sliding window set in the to-be-processed surgery video, a target video frame is determined, then according to the stage matching information of all target video frames, a current surgery stage in the time sliding window is determined, and finally based on the current surgery stage and all surgery stages, surgery stage prompting information of a visual interface is generated. By applying the technical solution of the present application, the continuity and stability of surgery stage recognition are effectively improved through comprehensive analysis of continuous video frames by the time sliding window, and accurate prompting of the surgery stage is realized.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a method, apparatus, device, medium, and product for indicating surgical stages. Background Technology

[0002] Minimally Invasive Surgery (MIS) involves entering the body through a small incision and relies on an endoscope to provide a surgical view. The operating space is limited and the procedures are complex. In MIS, surgeons typically perform the procedure based on endoscopic video feeds and rely on surgical stage recognition and progress indicators to determine the current stage of the surgery.

[0003] Existing surgical stage indication methods are mainly based on a preset linear process model, where the preset surgical operations are strictly executed in a fixed order. By processing the endoscopic images step by step and combining them with a smoothing filtering algorithm, short-term recognition jumps are eliminated to determine the current surgical stage.

[0004] However, existing methods cannot adapt to non-standard operations such as skipping steps or reverting to the previous procedure due to unforeseen circumstances. Furthermore, electrosurgical smoke, bleeding, or lens fogging can easily cause fluctuations in the determination of surgical stages, resulting in poor stability in the identification of surgical stages and low accuracy in real-time prompts for surgical stages. Summary of the Invention

[0005] This application provides methods, devices, equipment, media, and products for providing prompts during the surgical stage, in order to solve the problem of low accuracy in real-time prompts during the surgical stage in complex minimally invasive surgical scenarios.

[0006] In a first aspect, embodiments of this application provide a method for providing prompts during the surgical phase, comprising:

[0007] Acquire a surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed;

[0008] For each video frame to be processed, the video frame to be processed is input into the surgical stage recognition model to determine the stage matching information of the video frame to be processed. The stage matching information is used to indicate the matching probability of the video frame to be processed with each surgical stage. The surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame. The annotation information is used to indicate the sample surgical stage in which the sample video frame is located.

[0009] Based on the time sliding window set in the surgical video to be processed, determine the target video frame within the time sliding window;

[0010] Based on the stage matching information of all target video frames, determine the current surgical stage within the time sliding window;

[0011] Based on the current surgical stage and all surgical stages, generate surgical stage prompts on a visual interface.

[0012] In one or more embodiments, determining the current surgical stage within the time sliding window based on stage matching information of all target video frames includes:

[0013] Based on the stage matching information of all target video frames, determine the confidence level of all target video frames at each surgical stage;

[0014] The current surgical stage within the time sliding window is determined based on the confidence levels of all target video frames at each surgical stage.

[0015] In one or more embodiments, determining the current surgical stage within the time sliding window based on the confidence levels of all target video frames at each surgical stage includes:

[0016] The surgical stage with the highest confidence level is identified as the candidate surgical stage;

[0017] If the confidence level of the candidate surgical stage is greater than the preset confidence level, then the candidate surgical stage is determined as the current surgical stage;

[0018] If the confidence level of the candidate surgical stage is less than or equal to the preset confidence level, then the previous surgical stage corresponding to the previous time sliding window is determined as the current surgical stage.

[0019] In one or more embodiments, determining the confidence level of all target video frames at each surgical stage based on the stage matching information of all target video frames includes:

[0020] Based on the stage matching information of all target video frames, the surgical stage with the highest matching probability corresponding to each target video frame is determined as the target surgical stage of the target video frame.

[0021] Determine the number of consecutive times the target surgical stage occurs within the time sliding window;

[0022] The confidence level of the target surgical stage is determined based on the number of consecutive occurrences.

[0023] In one or more embodiments, generating surgical stage prompts for a visual interface based on the current surgical stage and all surgical stages includes:

[0024] All the surgical stages are arranged according to a preset surgical operation sequence to obtain the first surgical stage sequence;

[0025] The icons for the first surgical stage sequence are displayed in the prompt box of the visualization interface; wherein the prompt box is located in the non-central edge area of ​​the visualization interface.

[0026] The icon representing the current surgical stage will be highlighted in the prompt box;

[0027] The set of icons displayed in the prompt box is used as the surgical stage prompt information of the visualization interface.

[0028] In one or more embodiments, the method further includes:

[0029] In the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, the first surgical stage sequence is rearranged according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain the second surgical stage sequence.

[0030] The icon set of the second surgical stage sequence is displayed in place of the icon set of the first surgical stage sequence in the prompt box of the visualization interface.

[0031] Secondly, embodiments of this application provide a surgical stage prompting device, comprising:

[0032] An acquisition module is used to acquire a surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed.

[0033] The first determining module is used to input the video frame to be processed into the surgical stage recognition model for each video frame to be processed, and determine the stage matching information of the video frame to be processed. The stage matching information is used to indicate the matching probability of the video frame to be processed with each surgical stage. The surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame. The annotation information is used to indicate the sample surgical stage in which the sample video frame is located.

[0034] The second determining module is used to determine the target video frame within the time sliding window set in the surgical video to be processed.

[0035] The third determining module is used to determine the current surgical stage within the time sliding window based on the stage matching information of all target video frames;

[0036] The processing module is used to generate surgical stage prompts on a visual interface based on the current surgical stage and all surgical stages.

[0037] In one or more embodiments, the third determining module is specifically used for:

[0038] Based on the stage matching information of all target video frames, determine the confidence level of all target video frames at each surgical stage;

[0039] The current surgical stage within the time sliding window is determined based on the confidence levels of all target video frames at each surgical stage.

[0040] In one or more embodiments, the third determining module determines the current surgical stage within the time sliding window based on the confidence levels of all target video frames at each surgical stage, specifically for:

[0041] The surgical stage with the highest confidence level is identified as the candidate surgical stage;

[0042] If the confidence level of the candidate surgical stage is greater than the preset confidence level, then the candidate surgical stage is determined as the current surgical stage;

[0043] If the confidence level of the candidate surgical stage is less than or equal to the preset confidence level, then the previous surgical stage corresponding to the previous time sliding window is determined as the current surgical stage.

[0044] In one or more embodiments, the third determining module determines the confidence level of all target video frames at each surgical stage based on the stage matching information of all target video frames, specifically for:

[0045] Based on the stage matching information of all target video frames, the surgical stage with the highest matching probability corresponding to each target video frame is determined as the target surgical stage of the target video frame.

[0046] Determine the number of consecutive times the target surgical stage occurs within the time sliding window;

[0047] The confidence level of the target surgical stage is determined based on the number of consecutive occurrences.

[0048] In one or more embodiments, the processing module is specifically used for:

[0049] All the surgical stages are arranged according to a preset surgical operation sequence to obtain the first surgical stage sequence;

[0050] The icons for the first surgical stage sequence are displayed in the prompt box of the visualization interface; wherein the prompt box is located in the non-central edge area of ​​the visualization interface.

[0051] The icon representing the current surgical stage will be highlighted in the prompt box;

[0052] The set of icons displayed in the prompt box is used as the surgical stage prompt information of the visualization interface.

[0053] In one or more embodiments, the third determining module is further configured to:

[0054] In the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, the first surgical stage sequence is rearranged according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain the second surgical stage sequence.

[0055] The icon set of the second surgical stage sequence is displayed in place of the icon set of the first surgical stage sequence in the prompt box of the visualization interface.

[0056] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0057] The memory stores computer-executed instructions;

[0058] The processor executes computer execution instructions stored in the memory, such that the processor, when executed, is used to implement the method described in the first aspect and any of the embodiments above.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the first aspect and any of the embodiments above.

[0060] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, is used to implement the surgical stage prompting method as described in the first aspect and various possible implementations of the first aspect.

[0061] This application provides a method, apparatus, device, medium, and product for indicating surgical stages. The method first acquires a surgical video to be processed, wherein the surgical video includes at least one video frame to be processed. Then, for each video frame to be processed, the video frame is input into a surgical stage recognition model to obtain stage matching information characterizing the matching probability between the video frame to be processed and each surgical stage. Next, a target video frame is determined based on a time sliding window set in the surgical video to be processed. Then, the current surgical stage within the time sliding window is determined based on the stage matching information of all target video frames. Finally, based on the current surgical stage and all surgical stages, a surgical stage prompt message for a visual interface is generated. In the above method, based on at least one video frame to be processed in the surgical video to be processed, a surgical stage recognition model is used to determine stage matching information to characterize the matching probability between the video frame to be processed and each surgical stage. Multiple consecutive target video frames are selected through a time sliding window, and the target video frames are comprehensively analyzed in combination with the stage matching information. Compared with single-frame recognition, it can effectively reduce the recognition fluctuations caused by cross-step jumps, backtracking and rework, and image interference, improve the continuity and stability of surgical stage recognition, and provide real-time and accurate prompts on the visualization interface for the current surgical stage. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 1 ;

[0064] Figure 2 A schematic diagram of a visual interface for a surgical stage prompting method provided in an embodiment of this application;

[0065] Figure 3 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 2 ;

[0066] Figure 4 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 3 ;

[0067] Figure 5 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 4 ;

[0068] Figure 6 A schematic diagram of the surgical stage prompting device provided in an embodiment of this application;

[0069] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first:

[0073] Minimally invasive surgery typically involves entering the body through a narrow incision. The surgical field is small, the field of view is limited, and instruments frequently obstruct each other. Therefore, doctors cannot rely solely on experience to perceive the progress of the surgery. Instead, they need to rely on the endoscope to continuously obtain the surgical field of view and use surgical stage recognition and progress prompts to provide real-time assistance in order to determine the current stage of the surgical procedure.

[0074] In existing technologies, surgical stage prompts typically employ visual analysis methods based on a preset linear workflow. The preset surgical operations strictly follow a fixed sequence, and audiovisual prompts are provided for each stage. This method first establishes stage templates based on typical surgeries, then analyzes endoscopic video frames frame by frame to determine the surgical stage matching the current video frame. Some solutions further combine the order of instrument appearance, local image texture, or instrument movement characteristics to infer whether the surgery is in a specific step such as suturing, hemostasis, or separation. Other solutions use single-frame recognition with smoothing processing to suppress short-term misjudgments and reduce frequent prompt jumps.

[0075] While the aforementioned technologies can provide some assistance when the surgical procedure is relatively stable, the surgical field is clear, and the operation is strictly performed according to the established steps, their limitations are quite obvious in complex clinical scenarios. On the one hand, minimally invasive surgery does not always follow an ideal linear process. Surgeons may skip certain steps, return to previous steps to reprocess them, or even change the order of operations based on the actual situation due to differences in patient anatomy, bleeding risks, tissue adhesions, or unexpected situations during surgery. Traditional stage judgment methods based on fixed sequences are difficult to correctly understand this non-standard process and are prone to continuously outputting stage results that are inconsistent with the actual surgery. On the other hand, endoscopic images are highly susceptible to factors such as electrocautery smoke, lens fogging, instrument obstruction, bleeding staining, and visual field tremors. The recognition results within a single frame or short window are prone to frequent fluctuations between adjacent stages, leading to repeated switching of prompts. This not only reduces the surgeon's trust in the system but may also increase cognitive interference during critical operations.

[0076] Furthermore, if the recognition results are directly linked to the displayed prompts, the fixed information bar or stage bar may also obscure the lesion, blood vessel, or instrument operation area, making it impossible for doctors to simultaneously pay attention to auxiliary information and the core field of vision, further amplifying the risk of operational errors.

[0077] It is evident that while existing technologies can provide surgical stage prompts to a certain extent, they still have significant shortcomings in terms of adaptability to dynamic processes, stability under complex visual conditions, and the security of auxiliary prompt information display. They are insufficient to meet the comprehensive requirements of complex minimally invasive surgeries for real-time, accurate, and low-interference auxiliary prompts.

[0078] The surgical stage prompting method provided in this application aims to solve the aforementioned technical problems of the prior art. The technical concept of this application is as follows: Existing technical solutions mainly use single-frame video for surgical stage identification and provide simple progress prompts. In complex surgical scenarios involving step jumps or reversals, this is prone to identification fluctuations, leading to inaccurate surgical stage prompts. Prompting surgical stages primarily involves image analysis of real-time endoscopic video during the surgical process, combined with safety assistance prompts. If multiple video frames can be combined for comprehensive judgment, the stability of surgical stage identification can be effectively improved, thereby increasing the accuracy of surgical stage prompts. Therefore, this application's embodiment considers acquiring the surgical video to be processed and analyzing each frame of the video to be processed. For each video frame, stage matching information is determined, indicating the probability of matching between the video frame and each surgical stage. Then, based on a time sliding window set in the surgical video to be processed, target video frames within the time sliding window are selected. Next, the current surgical stage within the time sliding window is determined based on the stage matching information of all target video frames. Finally, based on the current surgical stage and all surgical stages, a visual interface for surgical stage prompts is generated. This technical solution does not rely on instantaneous judgment of a single frame. Instead, it enhances the adaptability to short-term occlusion, smoke interference, and image fluctuations by performing comprehensive temporal analysis on continuous video frames. This makes the determination of the current surgical stage more closely resemble the actual surgical process. Through this time-sliding window-based stage recognition logic, the continuity and stability of surgical stage judgment can be improved in complex surgical fields of view. This provides a reliable basis for generating subsequent surgical stage prompts consistent with the current progress, thereby improving the accuracy of real-time surgical stage prompts.

[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0080] Figure 1 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 1 .like Figure 1 As shown, the prompting method for this surgical stage includes the following steps:

[0081] S101. Obtain the surgical video to be processed;

[0082] The surgical video to be processed includes at least one video frame to be processed.

[0083] In this step, a surgical video to be processed, including at least one video frame, is acquired to provide reliable data for identification in subsequent surgical stages.

[0084] For example, the surgical video to be processed refers to the surgical video input that needs to be identified and analyzed for the surgical stage. It is usually an endoscopic video stream during a minimally invasive surgery, which contains multiple consecutive video frames to be processed. The video frame to be processed refers to a single image frame that is separated from the surgical video to be processed. Its function is to serve as the smallest processing unit for surgical stage inference.

[0085] In one possible implementation, the entity executing this surgical stage prompting method can be deployed in an endoscope workstation, surgical robot console, edge computing terminal, or server processing node, and receive intraoperative video data through a video acquisition interface. The video acquisition interface can be a high-definition multimedia interface, a serial digital interface, a network streaming media interface, or a video output interface defined by the equipment manufacturer, and the processing end accesses the video stream through a driver or video capture card.

[0086] In real-time assistance scenarios, the surgical video to be processed can be directly derived from the video stream continuously output during the operation; in offline analysis scenarios, the surgical video to be processed can also be derived from external surgical recording files, intraoperative cache files, or video files transmitted from remote consultation platforms. After reading the file, the processing end decodes it into continuous image frames according to a predetermined frame rate, thereby forming a sequence of video frames to be processed.

[0087] In one possible implementation, to ensure the comparability and stability of subsequent recognition results, this embodiment can also perform preprocessing operations after acquiring the surgical video to be processed. These preprocessing operations include decoding the video stream, frame synchronization, timestamp correction, resolution unification, color space conversion, and abnormal frame detection. For example, if the acquired video is in a compressed encoding format, it is first decoded to restore the original image sequence; if different devices have inconsistent output resolutions, the images are uniformly scaled to the required input size; if there are dropped frames, screen tearing, or long periods of stillness in the video, abnormal frames are identified through timestamp continuity determination and image difference determination, and abnormality markers are recorded. For common phenomena in the surgical environment such as sudden brightness changes, local reflections, smoke obstruction, and lens fogging, basic image enhancement processing can also be performed, such as brightness normalization, contrast-limited adaptive histogram equalization, color normalization, or lightweight noise reduction processing, thereby improving the recognizability of single-frame content.

[0088] In this embodiment, the video frames to be processed are not limited to the original video frames in the strict sense, but can also be analysis frames extracted from the original video at fixed intervals. For example, when the original video frame rate is high and processing resources are limited, a continuous sequence of video frames to be processed can be extracted from the surgical video to be processed according to a preset sampling rate, thereby reducing the computational pressure while ensuring the continuity of the stages.

[0089] S102. For each video frame to be processed, input the video frame to be processed into the surgical stage recognition model to determine the stage matching information of the video frame to be processed.

[0090] The stage matching information is used to indicate the probability of matching the video frame to be processed with each surgical stage.

[0091] The surgical stage identification model is trained based on multiple sample video frames and the corresponding annotation information for each sample video frame. The annotation information is used to indicate the surgical stage of the sample video frame.

[0092] For example, the surgical stage recognition model is an artificial intelligence model used to determine the surgical stage to which the video frame to be processed belongs; the sample video frame is historical image frame data used to train the surgical stage recognition model; the annotation information is a training label that is manually or semi-automatically marked to indicate the surgical stage of the sample video frame; the sample surgical stage corresponds to the real surgical stage category, including: incision, exposure, separation, clamping, hemostasis, resection, suturing, examination, etc.

[0093] It should be understood that before step S102, sample video frames are first extracted from historical minimally invasive surgical recordings, and clinically experienced annotators annotate the surgical stages according to the actual progress of the surgical operation, determining the annotation information corresponding to each sample video frame. To improve the adaptability of the surgical stage recognition model to complex scenarios, in addition to stage labels, the annotation information can also include extended supervisory information such as instrument type, instrument appearance area, and lesion location.

[0094] Next, the surgical stage recognition model is trained based on the processed sample video frames. This model can be implemented using convolutional neural networks, visual Transformers, a combination of convolutional and temporal networks, or a multi-branch fusion structure. If a single-frame image classification path is used, the sample video frames are input into the image feature extraction backbone network to extract spatial texture features, color features, instrument edge features, and tissue region features. The predicted probabilities for each surgical stage are then output via a classification head.

[0095] During training, supervised learning is used to minimize the loss function between the surgical stage recognition model output and the labeled information. The loss function can be cross-entropy loss, or additional focal loss, label smoothing loss, or center constraint loss can be applied. If there are few samples for certain surgical stages, training bias can be suppressed through resampling or class weighting. After training, validation set evaluation and threshold calibration yield a surgical stage recognition model suitable for online inference.

[0096] During the actual surgical stage identification in step S102, each video frame to be processed is input into the trained surgical stage identification model according to the time sequence of the video frames. The surgical stage identification model outputs a stage matching information vector for each video frame to be processed, with each dimension of the vector corresponding to a candidate surgical stage. To enhance engineering feasibility, a frame number, timestamp, image quality score, and model confidence label can be added to the stage matching information of each frame.

[0097] In this embodiment, by forming a matching probability for each video frame to be processed corresponding to all surgical stages, instead of just outputting a single category label, richer judgment information is retained. At the same time, the surgical stage recognition model trained based on sample video frames and annotation information can learn stable feature mapping relationships of different surgical stages in instrument posture, tissue texture, and image structure, so that the recognition process is transformed from experience judgment into a quantifiable and repeatable model reasoning process, improving the objectivity and consistency of stage analysis.

[0098] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0099] S103. Determine the target video frame within the time sliding window set in the surgical video to be processed.

[0100] For example, a time sliding window refers to a set of frames that move continuously with the video stream in the time dimension; the target video frame is a set of multiple video frames located within the time sliding window that participate in the identification of the surgical stage.

[0101] In one possible implementation, a time sliding window of fixed or adjustable length is set on the timeline of the surgical video to be processed, and it continuously moves forward according to a set update step size. The time sliding window is defined by the number of frames, directly selecting the latest video frame and a preset number of video frames before it as the target video frames. After receiving each new image frame, the time sliding window is advanced by one frame, and the oldest frame is moved out simultaneously, forming an overlapping update mechanism.

[0102] Furthermore, the parameters of the time-sliding window can be dynamically adjusted based on the actual surgical scenario. When the surgery enters a delicate operational phase, such as hemostasis, suturing, or critical tissue separation, the changes in the surgical field are subtle, but the stability requirements for recognition are high. In this case, the length of the time-sliding window can be appropriately increased, allowing more video frames to participate in the judgment, thereby suppressing short-term misjudgments. When the surgery is in a phase of rapid switching or large-scale instrument movement, the length of the time-sliding window can be shortened to avoid delayed prompts and enhance the response speed to phase changes. Through this adaptive window mechanism, stability and real-time performance can be effectively ensured.

[0103] In this embodiment, by selecting continuous or quasi-continuous target video frames in the time dimension, an intermediate layer mechanism is constructed to transition from single-frame recognition to multi-frame joint judgment, so that the real stage features accumulate and appear in time, thereby weakening the occasional recognition fluctuations caused by brief occlusion, smoke, reflection and shaking, and improving the continuity and reliability of intraoperative assistance.

[0104] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0105] S104. Determine the current surgical stage within the time sliding window based on the stage matching information of all target video frames.

[0106] In this step, the stage matching information of all target video frames is analyzed and processed. For each target video frame, the matching probability between the target video frame and each surgical stage is determined, thereby determining the current surgical stage within the time sliding window that includes all target video frames.

[0107] In one possible implementation, the stage matching information corresponding to all target video frames within the time sliding window is first determined, and statistical aggregation is performed according to the surgical stage category. The mean, weighted average, maximum value, or median of the matching probability value of the same surgical stage in all target video frames are calculated to obtain the window-level surgical stage score. Then, the surgical stage with the highest score is selected as the candidate surgical stage.

[0108] To avoid frequent jumps in surgical stage output, this embodiment can also introduce a continuous confidence assessment mechanism. Specifically, the window-level surgical stage score of the candidate surgical stage is first determined as the confidence level. When the confidence score of a candidate surgical stage within a time sliding window is higher than a preset confirmation threshold, and the consecutive count of the candidate surgical stage within the time sliding window is higher than a preset count, and the confidence fluctuation of the candidate surgical stage is lower than a preset variance threshold, then the candidate surgical stage is determined as the current surgical stage. It should be understood that if multiple candidate surgical stages meet the above conditions, the candidate surgical stage with the highest confidence level is determined as the current surgical stage; if no candidate surgical stage meets the above conditions, the previous surgical stage remains unchanged, that is, the previous surgical stage is determined as the current surgical stage.

[0109] It should be understood that the above thresholds can be calibrated using a validation dataset, or they can be set separately for different surgical procedures to accommodate the complexity of different surgical processes.

[0110] To address scenarios where video information is missing due to severe obstruction or bleeding, this embodiment can also integrate auxiliary information during the time-sliding window stage determination. Auxiliary information may include the robotic arm's motion trajectory, instrument displacement, instrument posture changes, instrument type switching records, or user interaction logs. For example, when the video image is largely obscured by smoke, but the suturing instrument is detected continuously performing repetitive retraction and extension movements in the target area, this action signal can be used as supporting evidence for the suturing surgery stage. If this action signal is captured during the time-sliding window stage determination, the confidence level corresponding to the suturing surgery stage can be effectively improved, thereby enhancing the continuity of the current surgical stage.

[0111] In this embodiment, by statistically fusing the stage matching information of multiple frames within a time sliding window, random fluctuations at the single-frame level are transformed into stable trends at the time series level, thereby making the final current surgical stage closer to the actual operation process. Furthermore, this scheme does not rely on a linear fixed procedure and has better adaptability to non-standard procedures such as returning to previous steps during surgery, skipping some steps, or temporarily changing the order. It should be understood that the above example is merely illustrative and not limiting.

[0112] S105. Based on the current surgical stage and all surgical stages, generate surgical stage prompts on a visual interface.

[0113] In this step, the current surgical stage and all surgical stages are displayed in the visualization interface, and the displayed information serves as the surgical stage prompts in the visualization interface.

[0114] For example, the displayed information includes display settings for the current surgical stage and all surgical stages.

[0115] In one possible implementation, step S105 may include the following steps:

[0116] Step 1: Arrange all surgical stages according to the preset surgical operation sequence to obtain the first surgical stage sequence;

[0117] For example, the sequence of the first surgical stage is arranged according to the preset surgical operation order of the standard procedure template.

[0118] Step 2: Display the icons of the first surgical stage sequence in the prompt box of the visualization interface;

[0119] The prompt box is located in the non-central edge area of ​​the visual interface;

[0120] For example, the visualization interface can be set on a surgical monitor, console display, or auxiliary terminal. The prompt box uses a semi-transparent panel to display a set of icons for each stage, which can consist of graphic symbols corresponding to the names of the surgical stages. The prompt box is set within the edge display band of the visualization interface, avoiding the central area of ​​the visualization interface and the core lesion area, thereby reducing obstruction of the core surgical field.

[0121] Step 3: Highlight the icon for the current surgical stage in the prompt box.

[0122] For example, the icon for the current surgical stage can be highlighted by darkening the color, thickening the border, increasing the size, or increasing the brightness, so that it becomes the visual focus in the icon set.

[0123] In one possible implementation, after obtaining the current surgical stage, a stage sequence table (i.e., the first surgical stage sequence) is first generated according to the preset surgical operation sequence, and the stage sequence table is mapped to the corresponding icons. Then, the set of icons of the first surgical stage sequence is rendered into the prompt box of the visualization interface in order.

[0124] Step 4: Use the set of icons displayed in the prompt box as the surgical stage prompt information in the visual interface.

[0125] For example, the position of the prompt box can be dynamically determined according to the screen resolution and the safe area of ​​the surgical field. It is usually fixed in the non-central edge area such as the upper right corner, upper left corner, lower left corner or lower right corner, so as to ensure that the prompt information and the surgical screen are visible at the same time.

[0126] The surgical stage prompts include: the icon corresponding to the current surgical stage is highlighted during rendering, while the icons for other surgical stages remain in normal display mode.

[0127] Figure 2 This is a schematic diagram of a visual interface illustrating a surgical stage prompting method provided in an embodiment of this application. For example... Figure 2 As shown, the visualization interface includes a surgical screen area and a prompt box, with the prompt box positioned in the upper right corner of the visualization interface. The surgical screen area includes surgical instruments 201 and 202, and a core lesion area 203 located within the target tissue region. The prompt box includes an edge-floating, anti-obstruction user interface (UI) panel 204 (named "Intelligent Auxiliary Navigation Panel"), a highlighted area displaying the current stage status 205 (e.g., the current surgical stage is "Tissue Freeing (In Progress)"), and a dynamically rearranged, simplified step navigation 206.

[0128] In this embodiment, by peripheralizing the surgical stage information and highlighting the current stage, the surgeon can quickly ascertain the progress of the surgery without obstructing the main surgical field. Because the stage icons are arranged in a preset order and the current surgical stage is highlighted, the surgeon can intuitively determine the relationship between completed stages, the current stage, and subsequent stages, which helps maintain operational continuity and reduce visual interference in complex minimally invasive surgical scenarios.

[0129] In one possible implementation, the prompting method for this surgical stage further includes the following steps:

[0130] Step 1: In the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, the first surgical stage sequence is rearranged according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain the second surgical stage sequence.

[0131] For example, the second surgical stage sequence refers to a dynamic stage list generated based on the current surgical stage, the previous surgical stage, and the preset surgical operation sequence when it is detected that the current surgical stage and the previous surgical stage do not conform to the adjacent successor relationship in the preset surgical operation sequence. It is used to reflect the deviation relationship between the actual surgical progress and the preset process.

[0132] In one possible implementation, once the current surgical stage is determined, the previous surgical stage and the preset surgical operation sequence are read simultaneously, and a relationship check is performed to verify whether the current surgical stage is the next surgical stage after the previous one. If the check result indicates that the two do not constitute an adjacent successor relationship, the first surgical stage sequence is rearranged according to the stage identifier corresponding to the current surgical stage, the stage identifier corresponding to the previous surgical stage, and the sequential dependencies between stages in the preset surgical operation sequence. The rearrangement process may include keeping completed stages as completed, moving the current surgical stage to a highlighted position, and adjusting subsequent stages associated with the current stage to a pending state, thereby forming a second surgical stage sequence consistent with the actual intraoperative operation.

[0133] Step 2: Replace the icon set of the first surgical stage sequence with the icon set of the second surgical stage sequence in the prompt box of the visualization interface.

[0134] For example, the icons corresponding to each surgical stage in the second surgical stage sequence are extracted and replaced with the original set of icons for the first surgical stage sequence in the prompt box. The displayed content is then refreshed so that the doctor can intuitively see the current surgical progress and the update status of the surgical stage.

[0135] In this embodiment, when a cross-stage jump, return to a previous stage, or partial redo occurs during the operation, the visualization interface will not continue to display in a fixed order. Instead, it will rearrange and replace the icon set in real time based on the current state, thereby reducing the deviation between the prompt information and the actual operation and improving the reliability and usability of auxiliary prompts in complex minimally invasive surgical environments.

[0136] The above steps are illustrated with examples below:

[0137] If the preset surgical procedure sequence is: surgical stage A, surgical stage B, surgical stage C, and surgical stage D, then the sequence of the first surgical stage is: A, B, C, D.

[0138] 1) Cross-stage jump: If the previous surgical stage is surgical stage B, according to the preset surgical operation sequence, the next surgical stage should be surgical stage C. However, the current surgical stage is actually identified as surgical stage D. Since surgical stages B and D do not constitute an adjacent successor relationship, a cross-stage jump of surgical stages has occurred. In this case, the sequence of the first surgical stage is rearranged.

[0139] The rearrangement includes: keeping the completed stages (Surgery Stage A and Surgery Stage B) as completed, moving the current surgical stage (Surgery Stage D) after Surgery Stage B, and extending the unexecuted Surgery Stage C and adjusting it to a pending state. The rearranged second surgical stage sequence is: A, B, D, C.

[0140] Within the visualization prompt box, the icon sets corresponding to the second surgical stage sequences A, B, D, and C are replaced with the original icon sets for the first surgical stage sequences A, B, C, and D. At the same time, the current surgical stage (Surgical Stage D) is highlighted, and the visualization interface refreshes in real time to display the icon sets corresponding to the new sequences.

[0141] 2) Re-perform the surgery locally and jump back to the previous stage: If the previous surgical stage is surgical stage C, according to the preset surgical operation sequence, the next surgical stage should be surgical stage D. However, if the current surgical stage is actually identified as regressing to surgical stage B, which does not conform to the adjacent successor relationship, then the first surgical stage sequence is rearranged. That is, the second surgical stage sequence generated by the rearrangement is: A, C, B, C, D.

[0142] The rearrangement includes: following the rule of not disrupting completed stages, keeping completed stages (Surgery Stage A and Surgery Stage C) in a completed state, and maintaining the surgical stage sequence A, B, C, D, only highlighting the rollback to Surgery Stage B, changing Surgery Stage C to an unexecuted state (it needs to be checked whether it is executed a second time), and keeping Surgery Stage D in an unexecuted state.

[0143] Within the visualization prompt box, replace the original set of icons for the first surgical stage sequence A, B, C, D with the set of icons corresponding to the second surgical stage sequence A, C, B, C, D, while highlighting the current surgical stage (Surgical Stage B).

[0144] The surgical stage prompting method provided in this application first acquires a surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed; then, for each video frame to be processed, the video frame to be processed is input into a surgical stage recognition model to determine the stage matching information of the video frame to be processed, wherein the stage matching information is used to indicate the matching probability between the video frame to be processed and each surgical stage, and the surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame, the annotation information is used to indicate the sample surgical stage in which the sample video frame is located; then, according to the time sliding window set in the surgical video to be processed, the target video frame in the time sliding window is determined; then, according to the stage matching information of all target video frames, the current surgical stage in the time sliding window is determined; finally, based on the current surgical stage and all surgical stages, surgical stage prompting information on a visual interface is generated. In this embodiment, by introducing multiple video frames from a continuous surgical video into the time dimension for comprehensive analysis, a complete processing chain is constructed, from video acquisition, probability output, multi-frame selection via a time sliding window to surgical stage identification and prompts. By performing comprehensive analysis of continuous video frames in the time dimension, the adaptability to short-term occlusion, smoke interference, and image fluctuations is enhanced, making the determination of the current surgical stage closer to the actual operation process. The surgical stage prompts generated on the visual interface based on the current surgical stage and all surgical stages can not only quickly reflect changes in intraoperative operations, but also avoid frequent switching of prompts that may interfere with the doctor, thereby improving the accuracy of intraoperative progress prompts, assisted navigation, and interface linkage display.

[0145] Based on the above embodiments, Figure 3 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 2 .like Figure 3 As shown, one possible implementation of step S104 above includes the following steps:

[0146] S301. Based on the stage matching information of all target video frames, determine the confidence level of all target video frames at each surgical stage;

[0147] For example, stage matching information is used to characterize the correspondence between target video frames and each surgical stage, usually represented by the probability distribution, score vector, or confidence vector of each surgical stage; confidence is used to characterize the strength of support of a surgical stage by multiple frames within a time sliding window, and the higher the value, the more the surgical stage matches the current surgical field features.

[0148] In one possible implementation, the stage matching information corresponding to each target video frame within the time sliding window is first read and then summarized and calculated according to a preset surgical stage category to obtain the confidence level of each target video frame under each surgical stage. For the same surgical stage, the matching probabilities of multiple frames within the time sliding window can be statistically analyzed by mean, weighted average, or stability, with video frames closer to the current time and with higher image quality being assigned greater weight. If a video frame has low-quality features caused by smoke obscuring, fogging, bleeding coverage, or instrument obscuring, the confidence level corresponding to that video frame can be reduced.

[0149] In one possible implementation, step S301 described above may include the following steps:

[0150] Step 1: Based on the stage matching information of all target video frames, determine the surgical stage with the highest matching probability for each target video frame as the target surgical stage of the target video frame.

[0151] For example, the surgical stage with the highest matching probability refers to the surgical stage obtained by selecting the highest value from the probabilities of each surgical stage corresponding to a single target video frame.

[0152] Step 2: Determine the number of consecutive times the target surgical stage occurs within the time sliding window;

[0153] For example, the consecutive number of times refers to the number of frames in which the target surgical stage remains the highest probability surgical stage within the time sliding window. Within the time sliding window, the consecutive count can be based on the frame sequence number, or it can be converted into a continuous duration based on the sampling frame rate.

[0154] Step 3: Determine the confidence level of the target surgical stage based on the number of consecutive attempts.

[0155] For example, the confidence level increases with the number of consecutive iterations and can be determined in combination with a preset mapping relationship, a threshold function, or a linear normalization function.

[0156] In one possible implementation, firstly, all target video frames within the time sliding window are read, and the stage matching information for each target video frame is obtained. The stage matching information is typically a matching probability vector corresponding to each surgical stage. Next, a maximum probability search is performed on each target video frame, and the surgical stage corresponding to the maximum value in the matching probability vector is selected as the target surgical stage for that target video frame. Then, the target surgical stages are sequentially compared along the time axis of the time sliding window. When the target surgical stages of adjacent target video frames are consistent, the consecutive counts are accumulated. The consecutive counts are based on the first appearance position and duration of the target surgical stage within the time sliding window. When different surgical stages occur, the accumulation stops, resulting in a consecutive count. If a target video frame matching the target surgical stage exists after a different surgical stage has occurred, a second consecutive count is accumulated. If multiple consecutive counts are obtained within the time sliding window, the largest consecutive count is taken as the consecutive count of the target surgical stage within the time sliding window. For example, if the target surgical stage appears consecutively in frames 1-3 and frames 5-7 within the window, two consecutive counts (3 frames and 3 frames) are recorded respectively, and the maximum value (3 frames) is taken as the consecutive count of that stage. Finally, the confidence level of the target surgical stage is determined based on the consecutive count.

[0157] The confidence level can be determined by the following method: the greater the number of consecutive counts, the higher the confidence level. When the number of consecutive counts reaches a preset threshold, the confidence level of the target surgical stage rises to a higher level. When the number of consecutive counts is small, the confidence level is maintained at a lower level to avoid short-term misjudgment affecting the recognition results.

[0158] In one possible implementation, consecutive counts can be mapped to confidence levels as percentages and normalized by incorporating the length of the time sliding window, making the confidence levels comparable across different window lengths. This confidence level can then be used as a basis for confirming the current surgical stage, comparing it with the confidence levels of other surgical stages to identify more stable surgical stages within the time sliding window.

[0159] S302. Determine the current surgical stage within the time sliding window based on the confidence levels of all target video frames at each surgical stage.

[0160] In one possible implementation, after obtaining the confidence levels of all target video frames at each surgical stage, the confidence levels of the same surgical stage are compared and aggregated within a time-sliding window, and the surgical stage with the highest confidence level is selected as the current surgical stage. To avoid frequent switching of surgical stages due to individual abnormal frames, confirmation thresholds and switching thresholds are set. When the confidence level of a surgical stage does not reach the confirmation threshold, the previously confirmed stage remains unchanged. When multiple consecutive target video frames within the time-sliding window all belong to the same surgical stage and maintain a high confidence level and reach the switching threshold, that surgical stage is confirmed as the current surgical stage.

[0161] In this embodiment, the instantaneous judgment result of a single target video frame for each surgical stage is transformed into a confidence representation of multiple frames within a time sliding window, and the surgical stage identification is completed accordingly. This can improve the continuity and stability of stage judgment within the time sliding window, reduce the probability of false switching caused by interference such as smoke, fog, and occlusion, and make the output results more suitable for subsequent intraoperative stage prompts, surgical progress display, and other subsequent process analysis.

[0162] In one possible implementation, step S302 described above may include the following steps:

[0163] Step 1: Identify the surgical stage with the highest confidence level as the candidate surgical stage;

[0164] For example, a candidate surgical stage refers to the surgical stage with the highest confidence level obtained after statistical analysis of the stage matching information of each target video frame within the current time sliding window.

[0165] In one possible implementation, if multiple surgical stages have the same and maximum confidence levels, the stage that appears earlier in the pre-defined surgical procedure order is selected as the candidate surgical stage. For example, in the "incision → exposure → separation" procedure, if the confidence levels of "incision" and "exposure" are the same, then "incision" is selected as the candidate surgical stage.

[0166] Step 2: If the confidence level of the candidate surgical stage is greater than the preset confidence level, then the candidate surgical stage is determined as the current surgical stage.

[0167] For example, the pre-set confidence level refers to the threshold used to determine whether the current candidate surgical stage is sufficient to trigger a surgical stage switch, which is used to suppress short-term misjudgments caused by smoke, obstruction, fogging or shaking.

[0168] Step 3: If the confidence level of the candidate surgical stage is less than or equal to the preset confidence level, then the previous surgical stage corresponding to the previous time sliding window is determined as the current surgical stage.

[0169] For example, the previous surgical stage corresponding to the previous time sliding window refers to the surgical stage that has been confirmed in the previous time sliding window.

[0170] In one possible implementation, after obtaining the confidence levels of each surgical stage within the current time sliding window, the confidence levels of all surgical stages are compared, and the stage corresponding to the highest confidence level is selected as the candidate surgical stage. This confidence level can be calculated from the mean, weighted mean, or median of the matching probabilities of surgical stages corresponding to multiple target video frames within the time sliding window. The weights can be determined based on video frame clarity, temporal proximity, or anomaly markers. Next, the confidence level of the candidate surgical stage is compared with a preset confidence level. When the confidence level of the candidate surgical stage is higher than the preset confidence level, a stable surgical stage is considered to have been formed within the current time sliding window, and this candidate surgical stage is thus determined as the current surgical stage. When the confidence level of the candidate surgical stage is not higher than the preset confidence level, the previous surgical stage corresponding to the previous time sliding window is used to avoid frequent stage jumps due to single-frame deviations.

[0171] In this embodiment, the preset confidence level can be obtained by calibrating historical surgical video samples, and can be set to different values ​​based on different surgical procedures, different stages of complexity, and image quality fluctuations. The surgical stage recognition model can use convolutional networks, visual Transformers, or a combination thereof, to output the matching probability of each surgical stage for the input video frames to be processed, and then aggregate them through a time sliding window to form the surgical stage confidence level.

[0172] The surgical stage indication method provided in this application first determines the confidence level of each target video frame under each surgical stage based on the stage matching information of all target video frames. Then, based on the confidence level of each target video frame under each surgical stage, the current surgical stage within a time sliding window is determined. In this embodiment, by calculating the confidence level of the stage matching information of multiple target video frames within the time sliding window, the continuous features of the time-series video frames are fully utilized, avoiding fluctuations and misjudgments caused by surgical environment interference in single-frame recognition. Determining the current surgical stage based on the confidence level of each target video frame under each surgical stage can effectively filter out recognition jitter caused by instantaneous interference, improving the stability and accuracy of surgical stage recognition.

[0173] Based on the above embodiments, Figure 4 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 3 .like Figure 4 As shown, one possible implementation of this surgical stage prompting method includes the following steps:

[0174] S401. Obtain the preset surgical stage template and the video to be processed;

[0175] For example, the preset surgical stage template includes: the total number of surgical stages N and the specific identification information of each surgical stage (such as surgical stage number, stage name, and preset surgical operation order), and the video to be processed includes multiple video frames to be processed.

[0176] In one possible implementation, before the surgery begins, the doctor selects a preset surgical type instruction. Based on this instruction, a preset surgical stage template corresponding to that surgical procedure is loaded from the local machine or a server. The set of surgical stages arranged according to the preset surgical operation order constitutes the first surgical stage sequence.

[0177] By extracting the video to be processed according to the preset sampling frequency, multiple video frames to be processed can be obtained, that is, the sequence of video frames to be processed.

[0178] S402. Use the surgical stage recognition model to identify and process the video to be processed, and determine the stage matching information of the video frames to be processed.

[0179] For example, stage matching information is used to indicate the probability of a video frame being processed matching each surgical stage.

[0180] S403. Based on the time sliding window set in the surgical video to be processed, and combined with the stage matching information of all target video frames, determine the current surgical stage within the time sliding window;

[0181] S404. Display the specific identification information of the current surgical stage and all surgical stages in the prompt box of the visualization interface.

[0182] For example, the prompt box is located in the off-center edge area of ​​the visualization interface, and the current surgical stage is highlighted in the prompt box.

[0183] In one possible implementation, Figure 5 Flowchart of the surgical stage prompting method provided in the embodiments of this application Figure 4 .like Figure 5 As shown, the prompting method for this surgical stage also includes the following steps:

[0184] S501. Based on the stage matching information of all target video frames, determine the confidence level of all target video frames at each surgical stage;

[0185] S502. The surgical stage with the highest confidence level is identified as the candidate surgical stage;

[0186] In one possible implementation, if multiple surgical stages have the same confidence level and are all at the maximum value, the earlier stage is selected as the candidate surgical stage according to the preset surgical operation order.

[0187] S503. For each candidate surgical stage, determine whether the built-in confidence level of the candidate surgical stage in the time sliding window is greater than the preset confidence level. If yes, continue to step S504; otherwise, proceed to step 505.

[0188] S504. Determine the candidate surgical stage as the current surgical stage and rearrange the preset surgical operation sequence;

[0189] For example, the set of icons for the second surgical stage sequence obtained by rearranging the order of surgical operations is displayed in the prompt box of the visualization interface, and the icon of the current surgical stage is highlighted.

[0190] S505. Determine the previous surgical stage corresponding to the previous time sliding window as the current surgical stage.

[0191] Based on the above embodiments, the following are embodiments of the apparatus involved in this application:

[0192] Figure 6 This is a schematic diagram of the surgical stage prompting device provided in an embodiment of this application. Figure 6 As shown, the prompting device 600 for this surgical stage includes:

[0193] The acquisition module 610 is used to acquire the surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed.

[0194] The first determining module 620 is used to input the video frame to be processed into the surgical stage recognition model for each video frame to be processed, and determine the stage matching information of the video frame to be processed. The stage matching information is used to indicate the matching probability between the video frame to be processed and each surgical stage. The surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame. The annotation information is used to indicate the sample surgical stage in which the sample video frame is located.

[0195] The second determining module 630 is used to determine the target video frame in the time sliding window based on the time sliding window set in the surgical video to be processed;

[0196] The third determining module 640 is used to determine the current surgical stage within the time sliding window based on the stage matching information of all target video frames.

[0197] The processing module 650 is used to generate surgical stage prompts for a visual interface based on the current surgical stage and all surgical stages.

[0198] In one or more embodiments, the third determining module 640 is specifically used for:

[0199] Based on the stage matching information of all target video frames, determine the confidence level of all target video frames at each surgical stage;

[0200] The current surgical stage within the time sliding window is determined based on the confidence levels of all target video frames at each surgical stage.

[0201] In one or more embodiments, the third determining module 640 determines the current surgical stage within a time sliding window based on the confidence levels of all target video frames at each surgical stage, specifically for:

[0202] The surgical stage with the highest confidence level was identified as the candidate surgical stage;

[0203] If the confidence level of a candidate surgical stage is greater than the preset confidence level, then the candidate surgical stage is determined as the current surgical stage.

[0204] If the confidence level of a candidate surgical stage is less than or equal to the preset confidence level, then the previous surgical stage corresponding to the previous time sliding window is determined as the current surgical stage.

[0205] In one or more embodiments, the third determining module 640 determines the confidence level of all target video frames at each surgical stage based on the stage matching information of all target video frames, specifically for:

[0206] Based on the stage matching information of all target video frames, the surgical stage with the highest matching probability corresponding to each target video frame is determined as the target surgical stage of the target video frame.

[0207] Determine the number of consecutive times the target surgical stage occurs within a time sliding window;

[0208] The confidence level of the target surgical stage is determined based on the number of consecutive attempts.

[0209] In one or more embodiments, the processing module 650 is specifically used for:

[0210] Arrange all surgical stages according to the preset surgical operation sequence to obtain the first surgical stage sequence;

[0211] The icons for the first surgical stage sequence are displayed in a prompt box on the visualization interface; the prompt box is located in the non-central edge area of ​​the visualization interface.

[0212] Highlight the icon for the current surgical stage in the prompt box;

[0213] The set of icons displayed in the prompt box serves as the surgical stage prompt information in the visual interface.

[0214] In one or more embodiments, the third determining module 640 is further configured to:

[0215] In the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, the first surgical stage sequence is rearranged according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain the second surgical stage sequence.

[0216] The icon set for the second surgical stage sequence is displayed in the prompt box of the visualization interface, replacing the icon set for the first surgical stage sequence.

[0217] The surgical stage prompting device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0218] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 includes: a processor 710, a memory 720, and a bus 730;

[0219] The memory 720 is used to store the computer-executed instructions of the processor 710;

[0220] The processor 710 is configured to execute the technical solutions of any of the foregoing method embodiments by executing computer execution instructions.

[0221] The specific implementation process of processor 710 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0222] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0223] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0224] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0225] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0226] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0227] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0228] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0229] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0230] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0231] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0232] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory (RAM), magnetic disks, or optical disks.

[0233] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0234] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for indicating surgical stages, characterized in that, include: Acquire a surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed; For each video frame to be processed, the video frame to be processed is input into the surgical stage recognition model to determine the stage matching information of the video frame to be processed. The stage matching information is used to indicate the matching probability of the video frame to be processed with each surgical stage. The surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame. The annotation information is used to indicate the sample surgical stage in which the sample video frame is located. Based on the time sliding window set in the surgical video to be processed, the target video frame within the time sliding window is determined; wherein, the parameters of the time sliding window are dynamically adjusted in combination with the actual surgical scenario: when the surgery enters the fine operation stage, the length of the time sliding window is increased to suppress short-term misjudgment; when the surgery is in the rapid switching or large-scale instrument movement stage, the length of the time sliding window is shortened to enhance the response speed to stage changes; Based on the stage matching information of all target video frames, determine the confidence level of each target video frame under each surgical stage; wherein, determining the confidence level of each target video frame under each surgical stage includes: based on the stage matching information of all target video frames, determining the surgical stage with the highest matching probability corresponding to each target video frame as the target surgical stage of the target video frame; determining the number of consecutive occurrences of the target surgical stage within the time sliding window; and determining the confidence level of the target surgical stage based on the number of consecutive occurrences. The surgical stage with the highest confidence level is identified as the candidate surgical stage; Based on the current surgical stage and all surgical stages, generate surgical stage prompts in a visual interface; The process of generating a visual interface with surgical stage prompts based on the current surgical stage and all surgical stages includes: All the surgical stages are arranged according to a preset surgical operation sequence to obtain the first surgical stage sequence; The icon set of the first surgical stage sequence is displayed in the prompt box of the visualization interface, and the prompt box is located in the non-central edge area of ​​the visualization interface; The icon representing the current surgical stage will be highlighted in the prompt box; The set of icons displayed in the prompt box is used as the surgical stage prompt information of the visualization interface; The method further includes: In the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, the first surgical stage sequence is rearranged according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain the second surgical stage sequence. The icon set of the second surgical stage sequence is displayed in place of the icon set of the first surgical stage sequence in the prompt box of the visualization interface.

2. The method according to claim 1, characterized in that, The step of identifying the surgical stage with the highest confidence level as the candidate surgical stage includes: If the confidence level of the candidate surgical stage is greater than the preset confidence level, then the candidate surgical stage is determined as the current surgical stage; If the confidence level of the candidate surgical stage is less than or equal to the preset confidence level, then the previous surgical stage corresponding to the previous time sliding window is determined as the current surgical stage.

3. A surgical stage prompting device, characterized in that, include: An acquisition module is used to acquire a surgical video to be processed, wherein the surgical video to be processed includes at least one video frame to be processed. The first determining module is used to input the video frame to be processed into the surgical stage recognition model for each video frame to be processed, and determine the stage matching information of the video frame to be processed. The stage matching information is used to indicate the matching probability of the video frame to be processed with each surgical stage. The surgical stage recognition model is obtained by training the model based on multiple sample video frames and the annotation information corresponding to each sample video frame. The annotation information is used to indicate the sample surgical stage in which the sample video frame is located. The second determining module is used to determine the target video frame within the time sliding window set in the surgical video to be processed; wherein, the parameters of the time sliding window are dynamically adjusted in combination with the actual surgical scenario: when the surgery enters the fine operation stage, the length of the time sliding window is increased to suppress short-term misjudgment; when the surgery is in the rapid switching or large-scale instrument movement stage, the length of the time sliding window is shortened to enhance the response speed to stage changes; The third determining module is used to determine the confidence level of each target video frame under each surgical stage based on the stage matching information of all target video frames; wherein, determining the confidence level of each target video frame under each surgical stage includes: determining the surgical stage with the highest matching probability corresponding to each target video frame as the target surgical stage of the target video frame based on the stage matching information of all target video frames; determining the number of consecutive occurrences of the target surgical stage within the time sliding window; determining the confidence level of the target surgical stage based on the number of consecutive occurrences; and determining the surgical stage with the highest confidence level as the candidate surgical stage; The processing module is used to generate surgical stage prompts for a visual interface based on the current surgical stage and all surgical stages. The processing module is specifically used to arrange all the surgical stages according to a preset surgical operation sequence to obtain a first surgical stage sequence. The icon set of the first surgical stage sequence is displayed in the prompt box of the visualization interface, and the prompt box is located in the non-central edge area of ​​the visualization interface; The icon representing the current surgical stage will be highlighted in the prompt box; The set of icons displayed in the prompt box is used as the surgical stage prompt information of the visualization interface; The third determining module is further configured to, in the preset surgical operation sequence, if the current surgical stage is not the next surgical stage after the previous surgical stage, rearrange the first surgical stage sequence according to the current surgical stage, the previous surgical stage, and the preset surgical operation sequence to obtain a second surgical stage sequence. The icon set of the second surgical stage sequence is displayed in place of the icon set of the first surgical stage sequence in the prompt box of the visualization interface.

4. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in claim 1 or 2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in claim 1 or 2.

6. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of claim 1 or 2.

Images