A laparoscopic surgery video retrieval and visualization method and system

Abstract

The present application relates to a kind of laparoscopic surgery video retrieval and visualization method and system, which is achieved by "video analysis-data organization-multi-mode retrieval" efficient surgery video retrieval.In the video analysis stage, the deep learning model is combined with the hidden Markov model of medical knowledge constraint, automatically analyzes the video content and divides the standardization process unit, reduces the consumption of computing resources and training time.In the data organization stage, the feature entity bidirectional mapping is proposed, which converts the unstructured video data into a hierarchical data structure with medical semantics, facilitating storage and retrieval.In the multi-mode retrieval stage, three retrieval modes of time axis orientation, instrument orientation and process unit orientation are designed, combined with multi-track display and medical semantic visualization coding to realize dynamic visualization.Through the interactive feedback of multi-mode retrieval method and time sequence feature dynamic visualization, a professional retrieval experience that meets the training and research needs of medical personnel is provided.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for retrieval and visualization of laparoscopic surgical videos. Background Technology

[0002] Laparoscopic surgery, with its significant advantages of minimal trauma and rapid recovery, has been widely used in clinical practice and has become an important technique in modern surgery. However, its learning curve is long, and it demands extremely high skill levels from physicians, making surgical videos increasingly important as teaching resources. With the rapid increase in surgical volume, video data is exploding, and the management and retrieval of laparoscopic surgical videos faces a contradiction between the sheer volume of data and low efficiency, becoming a critical problem that urgently needs to be solved.

[0003] Currently, existing methods for retrieving laparoscopic surgery videos primarily rely on progress bar annotations. These methods involve adding simple notes to the original video progress bar and manually locating segments. Additionally, deep learning models are being explored for analysis and retrieval of surgical videos, utilizing object detection algorithms to identify surgical instruments.

[0004] Existing laparoscopic surgery video retrieval technologies suffer from the following problems: (1) the data organization method is simplistic, mostly involving adding notes to the original video progress bar; (2) the retrieval mode is limited to the video progress bar, requiring a sequential review of each video and struggling to support complex queries, while lacking a holistic presentation of information from multiple videos. These problems severely restrict the application value of surgical videos. Summary of the Invention

[0005] In view of this, it is necessary to provide a method and system for retrieval and visualization of laparoscopic surgical videos to solve the above-mentioned problems of the prior art.

[0006] To address the aforementioned problems, in a first aspect, embodiments of the present invention provide a method for retrieval and visualization of laparoscopic surgical videos, comprising:

[0007] Deep learning models are used to analyze instrument features in surgical videos, and a hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized surgical procedure units; where a surgical procedure unit is a standardized operation stage in the surgical process.

[0008] The data obtained from surgical video analysis is transformed into a hierarchical data structure, and a two-way mapping relationship is established between three types of entities: time, instrument characteristics, and process units.

[0009] Based on a hierarchical data structure, three interactive search modes are designed: time-axis-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0010] Dynamic visualization of surgical timelines is achieved through multi-track design and medical semantic visualization coding.

[0011] Preferably, the step of using a deep learning model to analyze instrument features in the surgical video includes:

[0012] Deep learning models were used to analyze the instrument features in laparoscopic surgery videos.

[0013] By combining the sliding window mid-range filtering method to eliminate instantaneous misjudgments, the instrument characteristic data is output.

[0014] Preferably, the hidden Markov model based on medical knowledge constraints performs temporal semantic parsing to divide the standardized surgical procedure into units, including:

[0015] Construct multidimensional temporal features of surgical videos; among which, the multidimensional temporal features of surgical videos include static instrument features, instrument dynamic features, specific instrument combination features, continuous instrument use features, and surgical progress features;

[0016] Using the multidimensional temporal features of surgical videos as the observation input to the Hidden Markov Model, and combining them with medical knowledge constraints, a Hidden Markov Model with medical knowledge constraints is established.

[0017] The Viterbi algorithm was used to process the observation sequence of the surgical video to preliminarily infer the sequence of surgical procedure units.

[0018] For parts of the surgical procedure unit sequence that do not conform to medical logic, optimization is performed through abnormal time period allocation and procedure unit correction to output a standardized surgical procedure unit sequence.

[0019] Preferably, the expression of the hidden Markov model constrained by the medical knowledge is:

[0020] MK-HMM=(S, O, A, B, π, M)

[0021] In the formula, MK-HMM represents a Hidden Markov Model constrained by medical knowledge; S = {S1, S2, ..., S...} n} represents the set of states, corresponding to n standardized surgical procedure units; O represents the set of observed states, A represents the state transition probability matrix, B represents the emission probability matrix, π represents the initial state probability vector, and M represents the set of medical knowledge constraints;

[0022] The medical knowledge constraint set M includes the sequence constraint matrix M_seq and the initial state constraint vector M_init(i); M = {M_seq, M_init(i)};

[0023] Based on the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constrained state transition matrix and the constrained initial state probability are obtained;

[0024] The expression for the constrained state transition matrix A'(i,j) is:

[0025] A'(i,j)=A(i,j)*M_seq(i,j)

[0026] In the formula, M_seq(i,j) is a sequence constraint matrix, which represents the legal transfer relationship between surgical procedure units; the matrix element M_seq(i,j) indicates whether the transfer from procedure unit i to j conforms to medical standards, with a value of 1 indicating a legal transfer and a value of 0 indicating an illegal transfer;

[0027] The expression for the constrained initial state probability π'(i) is:

[0028] π'(i) = π(i) * M_init(i)

[0029] In the formula, M_init(i) is the initial state constraint vector, which encodes the rationality of each process unit as the starting unit of the operation. For example, M_init(1) = 1 means that the "operative preparation" unit can be used as the starting unit, while M_init(7) = 0 means that "gallbladder removal" should not be used as the starting unit of the operation. The specific values ​​of these constraint matrices are defined by medical experts according to the standard surgical procedure.

[0030] Preferably, the step of converting the data obtained from surgical video analysis into a hierarchical data structure and establishing a bidirectional mapping relationship between three types of entities—time, instrument characteristics, and process units—includes:

[0031] Design a spatiotemporal feature mapping data model that includes time entities, instrument feature entities, and process unit entities, and encode the three types of core entities into a nested key-value pair structure to form a hierarchical data form;

[0032] Establish bidirectional mapping relationships between three types of entities: time-instrument feature, time-process unit, and process unit-instrument feature, forming a mapping network.

[0033] Preferably, the hierarchical data structure-based design includes three interactive retrieval modes: time-axis-oriented retrieval, device-oriented retrieval, and process unit-oriented retrieval, comprising:

[0034] Time-axis guided search mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video segment; it supports searching across multiple video files and automatic continuous playback; when selecting a time point for retrieval in the time-series visualization interface, the system automatically displays the surgical procedure unit and instrument characteristics corresponding to that time point through a bidirectional mapping relationship;

[0035] Medical Device-Oriented Retrieval Mode: Based on the feature-time inverse mapping in the hierarchical data structure, a fast index of medical device type and timestamp is established, and the median filtering algorithm is fused to reduce noise in the medical device detection results; it supports single medical device or multi-medical device combination retrieval; after the user selects a single medical device or medical device combination, the system calls the inverse mapping function of the hierarchical data organization layer to obtain the corresponding time period, and generates a playback instruction queue to automatically play the video;

[0036] Process Unit-Oriented Retrieval Mode: The Hidden Markov Model based on medical knowledge constraints parses the surgical video into standardized process units, with each process unit associated with medical auxiliary annotations. After the user selects a process unit identifier, the system calls the process unit-time inverse mapping function to obtain the time interval corresponding to the process unit, and simultaneously calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit. The system locates the video to the start time and simultaneously displays the medical auxiliary annotation content clearly on the interface, highlighting the corresponding interval on the timeline.

[0037] Preferably, the multi-track display design includes:

[0038] Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of instrument combinations is clearly displayed through vertical layering.

[0039] Adaptive time scale: Automatically adjusts the time scale density based on the total length of multiple videos and the current zoom level;

[0040] Color coding: Different color codes are used to distinguish different types of surgical instruments, improving visual recognition efficiency;

[0041] Density-perceptual representation: Visually expressing the frequency and intensity of instrument combinations through point density, revealing high-frequency periods of surgical procedures.

[0042] The medical semantic visualization encoding includes:

[0043] Process unit background labeling: Different surgical process units are marked with semi-transparent color blocks, and medical semantics are encoded through color and transparency;

[0044] Key point highlighting: Automatically identifies and highlights key time points during the surgical procedure;

[0045] Cross-video boundary markers: Clearly express the structure of multiple videos using vertical dashed lines and video identifiers.

[0046] Secondly, embodiments of the present invention provide a laparoscopic surgery video retrieval and visualization system, comprising:

[0047] The surgical video analysis module is used to analyze the instrument features in surgical videos using a deep learning model, and to perform temporal semantic parsing based on a hidden Markov model constrained by medical knowledge to divide the surgical process into standardized surgical procedure units; wherein, a surgical procedure unit is a standardized operation stage in the surgical process.

[0048] The hierarchical data organization module is used to transform the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument features, and process units.

[0049] The multi-mode interactive retrieval module is designed based on a hierarchical data structure to provide three interactive retrieval modes: time-axis-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0050] The visualization module is used to achieve dynamic visualization of surgical time sequences through multi-track design and medical semantic visualization encoding.

[0051] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein,

[0052] The memory is used to store programs;

[0053] The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the laparoscopic surgery video retrieval and visualization method as described in the first aspect embodiment of the present invention.

[0054] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the laparoscopic surgery video retrieval and visualization method as described in the first aspect embodiment of the present invention.

[0055] The laparoscopic surgery video retrieval and visualization method and system provided by this invention have the following advantages compared with the prior art:

[0056] This invention, based on a hierarchical data organization-based multimodal interactive retrieval method, innovatively constructs a "video analysis-data organization-multimodal retrieval" technical process, achieving efficient retrieval and intuitive presentation of laparoscopic surgical videos. This significantly enhances the application value of surgical videos in teaching, quality assessment, and skills analysis scenarios. Compared with existing technologies, this invention deeply integrates medical expertise to perform temporal modeling of surgical videos; it innovatively proposes a bidirectional mapping method to convert traditional surgical video data into hierarchical data that is easy to store and retrieve; and through multimodal retrieval methods and interactive feedback with dynamic visualization of temporal features, it provides medical personnel with a professional retrieval experience that meets their training and research needs. Attached Figure Description

[0057] Figure 1 Flowchart of the laparoscopic surgery video retrieval and visualization method provided by the present invention;

[0058] Figure 2 This is a schematic diagram of the laparoscopic surgery video retrieval and visualization method provided by the present invention;

[0059] Figure 3(a) is a diagram of the state transition probability matrix of the MK-HMM model provided by the present invention;

[0060] Figure 3(b) is a diagram of the emission probability matrix of the MK-HMM model provided by the present invention;

[0061] Figure 4(a) is a comparison between the actual annotation of the surgical procedure unit and the prediction results of MK-HMM;

[0062] Figure 4(b) is a time distribution diagram of the actual surgical procedure unit;

[0063] Figure 4(c) is a time distribution diagram of the predicted surgical procedure unit;

[0064] Figure 5 This invention provides a dynamic visualization diagram of time-series features.

[0065] Figure 6 This is a structural block diagram of the electronic device provided by the present invention.

[0066] The color differences in Figures 3-5 are used to distinguish technical features. Detailed Implementation

[0067] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0068] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0069] Existing surgical video retrieval and analysis technologies largely lack targeted searches for surgical instrument usage, failing to meet doctors' needs for analyzing specific instrument operation techniques. In terms of semantic parsing of the surgical process, they also do not fully integrate medical semantic annotation for retrieval, resulting in search results that are not intuitive or aligned with medical professional needs. Furthermore, the visualization of the temporal features of surgical videos lacks efficient, clear, and interactive retrieval methods, making it difficult for doctors to quickly and accurately obtain key information.

[0070] In view of this, the present invention aims to provide a method for retrieval and visualization of laparoscopic surgical videos. It implements a guided retrieval mode centered on surgical instruments, facilitating doctors to quickly locate and play specific instrument usage segments by establishing a reverse mapping from instrument type to time point. A unit retrieval mode based on surgical procedure semantics is constructed, mapping videos to standardized surgical procedure units and adding medical auxiliary annotations, making the retrieval results more in line with medical professional needs. Furthermore, through key technologies such as multi-track display design and medical semantic visualization encoding, the temporal characteristics of surgical videos are presented in a visually interactive manner, allowing doctors to intuitively observe and quickly interactively retrieve video segments of interest. The following will elaborate and describe this method through several embodiments.

[0071] Figure 1 This is a flowchart of the laparoscopic surgery video retrieval and visualization method provided by the present invention. Figure 2 This is a schematic diagram of the principle of the laparoscopic surgery video retrieval and visualization method provided by the present invention, with reference to... Figure 1 and Figure 2 The laparoscopic surgery video retrieval and visualization method provided by the present invention includes at least the following steps:

[0072] Step S1: Use a deep learning model to analyze the instrument features in the surgical video, and perform temporal semantic parsing based on a hidden Markov model constrained by medical knowledge to divide the surgical process into standardized units.

[0073] In this context, a surgical procedure unit is a standardized operational stage within the surgical process. In the description of this embodiment of the invention, a surgical procedure unit is also simply referred to as a procedure unit.

[0074] Specifically, this invention uses a deep learning model to analyze the instrument features in laparoscopic surgery videos.

[0075] In this embodiment, a deep learning model based on the D-FINE algorithm framework can be used to analyze the instrument features in laparoscopic surgical videos. A sliding window midpoint filtering method is then used to eliminate instantaneous misjudgments, outputting instrument feature data to provide data support for the temporal semantic analysis of the surgery. This method is not limited to a specific deep learning network; any target detection algorithm capable of accurately identifying surgical instruments can be applied.

[0076] In a preferred embodiment of the present invention, step S1 involves performing temporal semantic parsing based on a Medical Knowledge-constrained Hidden Markov Model (MK-HMM) to divide the procedure into standardized surgical procedure units, specifically including:

[0077] S11, construct multi-dimensional temporal features of surgical videos; among which, multi-dimensional temporal features of surgical videos include static instrument features, instrument dynamic features, specific instrument combination features, continuous instrument use features, and surgical progress features;

[0078] In this embodiment, a feature construction engineering based on the characteristics of laparoscopic surgery was designed to comprehensively capture the instrument combination patterns in the surgical procedure, taking into account the temporal characteristics of laparoscopic surgery.

[0079] 1) Static instrument characteristics: Record the state and combination of instruments at each moment to form a basic feature matrix:

[0080] F_static(t)=[S_1(t), S_2(t),..., S_n(t)]

[0081] Where S_i(t) represents the existence state (0 or 1) of the i-th tool at time t.

[0082] 2) Dynamic characteristics of laparoscopic surgical instruments: The dynamic changes of the instrument combination are captured by a time-series sliding window W(t,k).

[0083] F_dynamic(t)=[C_1(t), C_2(t),..., C_n(t), D(t), M(t), A(t)]

[0084] Where C_i(t) represents the number of times tool i changes state within window W(t, k) (tool switching frequency); D(t) represents the number of different tools used within the window (tool usage diversity); M(t) represents the tool used most frequently within the window (dominant tool); and A(t) represents the average number of tools used simultaneously within the window.

[0085] 3) Characteristics of Specific Instrument Combinations in Laparoscopic Surgery: This example defines five common instrument combination patterns and their durations in laparoscopic surgery. However, it is not limited to these five instrument combination patterns. Different numbers of instrument combination patterns can be defined depending on the specific type of laparoscopic surgery.

[0086] F_comb(t)=[C_{1,3}(t),C_{1,7}(t),C_{1,6}(t),C_{1,2}(t),C_{1,5}(t)]

[0087] F_dur(t)=[D_{1,3}(t), D_{1,7}(t), D_{1,6}(t), D_{1,2}(t), D_{1,5}(t)]

[0088] Where: 1-7 represent grippers, bipolar electrocoagulator, electric hook, scissors, clamping forceps, rinsing device, and specimen bag; C_{i,j}(t) represents the combined use status of instruments i and j at time t; D_{i,j}(t) represents the continuous use duration of the combination.

[0089] 4) Continuous instrument use characteristics: Calculate the continuous use time of each instrument to capture the persistent characteristics of instrument combinations during laparoscopic surgery.

[0090] F_cont(t)=[L_1(t), L_2(t),..., L_n(t)]

[0091] Where L_i(t) represents the number of consecutive frames used by tool i before time t.

[0092] 5) Surgical progress characteristics: Introducing time progress characteristics to characterize the relative position of the surgery within the overall progress:

[0093] F_prog(t) = t / T_total

[0094] Where T_total is the total number of frames in the surgical video.

[0095] Through the features constructed above, this invention transforms the instrument combination in laparoscopic surgery into a multi-dimensional feature vector, comprehensively describing the temporal dynamic characteristics of the instruments during the operation.

[0096] S12 uses the multidimensional temporal features of the surgical video as the observation input to the Hidden Markov Model, and combines medical knowledge constraints to establish a Hidden Markov Model with medical knowledge constraints.

[0097] Specifically, the MK-HMM model consists of a state set, an observation set, a state transition probability matrix, an observation probability matrix, and an initial state probability distribution. The state set represents different stages or operational states during the surgical process, while the observation set consists of multidimensional temporal features extracted from the surgical video. The parameters of the MK-HMM model are shown in Figures 3(a) and 3(b).

[0098] The constrained state transition matrix is ​​designed to constrain the process based on medical knowledge and the actual surgical procedure. For example, during a normal surgical procedure, certain operations must be performed in a specific order. Therefore, in the state transition matrix, the probability of state transitions that do not conform to this order is set to extremely low values ​​or even zero. This guides the model to learn state transition patterns that conform to medical logic.

[0099] Initial state probability: Based on medical knowledge, determine the initial probability distribution of each state at the start of surgery. For example, in most surgeries, the initial state is usually the preparation stage, so the state corresponding to the preparation stage will have a higher probability in the initial state probability distribution.

[0100] In this embodiment, the expression of the established hidden Markov model with medical knowledge constraints is:

[0101] MK-HMM=(S, O, A, B, π, M)

[0102] In the formula, MK-HMM represents a Hidden Markov Model constrained by medical knowledge; S = {S1, S2, ..., S...} n} represents the set of states, corresponding to n standardized surgical procedure units; in this embodiment, n = 7, corresponding to 7 standardized surgical procedure units (taking the seven surgical procedure units of cholecystectomy as an example).

[0103] O represents the set of observed states, obtained by clustering the feature space;

[0104] A represents the state transition probability matrix, A(i,j)=P(S_t=j|S_{t-1}=i);

[0105] B represents the emission probability matrix, B(i,k)=P(O_t=k|S_t=i);

[0106] π represents the initial state probability vector;

[0107] M represents the set of medical knowledge constraints; the set of medical knowledge constraints M includes the sequence constraint matrix M_seq and the initial state constraint vector M_init(i); M = {M_seq, M_init(i)};

[0108] Based on the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constrained state transition matrix and the constrained initial state probability are obtained;

[0109] Compared with the traditional Hidden Markov Model (HMM), the innovation of the MK-HMM model in this invention lies in the introduction of a medical knowledge constraint set M, which is implemented as follows:

[0110] Constrained state transition matrix: A'(i,j)=A(i,j)*M_seq(i,j)

[0111] In the formula, M_seq(i,j) is a sequence constraint matrix, which represents the legal transfer relationship between surgical procedure units. It is an n×n binary matrix (n is the number of procedure units); the matrix element M_seq(i,j) indicates whether the transfer from procedure unit i to j conforms to medical standards. A value of 1 indicates a legal transfer, and a value of 0 indicates an illegal transfer.

[0112] The expression for the constrained initial state probability π'(i) is: π'(i)=π(i)*M_init(i);

[0113] Here, M_init(i) is the initial state constraint vector, which encodes the rationality of each process unit as the starting unit of the operation. For example, M_init(1) = 1 means that the "operative preparation" unit can be used as the starting unit, while M_init(7) = 0 means that "gallbladder removal" should not be used as the starting unit of the operation. The specific values ​​of these constraint matrices are defined by medical experts according to the standard surgical procedure.

[0114] S13. The Viterbi algorithm is used to process the observation sequence of the surgical video to preliminarily infer the unit sequence of the surgical process.

[0115] This invention employs the Viterbi algorithm to verify the accuracy of the model's predictions. During the Viterbi algorithm's solution process, due to the complexity of video data or limitations of the model itself, localized short-term abnormal states may occur. For example, in a normal cholecystectomy surgery video, a short period might incorrectly predict the current stage as a stage inconsistent with the overall surgical procedure. To address this issue, when decoding the Viterbi optimal state using logarithmic domain probability (a crucial step in the Viterbi algorithm for calculating the most probable state sequence), the suboptimal state at each time step (i.e., the second most likely state besides the currently determined optimal state) and the confidence score of each candidate state at each time step (representing the credibility of each candidate state as the correct state) are additionally recorded. This additional recorded information provides a basis for subsequent optimization. For instance, when an optimal state is found to be clearly inconsistent with medical logic, the suboptimal state and confidence score can be referenced to re-evaluate and adjust the state at that time step, thereby improving the overall accuracy of the prediction results.

[0116] Figure 4 is a schematic diagram verifying the results of the surgical procedure analysis method of the present invention. Figure 4(a) shows a comparison between the ground truth distribution of surgical procedure units in video 80 and the prediction results of MK-HMM, with an accuracy rate of 96.98%, intuitively verifying the model's accuracy in dividing the seven standardized units (such as cholecystectomy steps 1-7). Figure 4(b) presents the temporal distribution of the actual stages in probabilistic form, reflecting the actual time proportion of each unit in the operation. Figure 4(c) shows the stage probability distribution predicted by MK-HMM, demonstrating the model's ability to analyze the temporal sequence of unit boundaries (such as the transition from stage 3 to 4).

[0117] This embodiment utilizes the Viterbi algorithm to preliminarily infer the sequence of surgical procedure units: the surgical video contains a series of complex image information, which can be regarded as an observation sequence. The Viterbi algorithm is an effective method for finding the most likely hidden state sequence in a Hidden Markov Model. In this invention, by processing the surgical video observation sequence with this algorithm and combining parameters such as the model's state transition probability and emission probability, the possible sequence of surgical procedure units in the video can be preliminarily inferred, that is, the order of the various stages of the surgery can be preliminarily predicted. For example, by analyzing the appearance of instruments and operational actions in the video, it can be inferred that the current stage of cholecystectomy is the Calot Triangle Dissection stage or the Gallbladder Packaging stage.

[0118] S14 optimizes the surgical procedure unit sequence by allocating abnormal time periods and correcting procedure units for parts that do not conform to medical logic, and outputs a standardized surgical procedure unit sequence.

[0119] In this embodiment, a dual optimization method is designed based on medical domain knowledge to significantly improve the robustness of state sequence prediction and consistency with medical constraints. The dual optimization method is designed according to the characteristics of laparoscopic surgery and includes abnormal time period allocation and process unit correction, specifically implemented as follows:

[0120] Abnormal Time Segment Allocation: When a short-term abnormal state segment 's' is detected during surgery, such as a small period in the surgical video where the judgment of the surgical stage is clearly incorrect or deviates from the standard procedure, a local time window is constructed to focus on this abnormal period. Then, the confidence level of the main states before and after the abnormal period (i.e., the reliability of the judgments on these states) and the state transition probability (i.e., the likelihood of transitioning from one state to another) are comprehensively considered. By solving a specific optimization problem, a weighted approach is used to redetermine the state assignments at each moment within the abnormal period. The advantage of this approach is that it makes the transitions between surgical states more natural and reasonable, conforming to the continuity and dynamic changes of the surgical procedure in actual operation, and avoiding abrupt and unreasonable state jumps.

[0121] Process Unit Correction: Some surgeries (such as cholecystectomy) have strict medical requirements regarding the sequence of surgical process units, with a specific order. Taking cholecystectomy as an example, the step "Gallbladder Packaging" (surgical process unit 5) must precede "Gallbladder Retraction" (surgical process unit 7). If, based on the initial surgical process unit sequence prediction, a situation is found that does not conform to this medical workflow logic, i.e., the state sequence has an incorrect order, then global constraint optimization is performed on the entire state sequence, replacing the illogical states with the suboptimal states for that time step. After this adjustment, the state sequence can better adapt to the medical workflow, ensuring that the entire surgical process unit sequence conforms to medical standards and requirements.

[0122] This invention designs a feature extraction engineering and dual optimization method specific to laparoscopic surgery, and encodes medical knowledge in mathematical form and integrates it into the state inference algorithm to construct an MK-HMM model. This achieves explicit modeling of the surgical process logic, while reducing inference time to the millisecond level, shortening model training time by more than 90%, and reducing computing resource requirements to a level that can be supported by ordinary workstations, making the deployment and application of the technology in the clinical environment more convenient and efficient.

[0123] Step S2 involves transforming the data obtained from surgical video analysis into a hierarchical data structure and establishing a bidirectional mapping relationship between three types of entities: time, instrument features, and process units.

[0124] Specifically, based on the data obtained from the surgical video analysis in step S1, a spatiotemporal feature mapping data model is designed, comprising time entities, instrument feature entities, and process unit entities. The spatiotemporal feature mapping data model designed in this invention defines three core entities:

[0125] 1) Time entity (T): Represents discrete time points and continuous time intervals in a laparoscopic surgery video sequence;

[0126] 2) Device Feature Entity (F): Composed of attributes such as timestamp, unique device identifier, device category code, location information, and device combination mode;

[0127] 3) Process unit entity (P): includes surgical process unit identifier, process unit switching marker and medical auxiliary annotation.

[0128] The core entities (time entity, instrument feature entity, and process unit entity) are encoded as nested key-value pair structures to construct a hierarchical data format, supporting fast access to multiple entities.

[0129] Furthermore, based on the three core entities, a bidirectional mapping relationship is established between these three entities: time-instrument feature, time-process unit, and process unit-instrument feature, forming a mapping network. The bidirectional mapping relationship specifically includes:

[0130] 1) Time-device feature mapping

[0131] Forward mapping (T→F): Given any time point t, quickly retrieve all instrument features existing at that time;

[0132] Reverse mapping (F→T): Given a specific instrument or combination of instruments, locate all time points or time periods in which it appears;

[0133] 2) Time-process unit mapping

[0134] Forward mapping (T→P): Given any time point t, determine the surgical procedure unit to which it belongs;

[0135] Reverse mapping (P→T): Given a specific surgical procedure unit, obtain its corresponding time interval [t_start, t_end];

[0136] 3) Process Unit-Device Feature Mapping

[0137] Forward mapping (P→F): Given a surgical procedure unit p, obtain all instrument features used in that unit;

[0138] Inverse mapping (F→P): Based on the instrument combination pattern, infer possible surgical procedure units.

[0139] This invention innovatively proposes a bidirectional mapping method for feature entities. By transforming instrument feature data and surgical procedure units acquired from the video analysis layer into specific data models, a bidirectional mapping relationship is established between the data models, thereby converting laparoscopic surgical videos into a hierarchical data structure. This innovative data organization method not only achieves standardized organization of feature data but also avoids repeated analysis of the same video, while enabling efficient data retrieval through the bidirectional mapping relationship.

[0140] The bidirectional mapping method proposed in this invention, compared with traditional video storage methods, not only preserves the key semantic information of the data but also achieves significantly optimized query performance. It provides O(1) access efficiency for each video frame through time-hash indexing and supports cross-video feature aggregation and temporal pattern analysis. This hierarchical data organization method is particularly suitable for processing large collections of laparoscopic surgical videos with wide durations and high content complexity, laying a solid technical foundation for the subsequent implementation of a multi-modal interactive retrieval layer.

[0141] Step S3: Based on the hierarchical data structure, three interactive search modes are designed: time-axis-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0142] This invention designs three complementary search modes, forming a multi-mode search method. This method allows for quick retrieval of relevant segments without manual searching, and supports secondary filtering and refinement of search results, providing a comprehensive search experience. The three interactive search modes include:

[0143] Time-axis guided search mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video segment; it supports searching across multiple video files and automatic continuous playback; when selecting a time point for retrieval in the time-series visualization interface, the system automatically displays the surgical procedure unit and instrument characteristics corresponding to that time point through a bidirectional mapping relationship;

[0144] Device-oriented retrieval mode: Based on the feature-time inverse mapping in the hierarchical data organization layer, a fast index of device type and timestamp is established, and the median filtering algorithm is fused to reduce noise in the device detection results; it supports single device or multi-device combination retrieval; after the user selects a single device or device combination, the system calls the inverse mapping function of the hierarchical data organization layer to obtain the corresponding time period, and generates a playback instruction queue to automatically play the video;

[0145] Process Unit-Oriented Retrieval Mode: The Hidden Markov Model based on medical knowledge constraints parses the surgical video into standardized process units, with each process unit associated with medical auxiliary annotations. After the user selects a process unit identifier, the system calls the process unit-time inverse mapping function to obtain the time interval corresponding to the process unit, and simultaneously calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit. The system locates the video to the start time and simultaneously displays the medical auxiliary annotation content clearly on the interface, highlighting the corresponding interval on the timeline.

[0146] In a preferred embodiment of the present invention, three interactive search modes—timeline-guided search, instrument-guided search, and process unit-guided search—are specifically described:

[0147] S31, Time-axis guided search mode

[0148] This invention implements a time-axis-based dual-endpoint precise retrieval mode, specifically including:

[0149] 1) Start-End Dual-Point Search: Doctors select precise start and end points on the time-series visualization interface, and the system automatically calculates the corresponding time intervals and plays the corresponding video segments;

[0150] 2) Cross-video continuous retrieval: Innovatively supports retrieval and automatic continuous playback across multiple video files, enabling seamless cross-video connection of segmented long surgical videos;

[0151] 3) Medical Information Display: During the selection process, key information such as the surgical procedure unit at that time point and the combination of instruments currently used is automatically displayed;

[0152] After receiving the time points t_start and t_end selected by the doctor in the visualization interface, the following steps are executed:

[0153] Locate video files: Identify the set of video files that contain the selected time interval;

[0154] Calculate the offset within the video: convert the global timestamp to a relative timestamp within each video file;

[0155] Build a playback queue: Generate a playback instruction queue of {video ID, start time, end time} and automatically play the video;

[0156] Seamless transition: Automatically switch to the next video in the video file set at the video switching point.

[0157] This precise retrieval model overcomes the limitations of existing technologies that rely on simple markings on progress bars based on timelines or event axes. It avoids the need to open video files one by one for retrieval, providing doctors with precise and rapid location capabilities based on medical semantics.

[0158] S32, Medical Device Guided Search Mode

[0159] This invention implements a guided retrieval mode centered on surgical instruments, specifically including:

[0160] 1) Instrument Center Mapping Retrieval: Establish a reverse mapping from instrument type to time point to achieve rapid location from instrument to time;

[0161] 2) Instrument combination mode search: Supports searching based on the combination mode of multiple instruments, such as the combined use scenario of "electrocautery hook + gripping forceps";

[0162] 3) Enhanced temporal filtering: The median filtering algorithm is integrated to reduce noise in the instrument detection results, improving the accuracy and stability of the retrieval.

[0163] Once the system obtains the device type or combination of devices selected by the doctor, it will perform the following steps:

[0164] Device selection processing: Receives single devices or combinations of multiple devices selected by doctors through the interactive interface, and forms search criteria;

[0165] Data mapping query: Call the reverse mapping function of the hierarchical data organization layer to obtain the corresponding time period;

[0166] Results processing and playback: Generate a video playback instruction queue and automatically play the video.

[0167] This innovative model incorporates information on surgical instrument usage, meeting the needs of surgeons to analyze specific instrument handling techniques. Through instrument-guided retrieval, doctors can quickly locate usage segments of specific instruments and directly play video clips of interest, significantly improving learning efficiency.

[0168] S33, Process Unit Guided Retrieval Mode

[0169] This invention implements a unit retrieval mode based on the semantics of surgical procedures, specifically including:

[0170] 1) Unit semantic mapping: Based on the aforementioned surgical temporal semantic parsing method, the video is divided and mapped to standardized surgical procedure units;

[0171] 2) Medical auxiliary annotations: Each process unit is accompanied by medical auxiliary annotations, such as the medical objectives, precautions and key operations of the unit;

[0172] After receiving the surgical procedure unit identifier selected by the doctor, perform the following steps:

[0173] Process unit selection: Select a single process unit or multiple consecutive process units via drop-down menus or a graphical interface;

[0174] Mapping from process unit to time interval: Call the P→T reverse mapping function to obtain the time interval corresponding to the process unit;

[0175] Information Acquisition: Use P→F mapping to obtain key device combination information and extract medical auxiliary annotation information;

[0176] Results presentation and playback: Locate the video to the start time, synchronously display the marked content, and highlight the corresponding interval on the timeline.

[0177] This process unit-based retrieval mode, by integrating medical semantic annotation, makes the retrieval results more intuitive and in line with the needs of medical professionals, which can greatly improve learning efficiency and shorten the learning time curve.

[0178] Step S4: Dynamic visualization of surgical timing is achieved through multi-track design and medical semantic visualization encoding.

[0179] Specifically, all the functions described above in this invention are implemented in a visual interactive manner, which makes it convenient for doctors to intuitively see the temporal characteristics of laparoscopic surgery videos and to conveniently and quickly perform interactive searches for video segments of interest.

[0180] A dynamic visualization diagram of time series features is shown below. Figure 5 As shown, step S4 specifically includes:

[0181] S41, Multi-track Display Design:

[0182] Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of instrument combinations is clearly displayed through vertical layering.

[0183] Adaptive time scale: Automatically adjusts the time scale density based on the total length of multiple videos and the current zoom level;

[0184] Color coding: Different color codes are used to distinguish different types of surgical instruments, improving visual recognition efficiency;

[0185] Density-perceptual representation: The frequency and intensity of instrument combinations are intuitively expressed through the density of points, revealing the high-frequency periods of surgical operations.

[0186] S42, Medical Semantic Visualization Coding:

[0187] Process unit background labeling: Different surgical process units are marked with semi-transparent color blocks, and medical semantics are encoded through color and transparency;

[0188] Key point highlighting: Automatically identifies and highlights key time points during the surgical process, such as decision points for instrument switching and tissue separation;

[0189] Cross-video boundary markers: The structure of multiple videos is clearly expressed through vertical dashed lines and video identifiers, making it easier to grasp the overall cross-video data.

[0190] This invention, based on a hierarchical data organization-based multimodal interactive retrieval method, innovatively constructs a "video analysis-data organization-multimodal retrieval" technical process, achieving efficient retrieval and intuitive presentation of laparoscopic surgical videos. This significantly enhances the application value of surgical videos in teaching, quality assessment, and skills analysis scenarios. Compared with existing technologies, this invention deeply integrates medical expertise to perform temporal modeling of surgical videos; it innovatively proposes a bidirectional mapping method to convert traditional surgical video data into hierarchical data that is easy to store and retrieve; and through multimodal retrieval methods and interactive feedback with dynamic visualization of temporal features, it provides medical personnel with a professional retrieval experience that meets their training and research needs.

[0191] In a preferred embodiment of the present invention, a laparoscopic surgery video retrieval and visualization system is provided, the system comprising:

[0192] The surgical video analysis module is used to analyze the instrument features in surgical videos using a deep learning model, and to perform temporal semantic parsing based on a hidden Markov model constrained by medical knowledge to divide the surgical process into standardized surgical procedure units; wherein, a surgical procedure unit is a standardized operation stage in the surgical process.

[0193] The hierarchical data organization module is used to transform the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument features, and process units.

[0194] The multi-mode interactive retrieval module is designed based on a hierarchical data structure to provide three interactive retrieval modes: time-axis-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0195] The visualization module is used to achieve dynamic visualization of surgical time sequences through multi-track design and medical semantic visualization encoding.

[0196] The laparoscopic surgery video retrieval and visualization system provided by the present invention is used to execute the laparoscopic surgery video retrieval and visualization methods provided in the foregoing embodiments. The laparoscopic surgery video retrieval and visualization methods have been described in detail in the foregoing embodiments, and will not be repeated here.

[0197] This invention provides a method and system for retrieval and visualization of laparoscopic surgical videos. It innovatively constructs a "video analysis-data organization-multi-modal retrieval" technical process, accurately analyzing and efficiently utilizing surgical video resources. At the "video analysis" level, it introduces a temporal semantic parsing method based on medical knowledge constraints. By explicitly modeling the surgical procedure logic and medical standards, it significantly reduces computational resource requirements and model training time. At the "data organization" level, it innovatively proposes a bidirectional mapping method for feature entities, transforming massive amounts of complex video data into efficient hierarchical data. At the "multi-modal retrieval" level, it designs a medical knowledge-driven multi-modal interactive retrieval method. Through multi-track display and medical semantic visualization encoding, it supports the presentation of multiple video information, providing surgeons with an intuitive and efficient surgical video learning experience. Through this complete technical process, this invention effectively solves the various limitations of existing technologies when dealing with complex laparoscopic surgical videos, significantly enhancing the application value of surgical videos in medical education and training.

[0198] Figure 6 A structural block diagram of the electronic device provided by the present invention, such as Figure 6As shown, the present invention also provides an electronic device, which 600 can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing device. The electronic device 600 includes a processor 601 and a memory 602, wherein the memory 602 stores a laparoscopic surgery video retrieval and visualization program 603.

[0199] In some embodiments, memory 602 may be an internal storage unit of a computer device, such as a hard disk or memory. In other embodiments, memory 602 may be an external storage device of a computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, memory 602 may include both internal and external storage units of the computer device. Memory 602 is used to store application software and various types of data installed on the computer device, such as program code for installing the computer device. Memory 602 can also be used to temporarily store data that has been output or will be output. In one embodiment, when the laparoscopic surgery video retrieval and visualization program 603 is executed by processor 601, the following steps are implemented:

[0200] Deep learning models are used to analyze instrument features in surgical videos, and a hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized surgical procedure units; where a surgical procedure unit is a standardized operation stage in the surgical process.

[0201] The data obtained from surgical video analysis is transformed into a hierarchical data structure, and a two-way mapping relationship is established between three types of entities: time, instrument characteristics, and process units.

[0202] Based on a hierarchical data structure, three interactive search modes are designed: time-axis-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0203] Dynamic visualization of surgical timelines is achieved through multi-track design and medical semantic visualization coding.

[0204] In some embodiments, processor 601 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 602 or process data, such as executing a laparoscopic surgery video retrieval and visualization program.

[0205] This embodiment also provides a computer-readable storage medium storing a laparoscopic surgery video retrieval and visualization program. When the laparoscopic surgery video retrieval and visualization program is executed by a processor, it performs the following steps:

[0206] Deep learning models are used to analyze instrument features in surgical videos, and a hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized surgical procedure units; where a surgical procedure unit is a standardized operation stage in the surgical process.

[0207] The data obtained from surgical video analysis is transformed into a hierarchical data structure, and a two-way mapping relationship is established between three types of entities: time, instrument characteristics, and process units.

[0208] Based on a hierarchical data structure, three interactive search modes are designed: time-axis-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode achieves rapid location by calling the corresponding bidirectional mapping relationship.

[0209] Dynamic visualization of surgical time sequence is achieved through multi-track design and medical semantic visualization encoding.

[0210] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for retrieving and visualizing laparoscopic surgery videos, characterized in that, include: Deep learning models are used to analyze instrument features in surgical videos, and a hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized surgical procedure units; where a surgical procedure unit is a standardized operation stage in the surgical process. The Hidden Markov Model (HMM) based on medical knowledge constraints performs temporal semantic parsing to divide standardized surgical procedure units. This includes: constructing multi-dimensional temporal features of the surgical video; these features include static instrument features, dynamic instrument features, specific instrument combination features, continuous instrument use features, and surgical progress features; using these multi-dimensional temporal features as the observation input to the HMM, and combining them with medical knowledge constraints to establish a medical knowledge-constrained HMM; processing the observation sequence of the surgical video using the Viterbi algorithm to initially infer the surgical procedure unit sequence; and optimizing the surgical procedure unit sequence for parts that do not conform to medical logic through abnormal time period allocation and procedure unit correction, outputting a standardized surgical procedure unit sequence. The expression for the hidden Markov model constrained by the medical knowledge is as follows: MK-HMM=(S, O, A, B, π, M) In the formula, MK-HMM represents a Hidden Markov Model constrained by medical knowledge; S = {S1, S2, ..., S...} n } represents the set of states, corresponding to n standardized surgical procedure units; O represents the set of observed states, A represents the state transition probability matrix, B represents the emission probability matrix, π represents the initial state probability vector, and M represents the set of medical knowledge constraints; The medical knowledge constraint set M includes the sequence constraint matrix M_seq and the initial state constraint vector M_init(i); M={M_seq, M_init(i)}; Based on the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constrained state transition matrix and the constrained initial state probability are obtained; The expression for the constrained state transition matrix A'(i,j) is: A'(i,j)=A(i,j)*M_seq(i,j) In the formula, M_seq(i,j) is a sequence constraint matrix, which represents the legal transfer relationship between surgical procedure units; the matrix element M_seq(i,j) indicates whether the transfer from procedure unit i to j conforms to medical standards, with a value of 1 indicating a legal transfer and a value of 0 indicating an illegal transfer; The expression for the constrained initial state probability π'(i) is: π'(i) = π(i) * M_init(i) In the formula, M_init(i) is the initial state constraint vector, which encodes the rationality of each process unit as the starting unit of the operation; The data obtained from surgical video analysis is transformed into a hierarchical data structure, and a two-way mapping relationship is established between three types of entities: time, instrument characteristics, and process units. Based on a hierarchical data structure, three interactive search modes are designed: time-axis-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode achieves rapid location by calling the corresponding bidirectional mapping relationship. Dynamic visualization of surgical time sequence is achieved through multi-track design and medical semantic visualization encoding.

2. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that, The method of using a deep learning model to analyze instrument features in surgical videos includes: Deep learning models were used to analyze the instrument features in laparoscopic surgery videos. By combining the sliding window mid-range filtering method to eliminate instantaneous misjudgments, the instrument characteristic data is output.

3. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that, The process of transforming data obtained from surgical video analysis into a hierarchical data structure and establishing a bidirectional mapping relationship between three types of entities—time, instrument characteristics, and process units—includes: Design a spatiotemporal feature mapping data model that includes time entities, instrument feature entities, and process unit entities, and encode the three types of core entities into a nested key-value pair structure to form a hierarchical data form; Establish bidirectional mapping relationships between three types of entities: time-instrument feature, time-process unit, and process unit-instrument feature, forming a mapping network.

4. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that, Based on a hierarchical data structure, three interactive retrieval modes are designed: time-axis-oriented retrieval, device-oriented retrieval, and process unit-oriented retrieval. Time-axis guided search mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video segment; it supports searching across multiple video files and automatic continuous playback; when selecting a time point for retrieval in the time-series visualization interface, the system automatically displays the surgical procedure unit and instrument characteristics corresponding to that time point through a bidirectional mapping relationship; Medical Device-Oriented Retrieval Mode: Based on the feature-time inverse mapping in the hierarchical data structure, a fast index of medical device type and timestamp is established, and the median filtering algorithm is fused to reduce noise in the medical device detection results; it supports single medical device or multi-medical device combination retrieval; after the user selects a single medical device or medical device combination, the system calls the inverse mapping function of the hierarchical data organization layer to obtain the corresponding time period, and generates a playback instruction queue to automatically play the video; Process Unit-Oriented Retrieval Mode: The Hidden Markov Model based on medical knowledge constraints parses the surgical video into standardized process units, with each process unit associated with medical auxiliary annotations. After the user selects a process unit identifier, the system calls the process unit-time inverse mapping function to obtain the time interval corresponding to the process unit, and simultaneously calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit. The system locates the video to the start time and simultaneously displays the medical auxiliary annotation content clearly on the interface, highlighting the corresponding interval on the timeline.

5. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that, The multi-track display design includes: Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of instrument combinations is clearly displayed through vertical layering. Adaptive time scale: Automatically adjusts the time scale density based on the total length of multiple videos and the current zoom level; Color coding: Different color codes are used to distinguish different types of surgical instruments, improving visual recognition efficiency; Density-perceptual representation: Visually expressing the frequency and intensity of instrument combinations through point density, revealing high-frequency periods of surgical procedures. The medical semantic visualization encoding includes: Process unit background labeling: Different surgical process units are marked with semi-transparent color blocks, and medical semantics are encoded through color and transparency; Key point highlighting: Automatically identifies and highlights key time points during the surgical procedure; Cross-video boundary markers: Clearly express the structure of multiple videos using vertical dashed lines and video identifiers.

6. A laparoscopic surgery video retrieval and visualization system, the system being used to execute the laparoscopic surgery video retrieval and visualization method according to any one of claims 1-5, characterized in that, The system includes: The surgical video analysis module is used to analyze the instrument features in surgical videos using a deep learning model, and to perform temporal semantic parsing based on a hidden Markov model constrained by medical knowledge to divide the surgical process into standardized surgical procedure units; wherein, a surgical procedure unit is a standardized operation stage in the surgical process. The hierarchical data organization module is used to transform the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument features, and process units. The multi-mode interactive retrieval module is designed based on a hierarchical data structure to provide three interactive retrieval modes: time-axis-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid location by calling the corresponding bidirectional mapping relationship. The visualization module is used to achieve dynamic visualization of surgical time sequences through multi-track design and medical semantic visualization encoding.

7. An electronic device, Its features are, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the laparoscopic surgery video retrieval and visualization method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the laparoscopic surgery video retrieval and visualization method according to any one of claims 1 to 5.

Images