Laparoscopic surgery video retrieval and visualization method and system

Abstract

The invention relates to a laparoscopic surgery video retrieval and visualization method and system. According to the method, efficient surgery video retrieval is achieved through video analysis, data organization and multi-mode retrieval. In a video analysis stage, a deep learning model is combined with a hidden Markov model constrained by medical knowledge, video content is automatically analyzed, standardized process units are divided, computing resource consumption is reduced, and training time is shortened. In the data organization stage, feature entity bidirectional mapping is provided, unstructured video data is converted into a hierarchical data structure with medical semantics, and storage and retrieval are facilitated. In the multi-mode retrieval stage, three retrieval modes of time axis guiding, instrument guiding and process unit guiding are designed, and dynamic visualization is achieved in combination with multi-track display and medical semantic visualization coding. Through a multi-mode retrieval method and time sequence characteristic dynamic visual interaction feedback, specialized retrieval experience meeting training and research requirements of medical personnel is provided for the medical personnel.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a laparoscopic surgery video retrieval and visualization method and system. Background Art

[0002] Laparoscopic surgery, with its significant advantages of minimal trauma and rapid recovery, has been widely used clinically and has become a key technique in modern surgery. However, its long learning curve and high skill requirements for physicians have made surgical videos increasingly important as teaching resources. With the rapid increase in surgical procedures, video data is exploding. The management and retrieval of laparoscopic surgical videos faces a dilemma between the sheer volume of data and low efficiency, making it a critical issue that needs to be addressed.

[0003] Currently, existing techniques for retrieving laparoscopic surgery videos primarily rely on progress bar marking, adding simple notes to the original video progress bar and locating the segments through manual annotation. Furthermore, deep learning models are also being explored for surgical video analysis and retrieval, using object detection algorithms to identify surgical instruments.

[0004] Existing laparoscopic surgery video retrieval technology has the following problems: (1) The data organization method is simple, mostly adding notes to the original video progress bar; (2) The retrieval mode is limited to the video progress bar, requiring the video to be traversed one by one, and it is difficult to support complex queries and lacks a holistic presentation of multi-video information. These problems seriously restrict the application value of surgical videos. Summary of the Invention

[0005] In view of this, it is necessary to provide a laparoscopic surgery video retrieval and visualization method and system to solve the above-mentioned problems in the prior art.

[0006] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a laparoscopic surgery video retrieval and visualization method, comprising:

[0007] A deep learning model is used to analyze the instrument features in surgical videos. A hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure.

[0008] The data obtained from surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between three types of entities: time, instrument characteristics, and process units;

[0009] Based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode can achieve rapid positioning by calling the corresponding bidirectional mapping relationship.

[0010] Dynamic visualization of surgical sequence is achieved through multi-track design and medical semantic visualization encoding.

[0011] Preferably, the use of a deep learning model to analyze instrument features in surgical videos includes:

[0012] Use deep learning models to analyze instrument features in laparoscopic surgery videos;

[0013] Combined with the sliding window median filtering method, instantaneous misjudgment is eliminated and the device feature data is output.

[0014] Preferably, the hidden Markov model based on medical knowledge constraints performs temporal semantic parsing and divides the standardized surgical process units, including:

[0015] Constructing multi-dimensional temporal features of surgical videos; wherein, multi-dimensional temporal features of surgical videos include static instrument features, dynamic instrument features, specific instrument combination features, continuous instrument use features, and surgical progress features;

[0016] The multi-dimensional temporal features of surgical videos are used as observation inputs in the hidden Markov model, combined with medical knowledge constraints, to establish a medical knowledge-constrained hidden Markov model.

[0017] The Viterbi algorithm is used to process the observation sequence of the surgical video and preliminarily infer the sequence of surgical process units;

[0018] For the parts of the surgical process unit sequence that do not conform to medical logic, optimization is performed through abnormal time period allocation and process unit correction to output a standardized surgical process unit sequence.

[0019] Preferably, the expression of the hidden Markov model constrained by medical knowledge is:

[0020] MK-HMM=(S, O, A, B, π, M)

[0021] Where MK-HMM represents the hidden Markov model constrained by medical knowledge; S = {S1, S2, ..., S n} is a state set corresponding to n standardized surgical process units; O represents the observed state set, A represents the state transition probability matrix, B represents the emission probability matrix, π represents the initial state probability vector, and M represents the medical knowledge constraint set;

[0022] Wherein, the medical knowledge constraint set M includes a sequence constraint matrix M_seq and an initial state constraint vector M_init(i); M={M_seq, M_init(i)};

[0023] According to the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constraint state transfer matrix and the constraint initial state probability are obtained;

[0024] The expression of the constraint state transfer matrix A'(i, j) is:

[0025] A'(i,j)=A(i,j)*M_seq(i,j)

[0026] Where M_seq(i, j) is the sequence constraint matrix, which represents the legal transfer relationship between surgical process units. The matrix element M_seq(i, j) indicates whether the transfer from process unit i to j complies with medical standards. The value of 1 indicates a legal transfer, and the value of 0 indicates an illegal transfer.

[0027] The expression of the constrained initial state probability π'(i) is:

[0028] π'(i)=π(i)*M_init(i)

[0029] Where M_init(i) is the initial state constraint vector, encoding the rationality of each process unit as the starting unit of the surgery. For example, M_init(1) = 1 indicates that the "surgery preparation" unit can be used as the starting unit, while M_init(7) = 0 indicates that "gallbladder removal" should not be used as the starting unit of the surgery. The specific values ​​of these constraint matrices are defined by medical experts based on standard surgical procedures.

[0030] Preferably, the data obtained from the surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between three types of entities: time, instrument characteristics, and process units, including:

[0031] Design a spatiotemporal feature mapping data model that includes time entities, device feature entities, and process unit entities. Encode the three core entities into a nested key-value pair structure to form a hierarchical data format.

[0032] A bidirectional mapping relationship is established between the three entities of time-device characteristics, time-process unit, and process unit-device characteristics to form a mapping network.

[0033] Preferably, the three interactive search modes of designing timeline-oriented search, instrument-oriented search and process unit-oriented search based on the hierarchical data structure include:

[0034] Timeline-guided retrieval mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video clip; supports retrieval and automatic continuous playback across multiple video files; when a time point is selected for retrieval in the time sequence visualization interface, the system automatically displays the corresponding surgical process unit and instrument characteristics through a bidirectional mapping relationship;

[0035] Device-oriented retrieval mode: Based on the feature-time reverse mapping in the hierarchical data structure, a fast index of device type and timestamp is established, and a median filter algorithm is integrated to reduce noise in device detection results. The system supports retrieval of single devices or multiple device combinations. After the user selects a single device or a combination of devices, the system calls the reverse mapping function of the hierarchical data organization layer to obtain the corresponding time period, generates a playback instruction queue, and automatically plays the video.

[0036] Process unit-oriented retrieval mode: The hidden Markov model based on medical knowledge constraints parses the surgical video into standardized process units, and each process unit is associated with medical auxiliary annotations; after the user selects the process unit identifier, the system calls the process unit-time reverse mapping function to obtain the time interval corresponding to the process unit, and at the same time calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit; the system locates the video to the start time, and synchronously displays the medical auxiliary annotation content clearly on the interface, and highlights the corresponding interval on the timeline.

[0037] Preferably, the multi-track display design includes:

[0038] Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of the instrument combination is clearly displayed through vertical layering;

[0039] Time scale adaptation: automatically adjusts the time scale density based on the total length of multiple videos and the current zoom level;

[0040] Color coding: Use different color codes to distinguish different types of surgical instruments to improve visual recognition efficiency;

[0041] Density perception expression: The density of points is used to intuitively express the frequency and intensity of instrument combinations, revealing the high-frequency periods of surgical operations:

[0042] The medical semantic visualization coding includes:

[0043] Process unit background identification: Use translucent color blocks to mark different surgical process units, encoding medical semantics through color and transparency;

[0044] Key point highlighting: automatically identifies and highlights key time points during surgery;

[0045] Cross-video boundary marking: vertical dotted lines and video logos are used to clearly express the multi-video structure.

[0046] In a second aspect, an embodiment of the present invention provides a laparoscopic surgery video retrieval and visualization system, comprising:

[0047] The surgical video analysis module uses a deep learning model to analyze instrument features in surgical videos, performs temporal semantic parsing based on a hidden Markov model constrained by medical knowledge, and divides the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure.

[0048] Hierarchical data organization module, used to convert the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument characteristics, and process units;

[0049] A multi-mode interactive retrieval module is used to design three interactive retrieval modes based on a hierarchical data structure: timeline-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid positioning by calling the corresponding bidirectional mapping relationship.

[0050] A visualization module is used to achieve dynamic visualization of surgical sequences through multi-track design and medical semantic visualization encoding.

[0051] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, wherein:

[0052] The memory is used to store programs;

[0053] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the laparoscopic surgery video retrieval and visualization method as described in the embodiment of the first aspect of the present invention.

[0054] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the laparoscopic surgery video retrieval and visualization method as described in the embodiment of the first aspect of the present invention.

[0055] The laparoscopic surgery video retrieval and visualization method and system provided by the present invention have the following beneficial effects compared with the existing technology:

[0056] This invention is based on a multimodal interactive retrieval method for hierarchical data organization, and innovatively constructs a "video analysis-data organization-multimodal retrieval" technical process, achieving efficient retrieval and intuitive presentation of laparoscopic surgery videos, greatly enhancing the application value of surgical videos in scenarios such as teaching, quality assessment, and skill analysis. Compared with existing technologies, this invention deeply integrates medical expertise to perform temporal modeling of surgical videos; innovatively proposes a two-way mapping method to convert traditional surgical video data into hierarchical data that is easy to store and retrieve; through multimodal retrieval methods and interactive feedback of dynamic visualization of temporal features, it provides medical personnel with a professional retrieval experience that meets their training and research needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flow chart of the laparoscopic surgery video retrieval and visualization method provided by the present invention;

[0058] Figure 2 A schematic diagram of the laparoscopic surgery video retrieval and visualization method provided by the present invention;

[0059] FIG3( a ) is a state transition probability matrix diagram of the MK-HMM model provided by the present invention;

[0060] FIG3( b ) is a transmission probability matrix diagram of the MK-HMM model provided by the present invention;

[0061] Figure 4(a) is a comparison of the actual annotation of the surgical process unit and the MK-HMM prediction results;

[0062] Figure 4(b) is the time distribution diagram of the real surgical process unit;

[0063] Figure 4(c) is a time distribution diagram of the predicted surgical process unit;

[0064] Figure 5 A schematic diagram of dynamic visualization of the timing characteristics provided by the present invention;

[0065] Figure 6 This is a structural block diagram of the electronic device provided by the present invention.

[0066] The color differences in Figures 3 to 5 are used to distinguish technical features. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0068] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] Most existing surgical video retrieval and analysis technologies lack targeted search capabilities for surgical instrument usage, failing to meet doctors' needs for analyzing specific instrument manipulation techniques. Furthermore, semantic analysis of surgical procedures is not fully integrated with medical semantic annotation, resulting in unintuitive search results that meet medical professional needs. Furthermore, there is a lack of efficient, clear, and interactively searchable visualization methods for surgical video temporal features, making it difficult for doctors to quickly and accurately obtain key information.

[0070] In view of this, the present invention aims to provide a laparoscopic surgery video retrieval and visualization method. A guided retrieval mode centered on surgical instruments is implemented, and by establishing a reverse mapping from instrument type to time point, etc., it is convenient for doctors to quickly locate and play specific instrument usage clips. A unit retrieval mode based on surgical process semantics is constructed, and the video is mapped to a standardized surgical process unit and medical auxiliary annotations are attached to make the retrieval results more in line with medical professional needs. In addition, through key technologies such as multi-track display design and medical semantic visualization coding, the timing characteristics of surgical videos are presented in a visual interactive manner, which is convenient for doctors to intuitively observe and quickly interactively retrieve video clips of interest. The following will be explained and introduced through multiple embodiments.

[0071] Figure 1 This is a flow chart of the laparoscopic surgery video retrieval and visualization method provided by the present invention. Figure 2 The schematic diagram of the laparoscopic surgery video retrieval and visualization method provided by the present invention is shown in FIG. Figure 1 and Figure 2 The laparoscopic surgery video retrieval and visualization method provided by the present invention comprises at least the following steps:

[0072] Step S1: Use a deep learning model to analyze the instrument features in the surgical video, perform temporal semantic parsing based on a hidden Markov model constrained by medical knowledge, and divide the standardized surgical process units;

[0073] The surgical process unit is a standardized operation stage during the surgical process. In the description of the embodiment of the present invention, the surgical process unit is also referred to as a process unit for short.

[0074] Specifically, the present invention uses a deep learning model to analyze the instrument features in laparoscopic surgery videos.

[0075] In this embodiment, a deep learning model based on the D-FINE algorithm framework can be used to analyze the characteristics of instruments in laparoscopic surgery videos. This model, combined with a sliding window median filter method to eliminate transient misjudgments, outputs instrument feature data, providing data support for semantic parsing of surgical sequence sequences. This method is not limited to a specific deep learning network; any object detection algorithm that can accurately identify surgical instruments is applicable.

[0076] In a preferred embodiment of the present invention, in step S1, temporal semantic parsing is performed based on a Medical Knowledge-constrained Hidden Markov Model (MK-HMM) to divide the standardized surgical process into units, specifically including:

[0077] S11, constructing multi-dimensional temporal features of the surgical video; wherein the multi-dimensional temporal features of the surgical video include static instrument features, dynamic instrument features, specific instrument combination features, continuous instrument use features, and surgical progress features;

[0078] Specifically, this embodiment designs a feature construction project based on the timing characteristics of laparoscopic surgery to comprehensively capture the instrument combination pattern in the surgical process:

[0079] 1) Static device characteristics: Record the status and combination of devices at each moment to form a basic feature matrix:

[0080] F_static(t)=[S_1(t), S_2(t),..., S_n(t)]

[0081] Among them, S_i(t) represents the existence status (0 or 1) of the i-th tool at time t.

[0082] 2) Dynamic characteristics of laparoscopic surgical instruments: The dynamic change characteristics of the instrument combination are captured through the time-series sliding window W(t, k):

[0083] F_dynamic(t)=[C_1(t), C_2(t),..., C_n(t), D(t), M(t), A(t)]

[0084] Among them, C_i(t) represents the number of state changes of tool i in the window W(t, k) (tool switching frequency); D(t) represents the number of different tools used in the window (tool usage diversity); M(t) represents the tool with the highest frequency of use in the window (dominant tool); A(t) represents the average number of tools used simultaneously in the window.

[0085] 3) Laparoscopic instrument combination characteristics: This example defines five common instrument combination patterns and their durations in laparoscopic surgery. However, this example is not limited to the following five instrument combination patterns. Different numbers of instrument combination patterns can be defined according to the specific type of laparoscopic surgery:

[0086] F_comb(t)=[C_{1,3}(t),C_{1,7}(t),C_{1,6}(t),C_{1,2}(t),C_{1,5}(t)]

[0087] F_dur(t)=[D_{1,3}(t), D_{1,7}(t), D_{1,6}(t), D_{1,2}(t), D_{1,5}(t)]

[0088] Where: 1-7 represent grasping forceps, bipolar coagulation device, electric hook, scissors, clamp applier, irrigator, and specimen bag; C_{i, j}(t) represents the combined usage status of instruments i and j at time t; D_{i, j}(t) represents the continuous usage duration of the combination.

[0089] 4) Continuous use characteristics of instruments: Calculate the continuous use time of each instrument to capture the continuous characteristics of the instrument combination in laparoscopic surgery:

[0090] F_cont(t)=[L_1(t), L_2(t),..., L_n(t)]

[0091] Among them, L_i(t) represents the number of consecutive frames of use of tool i before time t.

[0092] 5) Surgery progress feature: Introduce the time progress feature to represent the relative position of the surgery in the overall process:

[0093] F_prog(t)=t / T_total

[0094] Where T_total is the total number of frames of the surgical video.

[0095] Through the above-constructed features, the present invention converts the instrument combination in laparoscopic surgery into a multi-dimensional feature vector, which comprehensively describes the temporal dynamic characteristics of the instruments during the operation.

[0096] S12, taking the multi-dimensional temporal features of the surgical video as the observation input in the hidden Markov model, combined with medical knowledge constraints, a medical knowledge constrained hidden Markov model is established.

[0097] Specifically, the MK-HMM model consists of a state set, an observation set, a state transition probability matrix, an observation probability matrix, and an initial state probability distribution. The state set represents the different stages or operating states of the surgical process, while the observation set is the multidimensional temporal features extracted from the surgical video. The parameters of the MK-HMM model are shown in Figures 3(a) and 3(b).

[0098] The constrained state transition matrix imposes constraints based on medical knowledge and the actual surgical process. For example, during a normal surgical procedure, certain operations must be performed in a specific order. Therefore, in the state transition matrix, the probabilities of state transitions that do not conform to this order are set to extremely low values ​​or even zero. This guides the model to learn state transition patterns that conform to medical logic.

[0099] Initial state probability: Incorporating common medical knowledge, determine the initial probability distribution of each state at the start of the surgery. For example, in most surgeries, the initial state is usually the preparation phase, so the state corresponding to the preparation phase will have a higher probability in the initial state probability distribution.

[0100] In this embodiment, the expression of the medical knowledge-constrained hidden Markov model is:

[0101] MK-HMM=(S, O, A, B, π, M)

[0102] Where MK-HMM represents the hidden Markov model constrained by medical knowledge; S = {S1, S2, ..., S n} is a state set, corresponding to n standardized surgical process units; in this embodiment, n=7, corresponding to 7 standardized surgical process units (taking the seven surgical process units of cholecystectomy as an example).

[0103] O represents the set of observed states, which is obtained by clustering the feature space;

[0104] A represents the state transition probability matrix, A(i, j) = P(S_t = j | S_{t-1} = i);

[0105] B represents the emission probability matrix, B(i, k) = P(O_t = k | S_t = i);

[0106] π represents the initial state probability vector;

[0107] M represents a medical knowledge constraint set; the medical knowledge constraint set M includes a sequence constraint matrix M_seq and an initial state constraint vector M_init(i); M={M_seq, M_init(i)};

[0108] According to the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constraint state transfer matrix and the constraint initial state probability are obtained;

[0109] Compared with the traditional Hidden Markov Model (HMM), the innovation of the MK-HMM model in the embodiment of the present invention is the introduction of the medical knowledge constraint set M, which is specifically implemented as follows:

[0110] Constrained state transfer matrix: A'(i, j) = A(i, j) * M_seq(i, j)

[0111] Where M_seq(i, j) is a sequence constraint matrix that represents the legal transfer relationship between surgical process units. It is an n×n binary matrix (n is the number of process units). The matrix element M_seq(i, j) indicates whether the transfer from process unit i to j complies with medical standards. The value of 1 indicates a legal transfer, and the value of 0 indicates an illegal transfer.

[0112] The expression of the constrained initial state probability π'(i) is: π'(i) = π(i)*M_init(i);

[0113] M_init(i) is the initial state constraint vector, encoding the rationality of each process unit as the starting unit of the operation. For example, M_init(1) = 1 indicates that the "operation preparation" unit can be used as the starting unit, while M_init(7) = 0 indicates that "gallbladder removal" should not be used as the starting unit of the operation. The specific values ​​of these constraint matrices are defined by medical experts based on standard surgical procedures.

[0114] S13, using the Viterbi algorithm to process the observation sequence of the surgical video and preliminarily infer the surgical process unit sequence.

[0115] The present invention uses the Viterbi algorithm to verify the accuracy of the model prediction. During the Viterbi algorithm solution process, due to the complexity of the video data or some limitations of the model itself, local short-term abnormal states may occur. For example, in a normal cholecystectomy video, the current stage may be mistakenly predicted as a stage that does not conform to the overall surgical process for a short period of time. To solve this problem, when using logarithmic domain probability to perform Viterbi optimal state decoding (this is the key step in the Viterbi algorithm to calculate the most likely state sequence), the suboptimal state of each time step (that is, the state with the second highest probability besides the current state determined to be optimal) and the confidence score of each candidate state at each moment (indicating the degree of confidence of each candidate state as the correct state) are additionally recorded. This additional recorded information can provide a basis for subsequent optimization. For example, when it is found that a certain optimal state clearly does not conform to medical logic, the suboptimal state and confidence score can be referenced to re-evaluate and adjust the state of that time step to improve the accuracy of the overall prediction result.

[0116] Figure 4 is a schematic diagram of the results of the surgical process analysis method of the present invention. Figure 4(a) shows the comparison of the ground truth distribution of the surgical process units in video 80 and the MK-HMM prediction results, with an accuracy of 96.98%, visually verifying the accuracy of the model's division of 7 standardized units (such as cholecystectomy steps 1-7). Figure 4(b) presents the time distribution of the actual stages in a probabilistic form, reflecting the actual time proportion of each unit in the operation. Figure 4(c) shows the stage probability distribution predicted by MK-HMM, reflecting the model's ability to parse the temporal sequence of unit boundaries (such as the transition from stage 3 to stage 4).

[0117] This embodiment uses the Viterbi algorithm to preliminarily infer the sequence of surgical process units: the surgical video contains a series of complex image information, which can be regarded as an observation sequence. The Viterbi algorithm is an effective method for finding the most likely hidden state sequence in a hidden Markov model. In the present invention, the surgical video observation sequence is processed by the algorithm, and combined with parameters such as the state transition probability and emission probability of the model, the corresponding possible surgical process unit sequence in the video can be preliminarily inferred, that is, the sequence of each stage of the operation can be preliminarily inferred. For example, by analyzing the appearance of instruments, operating actions and other observation information in the video, it is inferred that the current stage of the cholecystectomy surgery is the Calot Triangle Dissection stage, the Gallbladder Packaging stage, etc.

[0118] S14, for the part of the surgical process unit sequence that does not conform to medical logic, optimization is performed through abnormal time period allocation and process unit correction, and a standardized surgical process unit sequence is output.

[0119] In this embodiment, a dual optimization method is designed based on medical domain knowledge to significantly improve the robustness of state sequence prediction and consistency with medical constraints. The dual optimization method is designed based on the characteristics of the laparoscopic surgery process and includes abnormal time period allocation and process unit correction. The specific implementation is as follows:

[0120] Abnormal time period allocation: When a short abnormal state segment s is detected during the surgical process, such as a short period in the surgical video where the judgment of the surgical stage is obviously incorrect or does not conform to the normal process, a local time window is constructed to focus on this abnormal period. Then, the confidence level of the main states before and after the abnormal period (i.e., the reliability of the judgment of these states) and the state transition probability (i.e., the probability of changing from one state to another) are comprehensively considered. By solving a specific optimization problem, a weighted approach is used to re-determine the state attribution at each moment within the abnormal period. The advantage of this is that it can make the transition of surgical states more natural and reasonable, conforming to the continuity and dynamic changes of the process in actual surgical operations, and avoiding abrupt and unreasonable state jumps.

[0121] Process unit correction: Because some surgeries (such as cholecystectomy) have strict medical requirements for the order of surgical process units, there is a clear order of precedence. Taking cholecystectomy as an example, the step "GallbladderPackaging" (surgical process unit 5) must be performed before "GallbladderRetraction" (surgical process unit 7). If, based on the preliminary prediction of the surgical process unit sequence, it is found that there are situations that do not conform to the logic of this medical process, that is, the state sequence has an incorrect order, then a global constrained optimization is performed on the entire state sequence, replacing the illogical state with the suboptimal state for that time step. After this adjustment, the state sequence can better adapt to the medical process, so that the entire surgical process unit sequence meets medical standards and requirements.

[0122] The present invention designs a feature extraction project and dual optimization method specific to laparoscopic surgery, encodes medical domain knowledge in mathematical form and integrates it into a state inference algorithm to construct an MK-HMM model, thereby achieving explicit modeling of the surgical process logic. Meanwhile, the inference time is reduced to milliseconds, the model training time is shortened by more than 90%, and the computing resource requirements are reduced to a level that can be supported by ordinary workstations, making the deployment and application of the technology in clinical environments more convenient and efficient.

[0123] In step S2, the data obtained from the surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between the three entities of time, instrument characteristics, and process units.

[0124] Specifically, based on the data obtained from the surgical video analysis in step S1, a spatiotemporal feature mapping data model is designed, which includes a time entity, an instrument feature entity, and a process unit entity. The spatiotemporal feature mapping data model designed by the present invention defines three core entities:

[0125] 1) Temporal entity (T): represents discrete time points and continuous time intervals in laparoscopic surgery video sequences;

[0126] 2) Device feature entity (F): consists of attributes such as timestamp, device unique identifier, device category code, location information, and device combination mode;

[0127] 3) Process unit entity (P): includes surgical process unit identifier, process unit switching mark and medical auxiliary annotation.

[0128] Among them, the core entities (time entity, equipment feature entity and process unit entity) are encoded as a nested key-value pair structure to build a hierarchical data form, supporting fast access to multiple entities.

[0129] Furthermore, based on the three core entities, bidirectional mapping relationships are established between the three entities of time-device characteristics, time-process unit, and process unit-device characteristics to form a mapping network. The bidirectional mapping relationships specifically include:

[0130] 1) Time-device feature mapping

[0131] Forward mapping (T→F): Given any time point t, quickly retrieve all device features that exist at that moment;

[0132] Reverse mapping (F→T): Given a specific device or device combination, locate all the time points or time periods in which it appears;

[0133] 2) Time-Process Unit Mapping

[0134] Forward mapping (T→P): Given any time point t, determine the surgical process unit to which it belongs;

[0135] Reverse mapping (P→T): Given a specific surgical process unit, obtain its corresponding time interval [t_start, t_end];

[0136] 3) Process unit-device feature mapping

[0137] Forward mapping (P→F): Given a surgical procedure unit p, obtain the features of all instruments used in the unit;

[0138] Reverse mapping (F→P): Based on the instrument combination pattern, possible surgical process units are inferred.

[0139] This paper innovatively proposes a bidirectional mapping method for feature entities. By converting instrument feature data and surgical process units acquired from the video analysis layer into specific data models, a bidirectional mapping relationship between the data models is established, thereby converting laparoscopic surgical videos into a hierarchical data structure. This innovative data organization method not only standardizes the feature data, but also avoids repeated analysis of the same video. Furthermore, the bidirectional mapping relationship enables efficient data retrieval.

[0140] Compared to traditional video storage methods, the bidirectional mapping method proposed in this paper not only preserves the key semantic information of the data but also achieves significantly improved query performance. It provides O(1) access efficiency for each video frame through a time hash index and supports cross-video feature aggregation and temporal pattern analysis. This hierarchical data organization method is particularly suitable for processing large collections of laparoscopic surgery videos with a wide range of durations and high content complexity, laying a solid technical foundation for the subsequent implementation of a multimodal interactive retrieval layer.

[0141] In step S3, based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search; each search mode achieves rapid positioning by calling the corresponding bidirectional mapping relationship.

[0142] This invention designs three complementary search modes to form a multi-modal search method. It can quickly retrieve interesting fragments without manual searching one by one, and supports secondary screening and refinement of search results, realizing a coarse-to-fine search experience. The three interactive search modes include:

[0143] Timeline-guided retrieval mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video clip; supports retrieval and automatic continuous playback across multiple video files; when a time point is selected for retrieval in the time sequence visualization interface, the system automatically displays the corresponding surgical process unit and instrument characteristics through a bidirectional mapping relationship;

[0144] Device-oriented retrieval mode: Based on the feature-time reverse mapping in the hierarchical data organization layer, a fast index of device type and timestamp is established, and a median filter algorithm is integrated to reduce noise in device detection results. The system supports retrieval of single devices or multiple device combinations. After the user selects a single device or a combination of devices, the system calls the reverse mapping function in the hierarchical data organization layer to obtain the corresponding time period and generates a playback instruction queue to automatically play the video.

[0145] Process unit-oriented retrieval mode: The hidden Markov model based on medical knowledge constraints parses the surgical video into standardized process units, and each process unit is associated with medical auxiliary annotations; after the user selects the process unit identifier, the system calls the process unit-time reverse mapping function to obtain the time interval corresponding to the process unit, and at the same time calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit; the system locates the video to the start time, and synchronously displays the medical auxiliary annotation content clearly on the interface, and highlights the corresponding interval on the timeline.

[0146] In a preferred embodiment of the present invention, three interactive search modes, namely, timeline-oriented search, instrument-oriented search, and process unit-oriented search, are specifically described:

[0147] S31, timeline-oriented search mode

[0148] The present invention realizes a double-endpoint precise retrieval mode based on the time axis, specifically including:

[0149] 1) Start-end dual-point search: The doctor selects the precise start and end points on the time series visualization interface, and the system automatically calculates the corresponding time interval and plays the corresponding video segment;

[0150] 2) Cross-video continuous retrieval: Innovatively supports retrieval and automatic continuous playback across multiple video files, achieving seamless cross-video connection of segmented long-duration surgical videos;

[0151] 3) Medical information display: During the site selection process, key information such as the surgical process unit at that time point and the currently used instrument combination will be automatically displayed;

[0152] After receiving the time points t_start and t_end selected by the doctor on the visual interface, the following steps are performed:

[0153] Locate video files: determine the set of video files that contain the selected time interval;

[0154] Calculate the offset within the video: convert the global timestamp into a relative timestamp within each video file;

[0155] Build a play queue: Generate a play instruction queue of {video ID, start time, end time} and automatically play the video;

[0156] Transition processing: Automatically switch to the next video in the video file set at the video switching point.

[0157] This precise retrieval model overcomes the limitations of existing technologies based on simple markings on the progress bar of the timeline or event axis, avoids the retrieval method of opening video files one by one, and provides doctors with precise and rapid positioning capabilities based on medical semantics.

[0158] S32, instrument-guided retrieval mode

[0159] The present invention implements a guided search mode centered on surgical instruments, specifically including:

[0160] 1) Device center mapping retrieval: Establish a reverse mapping from device type to time point to achieve rapid positioning from device to time;

[0161] 2) Instrument combination mode search: supports searches based on the combination modes of multiple instruments, such as the combination usage scenario of "electric coagulation hook + grasping forceps";

[0162] 3) Enhanced time series filtering: The median filtering algorithm is integrated to reduce noise in device detection results, improving the accuracy and stability of retrieval;

[0163] After the system obtains the device type or device combination selected by the doctor, it performs the following steps:

[0164] Instrument selection processing: receiving a single instrument or a combination of instruments selected by the doctor through the interactive interface to form query conditions;

[0165] Data mapping query: call the reverse mapping function of the hierarchical data organization layer to obtain the corresponding time period;

[0166] Result processing and playback: Generate a video playback instruction queue and automatically play the video.

[0167] This model innovatively incorporates surgical instrument usage, meeting surgeons' needs for analyzing specific instrument manipulation techniques. Through instrument-guided search, doctors can quickly locate specific instrument usage clips and directly play the video clips of interest, greatly improving learning efficiency.

[0168] S33, process unit oriented search mode

[0169] The present invention implements a unit retrieval mode based on surgical procedure semantics, specifically including:

[0170] 1) Unit semantic mapping: Based on the aforementioned surgical sequence semantic parsing method, the video is divided and mapped into standardized surgical process units;

[0171] 2) Medical auxiliary annotation: Each process unit is attached with medical auxiliary annotation, such as the medical objectives, precautions and key operations of the unit;

[0172] After receiving the surgical procedure unit identifier selected by the physician, the following steps are performed:

[0173] Process unit selection: select a single process unit or multiple continuous process units through a drop-down menu or graphical interface;

[0174] Mapping of process units to time intervals: call the P→T reverse mapping function to obtain the time interval corresponding to the process unit;

[0175] Relevant information acquisition: call P→F mapping to obtain key device combination information and extract medical auxiliary annotation information;

[0176] Result presentation and playback: locate the video to the start time, synchronously display the annotated content, and highlight the corresponding interval on the timeline.

[0177] This process unit-based retrieval model integrates medical semantic annotations to make the retrieval results more intuitive and in line with medical professional needs, which can greatly improve learning efficiency and shorten the learning time curve.

[0178] Step S4: Dynamic visualization of surgical sequence is achieved through multi-track design and medical semantic visualization coding.

[0179] Specifically, all the above functions of the present invention are implemented in a visual interactive manner, so that doctors can intuitively see the temporal characteristics of laparoscopic surgery videos and conveniently and quickly perform interactive retrieval of video clips of interest.

[0180] The dynamic visualization diagram of time series features is as follows Figure 5 As shown, step S4 specifically includes:

[0181] S41, multi-track display design:

[0182] Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of the instrument combination is clearly displayed through vertical layering;

[0183] Time scale adaptation: automatically adjusts the time scale density based on the total length of multiple videos and the current zoom level;

[0184] Color coding: Use different color codes to distinguish different types of surgical instruments to improve visual recognition efficiency;

[0185] Density perception expression: The frequency and intensity of the instrument combination are intuitively expressed through the density of points, revealing the high-frequency periods of surgical operations.

[0186] S42, Medical Semantic Visual Coding:

[0187] Process unit background identification: Use translucent color blocks to mark different surgical process units, encoding medical semantics through color and transparency;

[0188] Key point highlighting: Automatically identify and highlight key time points during surgery, such as instrument switching, tissue separation, and other decision points;

[0189] Cross-video boundary marking: Vertical dotted lines and video identifiers are used to clearly express the multi-video structure, making it easier to grasp the overall cross-video data.

[0190] This invention is based on a multimodal interactive retrieval method for hierarchical data organization, and innovatively constructs a "video analysis-data organization-multimodal retrieval" technical process, achieving efficient retrieval and intuitive presentation of laparoscopic surgery videos, greatly enhancing the application value of surgical videos in scenarios such as teaching, quality assessment, and skill analysis. Compared with existing technologies, this invention deeply integrates medical expertise to perform temporal modeling of surgical videos; innovatively proposes a two-way mapping method to convert traditional surgical video data into hierarchical data that is easy to store and retrieve; through multimodal retrieval methods and interactive feedback of dynamic visualization of temporal features, it provides medical personnel with a professional retrieval experience that meets their training and research needs.

[0191] In a preferred embodiment of the present invention, the present invention provides a laparoscopic surgery video retrieval and visualization system, the system comprising:

[0192] The surgical video analysis module uses a deep learning model to analyze instrument features in surgical videos, performs temporal semantic parsing based on a hidden Markov model constrained by medical knowledge, and divides the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure.

[0193] Hierarchical data organization module, used to convert the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument characteristics, and process units;

[0194] A multi-mode interactive retrieval module is used to design three interactive retrieval modes based on a hierarchical data structure: timeline-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid positioning by calling the corresponding bidirectional mapping relationship.

[0195] A visualization module is used to achieve dynamic visualization of surgical sequences through multi-track design and medical semantic visualization encoding.

[0196] The laparoscopic surgery video retrieval and visualization system provided by the present invention is used to execute the laparoscopic surgery video retrieval and visualization methods provided in the aforementioned embodiments. The laparoscopic surgery video retrieval and visualization methods have been described in detail in the aforementioned embodiments and will not be repeated in this embodiment.

[0197] The present invention provides a laparoscopic surgery video retrieval and visualization method and system, which innovatively constructs the "video analysis-data organization-multimodal retrieval" technical process to accurately analyze and efficiently utilize surgical video resources. At the "video analysis" level, a temporal semantic parsing method based on medical knowledge constraints is introduced. By explicitly modeling the surgical process logic and medical specifications, the computing resource requirements and model training time are greatly reduced; at the "data organization" level, a feature-entity bidirectional mapping method is innovatively proposed to convert massive and complex video data into efficient hierarchical data; at the "multimodal retrieval" level, a medical knowledge-driven multimodal interactive retrieval method is designed. Through multi-track display and medical semantic visualization encoding and support for multi-video information presentation, it provides surgeons with an intuitive and efficient surgical video learning experience. Through this complete technical process, the present invention effectively solves the various limitations of existing technologies when dealing with complex laparoscopic surgical videos, and significantly improves the application value of surgical videos in medical education and training.

[0198] Figure 6 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 6As shown, the present invention further provides an electronic device, wherein the electronic device 600 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, or a server. The electronic device 600 includes a processor 601 and a memory 602, wherein the memory 602 stores a laparoscopic surgery video retrieval and visualization program 603.

[0199] In some embodiments, the memory 602 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 602 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 602 may also include both an internal storage unit of the computer device and an external storage device. The memory 602 is used to store application software and various types of data installed on the computer device, such as program codes installed on the computer device. The memory 602 may also be used to temporarily store data that has been output or is to be output. In one embodiment, when the laparoscopic surgery video retrieval and visualization program 603 is executed by the processor 601, the following steps are implemented:

[0200] A deep learning model is used to analyze the instrument features in surgical videos. A hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure.

[0201] The data obtained from surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between three types of entities: time, instrument characteristics, and process units;

[0202] Based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode can achieve rapid positioning by calling the corresponding bidirectional mapping relationship.

[0203] Dynamic visualization of surgical sequence is achieved through multi-track design and medical semantic visualization encoding.

[0204] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 602, such as executing a laparoscopic surgery video retrieval and visualization program.

[0205] This embodiment further provides a computer-readable storage medium storing a laparoscopic surgery video retrieval and visualization program. When the laparoscopic surgery video retrieval and visualization program is executed by a processor, the following steps are implemented:

[0206] A deep learning model is used to analyze the instrument features in surgical videos. A hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure.

[0207] The data obtained from surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between three types of entities: time, instrument characteristics, and process units;

[0208] Based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode can achieve rapid positioning by calling the corresponding bidirectional mapping relationship.

[0209] Dynamic visualization of surgical sequence is achieved through multi-track design and medical semantic visualization encoding.

[0210] The above-described embodiments merely illustrate several embodiments of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A laparoscopic surgery video retrieval and visualization method, characterized in that: include: A deep learning model is used to analyze the instrument features in surgical videos. A hidden Markov model based on medical knowledge constraints is used for temporal semantic parsing to divide the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure. The data obtained from surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between three types of entities: time, instrument characteristics, and process units; Based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search. Each search mode can achieve rapid positioning by calling the corresponding bidirectional mapping relationship. Dynamic visualization of surgical sequence is achieved through multi-track design and medical semantic visualization encoding.

2. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that: The deep learning model is used to analyze the instrument features in the surgical video, including: Use deep learning models to analyze instrument features in laparoscopic surgery videos; Combined with the sliding window median filtering method, instantaneous misjudgment is eliminated and the device feature data is output.

3. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that: The medical knowledge-constrained hidden Markov model performs temporal semantic analysis and divides the standardized surgical process into units, including: Constructing multi-dimensional temporal features of surgical videos; wherein, multi-dimensional temporal features of surgical videos include static instrument features, dynamic instrument features, specific instrument combination features, continuous instrument use features, and surgical progress features; The multi-dimensional temporal features of surgical videos are used as observation inputs in the hidden Markov model, combined with medical knowledge constraints, to establish a medical knowledge-constrained hidden Markov model. The Viterbi algorithm is used to process the observation sequence of the surgical video and preliminarily infer the sequence of surgical process units; For the parts of the surgical process unit sequence that do not conform to medical logic, optimization is performed through abnormal time period allocation and process unit correction to output a standardized surgical process unit sequence.

4. The laparoscopic surgery video retrieval and visualization method according to claim 3, characterized in that: The expression of the hidden Markov model constrained by medical knowledge is: MK-HMM=(S, O, A, B, π, M) Where MK-HMM represents the hidden Markov model constrained by medical knowledge; S = {S1, S2, ..., S n } is a state set corresponding to n standardized surgical process units; O represents the observed state set, A represents the state transition probability matrix, B represents the emission probability matrix, π represents the initial state probability vector, and M represents the medical knowledge constraint set; Wherein, the medical knowledge constraint set M includes a sequence constraint matrix M_seq and an initial state constraint vector M_init(i); M={M_seq, M_init(i)}; According to the sequence constraint matrix M_seq and the initial state constraint vector M_init(i), the constraint state transfer matrix and the constraint initial state probability are obtained; The expression of the constraint state transfer matrix A'(i, j) is: A'(i,j)=A(i,j)*M_seq(i,j) Where M_seq(i, j) is the sequence constraint matrix, which represents the legal transfer relationship between surgical process units. The matrix element M_seq(i, j) indicates whether the transfer from process unit i to j complies with medical standards. The value of 1 indicates a legal transfer, and the value of 0 indicates an illegal transfer. The expression of the constrained initial state probability π'(i) is: π'(i)=π(i)*M_init(i) Where M_init(i) is the initial state constraint vector, which encodes the rationality of each process unit as the starting unit of the operation.

5. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that: The data obtained from the surgical video analysis is converted into a hierarchical data structure, and a bidirectional mapping relationship is established between the three entities of time, instrument characteristics, and process units, including: Design a spatiotemporal feature mapping data model that includes time entities, device feature entities, and process unit entities. Encode the three core entities into a nested key-value pair structure to form a hierarchical data format. A bidirectional mapping relationship is established between the three entities of time-device characteristics, time-process unit, and process unit-device characteristics to form a mapping network.

6. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that: Based on the hierarchical data structure, three interactive search modes are designed: timeline-oriented search, instrument-oriented search, and process unit-oriented search, including: Timeline-guided retrieval mode: Based on the selected start and end points, the system automatically calculates the corresponding time interval and plays the corresponding surgical video clip; supports retrieval and automatic continuous playback across multiple video files; when a time point is selected for retrieval in the time sequence visualization interface, the system automatically displays the corresponding surgical process unit and instrument characteristics through a bidirectional mapping relationship; Device-oriented retrieval mode: Based on the feature-time reverse mapping in the hierarchical data structure, a fast index of device type and timestamp is established, and a median filter algorithm is integrated to reduce noise in device detection results. The system supports retrieval of single devices or multiple device combinations. After the user selects a single device or a combination of devices, the system calls the reverse mapping function of the hierarchical data organization layer to obtain the corresponding time period, generates a playback instruction queue, and automatically plays the video. Process unit-oriented retrieval mode: The hidden Markov model based on medical knowledge constraints parses the surgical video into standardized process units, and each process unit is associated with medical auxiliary annotations; after the user selects the process unit identifier, the system calls the process unit-time reverse mapping function to obtain the time interval corresponding to the process unit, and at the same time calls the process unit-instrument feature mapping function to obtain the instrument information corresponding to the process unit; the system locates the video to the start time, and synchronously displays the medical auxiliary annotation content clearly on the interface, and highlights the corresponding interval on the timeline.

7. The laparoscopic surgery video retrieval and visualization method according to claim 1, characterized in that: The multi-track display design includes: Instrument layered display: The number of tracks and vertical spacing are dynamically calculated based on the number of instrument categories. Each surgical instrument occupies an independent track, and the time distribution of the instrument combination is clearly displayed through vertical layering; Timescale Adaptation: Automatically adjust the timescale density based on the total length of multiple videos and the current zoom level; Color coding: Use different color codes to distinguish different types of surgical instruments to improve visual recognition efficiency; Density perception expression: The density of points is used to intuitively express the frequency and intensity of instrument combinations, revealing the high-frequency periods of surgical operations: The medical semantic visualization coding includes: Process unit background identification: Use translucent color blocks to mark different surgical process units, encoding medical semantics through color and transparency; Key point highlighting: automatically identifies and highlights key time points during surgery; Cross-video boundary marking: vertical dotted lines and video logos are used to clearly express the multi-video structure.

8. A laparoscopic surgery video retrieval and visualization system, characterized in that: include: The surgical video analysis module uses a deep learning model to analyze instrument features in surgical videos, performs temporal semantic parsing based on a hidden Markov model constrained by medical knowledge, and divides the surgical process into standardized units. The surgical process unit is a standardized operation stage during the surgical procedure. Hierarchical data organization module, used to convert the data obtained from surgical video analysis into a hierarchical data structure and establish a bidirectional mapping relationship between three types of entities: time, instrument characteristics, and process units; A multi-mode interactive retrieval module is used to design three interactive retrieval modes based on a hierarchical data structure: timeline-oriented retrieval, instrument-oriented retrieval, and process unit-oriented retrieval. Each retrieval mode achieves rapid positioning by calling the corresponding bidirectional mapping relationship. A visualization module is used to achieve dynamic visualization of surgical sequences through multi-track design and medical semantic visualization encoding.

9. An electronic device, It is characterized by: comprising a memory and a processor, wherein, The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the laparoscopic surgery video retrieval and visualization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the laparoscopic surgery video retrieval and visualization method described in any one of claims 1 to 7.

Images