Surgical video analysis method based on video large language model and related equipment

Abstract

A surgical video analysis method based on a video large language model and related equipment relate to the technical field of artificial intelligence. The method comprises: acquiring a surgical video and a text question corresponding to the surgical video; inputting the surgical video and the text question into a video large language model to acquire a text answer corresponding to a video segment in the surgical video output by the video large language model. The technical solution provided by the present application can improve the practical value of the surgical video as a teaching and reference resource.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a surgical video analysis method based on a video large language model and related equipment. BACKGROUND

[0002] In the field of surgical operations, accurately understanding and analyzing surgical videos is of great significance for medical teaching, surgical evaluation and surgical skill improvement. Doctors need to quickly obtain key information from a large number of surgical videos, including surgical steps, instrument use, surgical techniques and other professional content, which are crucial for improving surgical quality and training new doctors.

[0003] Currently, the analysis of surgical videos mainly relies on experienced doctors for manual viewing and interpretation, or uses traditional computer vision technology to process videos. Some medical institutions also use video analysis systems based on deep learning to identify specific scenes and actions in surgical videos through pre-set rules and models.

[0004] However, existing surgical video analysis methods cannot accurately understand and answer specific questions about video content. Traditional video analysis systems can only identify pre-set scenes and cannot perform flexible question-and-answer interactions, which severely limits the practical value of surgical videos as teaching and reference resources. SUMMARY

[0005] The present application provides a surgical video analysis method based on a video large language model and related equipment, which can improve the practical value of surgical videos as teaching and reference resources.

[0006] In a first aspect of the present application, a surgical video analysis method based on a video large language model is provided, comprising:

[0007] obtaining a surgical video and a text question corresponding to the surgical video;

[0008] inputting the surgical video and the text question into a video large language model to obtain a text answer corresponding to a video segment in the surgical video output by the video large language model.

[0009] In a second aspect of the present application, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the surgical video analysis method based on the video large language model as described above when executing the computer program.

[0010] In a third aspect of the present application, a non-transitory computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the surgical video analysis method based on the video large language model according to any of the above.

[0011] In a fourth aspect of the present application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the surgical video analysis method based on the video large language model according to any of the above.

[0012] In summary, the one or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0013] By inputting the surgical video and the corresponding text question into the video large language model and obtaining the text answer output by the model for the video segment, the technical problem of the lack of flexible question and answer interaction capability in the prior art surgical video analysis system is solved. Compared with the traditional computer vision system that only relies on manual viewing and interpretation or can only recognize pre-set scenes, the present application realizes intelligent understanding and accurate question and answer of the surgical video content, enabling doctors to quickly obtain key professional information such as surgical steps, instrument use and surgical skills, thereby improving the practical value of the surgical video as a teaching and reference resource. The present application breaks through the limitation of the existing video analysis method that can only recognize pre-set scenes, and realizes natural interaction with the content of the surgical video through intelligent processing of the video large language model, helping doctors to more efficiently obtain professional knowledge from a large number of surgical videos, and providing more advanced technical support for medical teaching, surgical evaluation and surgical skill improvement. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0015] Figure 1 is a flowchart of a surgical video analysis method based on a video large language model provided by an embodiment of the present application.

[0016] Figure 2 is a schematic diagram of a video encoder training process provided by an embodiment of the present application.

[0017] Figure 3 is a schematic diagram of a video large language model training process provided by an embodiment of the present application.

[0018] Figure 4Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] Reference should be made to Figure 1 , Figure 1 Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0021] S101, acquiring a surgical video and a text question corresponding to the surgical video.

[0022] Specifically, the surgical video can be understood as time series data composed of continuous image frames, each image frame contains visual information of the surgical site, especially the position, movement trajectory of the surgical instrument and the interaction with the surgical site. These video data can be real-time acquisition data from the operating room, or historical surgical records stored in the medical database, used to provide basic visual information for subsequent intelligent analysis.

[0023] Correspondingly, the text question refers to specific analysis requirements or query instructions proposed for the content of the surgical video. In the embodiments of the present application, the text question can be understood as a query sentence described in natural language, such as asking about the surgical state, instrument position, surgical step progress, etc. at a specific time point. The text question can be a real-time inquiry proposed by the surgeon during or after the operation, or a standard question set preset for the purpose of teaching, training, surgical review, etc., used to guide the video large language model to carry out targeted video understanding and information extraction.

[0024] The combination of the surgical video and the text question constitutes the input data basis of the video analysis method of the present application, for supporting the subsequent intelligent analysis process. By simultaneously processing the video data and the text query, the system can realize accurate understanding and targeted information extraction of the surgical video, and ultimately provide decision support and auxiliary analysis for medical practice.

[0025] S102, input the surgical video and the text question into the video large language model, and obtain the text answer corresponding to the video segment in the surgical video output by the video large language model.

[0026] The video large language model refers to an artificial intelligence model capable of simultaneously processing video and text modal information, and having video understanding and natural language interaction capabilities. In the embodiments of the present application, the video large language model can be understood as a multi-modal intelligent system composed of a video coding module, a text coding module and a large language module, wherein the video coding module is specially used for processing video data and extracting video features, and is particularly concerned about the position and motion features of surgical instruments; the text coding module is responsible for converting natural language form questions into model processable feature representations; and the large language module integrates the information of the two modalities, understands the video content and generates the corresponding text answer.

[0027] Further, in the large language module, the system first intelligently segments the video to ensure that each video segment contains complete semantic information, and then extracts the required information from the relevant video segment based on the specific requirements of the text question.

[0028] For example, when receiving a question such as "Where is the position of the surgical instrument at the 3rd minute of the surgery?", the system will automatically locate the 3rd minute video segment, analyze the specific position and operation state of the surgical instrument at this time point, and generate an accurate text answer, such as "At the 3rd minute, the surgical instrument is located in the right upper quadrant of the patient's abdomen, and is performing tissue separation operation".

[0029] The advantage of the processing method is that it can realize accurate understanding and directional analysis of the content of the surgical video, and help doctors quickly obtain the required specific surgical information. Through the intelligent analysis method of the video large language model, the efficiency and accuracy of surgical video analysis are significantly improved, which provides strong technical support for medical practice, and also provides an important reference basis for surgical teaching and surgical quality evaluation.

[0030] On the basis of the above-mentioned embodiments, as an optional embodiment, in S102, the step of inputting the surgical video and the text question into the video large language model to obtain the text answer corresponding to the video segment in the surgical video output by the video large language model can further include the following steps:

[0031] S201. Extract video features from surgical videos and text features from text problems using a video large language model.

[0032] Specifically, the video big language model extracts features from the input surgical video through the video encoding module. This process includes frame-level analysis of the video data, focusing on identifying and extracting key visual features such as the position information, movement trajectory, and operation status of surgical instruments.

[0033] During feature extraction, the video encoding module focuses on the area where surgical instruments are located, accurately capturing the interaction details between the instruments and the surgical site through a pre-trained masking mechanism. Simultaneously, the text encoding module processes the input text question, converting the natural language question into text feature vectors that the model can handle.

[0034] For example, when processing the text question "identify the location of the surgical instrument at the 5th minute in the video", the text encoding module extracts key features such as the time information "5th minute" and the query target "location of the surgical instrument". Through this feature extraction method, the visual information in the surgical video and the semantic information in the text question are converted into standardized feature representations.

[0035] S202. Using a video-based large language model, the surgical video is segmented and analyzed based on video and text features to obtain the text responses corresponding to the video segments in the surgical video.

[0036] Specifically, the video big language model first segments the extracted video features according to the time sequence, with each video segment containing continuous surgical operation information. During the segmentation process, the system combines the time information contained in the text features, such as specific time points like "minute 5" or surgical stage descriptions like "suture stage," to intelligently segment the video. Subsequently, the big language module converts the video features into video tokens and the text features into text tokens, and combines these two types of tokens according to their temporal relationship to form a complete input sequence.

[0037] For example, when the system receives a question asking about the location of surgical instruments at a specific time point, it first locates the corresponding video segment, then analyzes key information such as the instrument's position and movement trajectory within that segment, and finally generates an accurate text description, such as "At minute 5, the surgical instruments are located in the upper right quadrant of the abdominal cavity, and tissue separation is underway." This segmented analysis-based processing method ensures that the system's understanding of the surgical video is more accurate, and the generated answers are more in line with the doctor's actual needs.

[0038] Through the above segmentation analysis method, the system can not only accurately answer time-related questions, but also understand and describe the specific operation details in the surgical process, providing valuable surgical information support for doctors. This processing method significantly improves the accuracy and practicality of surgical video analysis, enabling the system to provide intelligent support for surgical teaching, surgical quality assessment and other medical scenarios.

[0039] The above embodiments describe the actual application process of the video large language model. Based on the above embodiments, the training process of the video large language model will be described below. Specifically, the process can further include the following steps:

[0040] S301, obtaining a sample surgical video, a sample text question corresponding to the sample surgical video, and an initial video large language model, the initial video large language model comprising a video encoding module, a text encoding module and a large language module.

[0041] Specifically, in order to enable the video large language model to accurately understand and analyze the surgical video, the model performance needs to be optimized through a training process. Before the training starts, the basic data and model components required for training need to be obtained.

[0042] Specifically, first, a large number of real surgical videos are collected as sample surgical videos, which contain different types of surgical operations, various surgical instrument usage scenarios and complete surgical procedure information. At the same time, sample text questions labeled by professional doctors are collected for these sample surgical videos, which cover key time points, instrument operation, surgical steps and other aspects of query requirements in the surgical process.

[0043] In addition, an initial video large language model needs to be constructed, which consists of three core modules: the video encoding module is responsible for processing video data and extracting video features, the text encoding module is used to understand text questions and extract text features, and the large language module is responsible for integrating the two features and generating answers. By obtaining these basic training data and model components required for training.

[0044] S302, training the video encoding module through the sample surgical video to obtain sample video features.

[0045] After obtaining the basic training data and model components, the video encoding module needs to be trained first to improve its ability to extract surgical video features. Specifically, the training of the video encoding module adopts a surgical instrument-centered feature learning strategy. First, the sample surgical video is cut into continuous video segments, and the position area of the surgical instrument in each segment is identified and labeled.

[0046] During the training process, the encoder Me of the video encoding module receives the video input with the mask, predicts the masked region, and the decoder Md reconstructs the original video content based on the output of the encoder. The parameters of the video encoding module are optimized by calculating the mean square error loss between the reconstruction result and the original video.

[0047] Based on the above embodiment, as an optional embodiment, the video encoding module includes a video encoder and a video decoder, and in S302, the step of training the video encoding module by the sample surgical video to obtain the sample video features can further include the following steps:

[0048] S401, cut the sample surgical video into sample video segments, and add a mask to the target position in the sample video segments to obtain a masked region and a non-masked region.

[0049] The target position refers to the region position that needs to be focused on and analyzed in the surgical video. In the embodiment of the application, the target position can be understood as the spatial region where the surgical instrument is located, including the instrument body and the operation region around it, which usually contains the position information, morphological features of the surgical instrument and the interaction state with the surrounding tissue. For example, when the surgical instrument is surgical forceps, the target position not only includes the spatial region occupied by the surgical forceps itself, but also includes the region where the clamping part of the surgical forceps contacts the surgical site; when an electrotome is used for cutting, the target position includes the electrotome and its action region.

[0050] Further, by adding a mask to the target position, the video encoding module can be guided to focus on learning the feature representation of the surgical instrument; secondly, by predicting the target position information blocked by the mask, the understanding ability of the model for the operation of the surgical instrument is improved; the target position information is also used to help the model establish the correlation between the position of the instrument, the moving track and the operation behavior in the surgical process, thereby improving the overall understanding ability of the model for the surgical video. Through the above-mentioned manner, the model can better capture the key information in the surgical process, and provide more accurate feature representation for subsequent video analysis.

[0051] Please refer to Figure 2 , Figure 2A schematic diagram of a video encoder training process is provided for embodiments of the present application. Specifically, a complete sample surgical video is cut into multiple continuous sample video segments at a fixed time interval, each segment needs to contain complete surgical operation semantic information. After obtaining the sample video segments, the system automatically identifies the target position in each segment, i.e. the spatial region where the surgical instrument is located, including the instrument body and its surrounding operation area. For example, when the video segment shows the surgical forceps performing tissue clamping, the target position includes the surgical forceps body and the area where the clamping part of the forceps contacts the surgical site. Subsequently, masks are added to these identified target positions, i.e. using a preset occlusion marker to cover these areas to form a mask region. The other areas in the video that are not occluded constitute a non-mask region, which usually contains surgical background environment, anatomical structure and other auxiliary information.

[0052] S402, optimizing the video encoder based on the mask region and the non-mask region.

[0053] Specifically, the video encoding module contains two key components, encoder Me and decoder Md, where the encoder Me is responsible for converting the video segment with masks into feature representation, and the decoder Md is responsible for reconstructing the original video content based on these features. In the training process, the video segment with masks is first input into the encoder Me, which processes the video content containing mask regions and non-mask regions to generate corresponding feature encodings. Subsequently, these feature encodings are input into the decoder Md, which attempts to reconstruct the regions occluded by masks, i.e. predicts the position and morphological features of the surgical instrument. By calculating the mean square error loss between the reconstruction result and the original video, the formula is expressed as follows:

[0054] ; where, ;

[0055] where, represents the mask region, represents the non-mask region, represents the encoder, represents the decoder.

[0056] For example, when the model processes a video segment containing the operation of surgical forceps, it can accurately infer and reconstruct the position and operation state of the surgical forceps through the surrounding non-masked area information even if the surgical forceps area is obscured by a mask. This training method based on mask reconstruction enables the video encoder to learn more rich visual feature representations, especially a deeper understanding of the spatial position, morphological characteristics, and operation mode of surgical instruments. Through this optimization process, the video encoder not only accurately captures key information in surgical videos, but also adapts to instrument recognition and tracking requirements in different surgical scenarios, providing high-quality feature representations for subsequent video analysis tasks, significantly improving the performance and reliability of the model in practical applications.

[0057] S303, train the text encoding module through the sample text question to obtain sample text features.

[0058] Specifically, the text encoding module receives sample text questions as input, which contain various query requirements during the surgical process, such as "Where is the position of the surgical instrument at the 3rd minute of the video?" and "Does bleeding occur during the surgical process?" and other professional questions. During the training process, the sample text questions are first preprocessed, including word segmentation, tokenization, and other operations, to convert natural language questions into input sequences that the model can process. Subsequently, the text encoding module processes these input sequences through an attention mechanism, focusing on key information in the questions such as time points, query targets, and operation types. For example, when processing the question "Identify the operation state of the surgical forceps at the 5th minute," the text encoding module will pay special attention to the time information "5th minute" and the query target "operation state of the surgical forceps."

[0059] In the above manner, the text encoding module gradually learns to map different types of surgery-related questions to standardized sample text features. These features not only contain semantic information about the questions, but also retain key attribute information such as time and space, providing reliable text representations for subsequent multi-modal analysis.

[0060] Based on the above embodiments, as an optional embodiment, the video encoding module and the text encoding module can also be subjected to contrastive learning based on sample video features and sample text features to align the output features of the video encoder and the text encoding module in time.

[0061] Specifically, first, select paired sample video features and sample text features from the training samples, which come from video content and corresponding text descriptions at the same time point or the same surgical stage.

[0062] For example, when the sample text feature describes "the surgical forceps separate the tissue at the 5th minute", it needs to be paired with the surgical operation feature at the 5th minute in the video. In the contrast learning process, a contrast loss function is used to measure the similarity between the two modal features. For positive sample pairs (i.e. video features and text features at the same time point), the system maximizes their similarity in the feature space; for negative sample pairs (i.e. video features and text features at different time points), the similarity is minimized.

[0063] In this way, the system gradually optimizes the parameters of the video encoding module and the text encoding module, so that the features output by them form a time-consistent representation in the semantic space. For example, when the video shows the surgical instrument performing suturing operation, the suturing-related features mentioned in the text description will have a higher similarity with the video features at the corresponding time point. This time alignment strategy based on contrast learning significantly improves the model's understanding of the temporal relationship between the surgical video content and the text description, enabling the system to more accurately locate and analyze the surgical operation at a specific time point.

[0064] By using the above technical solutions, not only the performance of the model in the time series analysis task is enhanced, but also the accuracy of the subsequent question and answer system is improved, providing a more reliable feature basis for intelligent analysis of surgical videos. This time-aligned feature representation enables the model to better understand and answer surgical-related questions involving temporal information.

[0065] S304, training the large language module through the sample video feature and the sample text feature to obtain a sample text answer.

[0066] Please refer to Figure 3 , Figure 3 A video large language model training process diagram is shown. Based on the above embodiment, as an optional embodiment, the video large language model further includes a classifier. In S304, the step of training the large language module through the sample video feature and the sample text feature to obtain a sample text answer can further include the following steps:

[0067] S501, training the classifier through the sample text feature to obtain a classification result of the sample text feature.

[0068] In the training process of the large language module, in order to better understand and classify different types of surgical questions, the classifier needs to be specially trained.

[0069] Specifically, the classifier receives the sample text features processed by the text encoding module as input, which contains various types of surgical question information. During training, the sample text features are first labeled with classification, and the questions are divided into different categories, such as time query category (asking about the status of the operation at a specific time point), operation type query category (asking about the specific operation of the surgical instrument), position query category (asking about the spatial position of the surgical instrument), etc.

[0070] For example, when processing the question "Where is the position of the surgical forceps at the 5th minute?", the classifier identifies it as a position query category question. Through classification learning of sample text features, the classifier gradually establishes the mapping relationship between question type and feature representation. During training, the cross-entropy loss function is used to evaluate the classification accuracy, and the parameters of the classifier are optimized through back propagation. This classification training method enables the system to accurately identify different types of surgical questions, thereby providing more targeted processing strategies for subsequent answer generation.

[0071] Further, when the system identifies a time query category question, it will pay special attention to the time sequence information in the video features; when identifying an operation type query, it will focus on analyzing the action features of the surgical instrument. The classifier trained in this way can accurately understand the type and query intention of the question, providing important support for generating more accurate answers. This classification-based question understanding method significantly improves the accuracy and efficiency of the system in processing different types of surgical questions, providing a more detailed feature classification basis for intelligent analysis of surgical videos.

[0072] S502, convert the sample video features into video tokens, and convert the sample text features into text tokens.

[0073] After obtaining the sample video features and sample text features, it is necessary to convert the features of the two different modalities into a unified token representation form for subsequent multi-modal analysis by the large language module.

[0074] Specifically, first process the sample video features, and map the continuous video features to the discrete video token space through linear projection and position encoding. This process preserves the time sequence information and spatial information in the video features, for example, when the video shows the surgical forceps performing a suturing operation, the corresponding video token will contain key information such as instrument position, movement trajectory, etc. At the same time, the sample text features are processed to convert the text features into a standard text token sequence.

[0075] During the conversion process, the system considers key elements such as professional terms, time information, and query targets in the text to ensure that the converted text tokens accurately express the semantic content of the question. For example, when processing the question "Identify the operation state of the surgical forceps at the 5th minute", the system converts it into a token sequence containing time markers, instrument markers, and operation markers. Through this feature-to-token conversion process, the system establishes a unified representation between video content and text questions, enabling the large language module to process information in the same feature space for both modalities.

[0076] Through the above conversion, not only the key information of the original features is preserved, but also the processing efficiency of the model is improved through standardized token representation. This token-based unified representation provides a reliable data foundation for subsequent multi-modal analysis and answer generation, significantly improving the system's performance in surgical video analysis tasks.

[0077] S503, input the video tokens and text tokens into the large language module to make the large language module generate sample text answers of the video tokens and text tokens based on the classification results.

[0078] Specifically, the large language module first receives the converted video token and text token sequences, and combines the classification results of the classifier. In the processing process, the large language module uses a multi-head attention mechanism to interactively analyze the two token sequences, and adjusts the processing strategy of different types of questions based on the classification results.

[0079] For example, when the classification result indicates that it is a time query type question such as "the operation state of the surgical forceps at the 5th minute", the system focuses on the information of the corresponding time point in the video token and highlights the relevant visual features through attention weights. In the process of generating answers, the large language module generates text answers through the decoder, while considering the specific needs of the question type. For position query type questions, the system generates detailed descriptions containing spatial position information; for operation type queries, it focuses on describing the specific operation state of the instrument. The model parameters are optimized by calculating the cross-entropy loss between the generated sample text answers and the standard answers, prompting the system to generate more accurate and professional answers.

[0080] Through the above multi-modal analysis method based on classification perception, the large language module can generate more targeted answers according to different types of questions. For example, for the same video showing the operation of surgical instruments, the system can generate "at the 5th minute, the surgical forceps are located in the right upper quadrant of the abdomen" or "the surgical forceps are performing tissue separation operation with moderate operation force" for different types of questions. This training method significantly improves the system's performance in surgical video analysis and question answering tasks, making the generated answers more accurate, professional, and in line with the needs of medical practice.

[0081] S305, based on the sample text, the initial video large language model is fine-tuned to obtain a video large language model.

[0082] After the basic training of each module is completed, the entire video large language model needs to be fine-tuned end-to-end to improve the overall performance and generalization ability of the model.

[0083] Figure 4 An example of an entity structure diagram of an electronic device is shown in Figure 4 As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can invoke the logical instructions in the memory 430 to execute the surgical video analysis method based on the video large language model.

[0084] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the surgical video analysis method based on the video large language model provided by the above-mentioned methods.

[0086] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the surgical video analysis method based on the video large language model provided by the above-mentioned methods.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A surgical video analysis method based on a video large language model, characterized in that, The method comprises: acquiring a surgery video and a text question corresponding to the surgery video; inputting the surgery video and the text question into a video large language model to acquire a text answer corresponding to a video segment in the surgery video output by the video large language model; a training process of the video large language model comprises: acquiring a sample surgery video, a sample text question corresponding to the sample surgery video, and an initial video large language model, wherein the initial video large language model comprises a video encoding module, a text encoding module, and a large language module; training the video encoding module through the sample surgery video to obtain sample video features; training the text encoding module through the sample text question to obtain sample text features; training the large language module through the sample video features and the sample text features to obtain a sample text answer; fine-tuning the initial video large language model based on the sample text answer to obtain the video large language model; the video encoding module comprises a video encoder, and the training of the video encoding module through the sample surgery video to obtain sample video features comprises: cutting the sample surgery video into sample video segments, and adding a mask to a target position in the sample video segments to obtain a mask region and a non-mask region; optimizing the video encoder based on the mask region and the non-mask region.

2. The method of claim 1, wherein the video-based large language model is a video-based large language model for surgery. inputting the surgery video and the text question into the video large language model to acquire a text answer corresponding to a video segment in the surgery video output by the video large language model comprises: extracting video features of the surgery video and text features of the text question through the video large language model; performing segment analysis on the surgery video based on the video features and the text features through the video large language model to obtain a text answer corresponding to a video segment in the surgery video.

3. The method of claim 1, wherein the video-based large language model is a video-based large language model for surgery. The target position comprises a position of an instrument.

4. The surgical video analysis method based on a video large language model according to claim 1, wherein, The method further comprises: performing contrast learning on the video encoding module and the text encoding module based on the sample video features and the sample text features to align output features of the video encoder and the text encoding module in time.

5. The method of claim 1, wherein, The video large language model further comprises a classifier; the training of the large language module through the sample video features and the sample text features to obtain a sample text answer comprises: training the classifier through the sample text features to obtain a classification result of the sample text features; converting the sample video features into video tokens and converting the sample text features into text tokens; inputting the video tokens and the text tokens into the large language module to enable the large language module to generate a sample text answer of the video tokens and the text tokens based on the classification result.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the surgery video analysis method based on the video large language model according to any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the surgery video analysis method based on the video large language model according to any one of claims 1 to 5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the surgical video analysis method based on the video large language model according to any one of claims 1 to 5.

Images