Operation video analysis method based on video big language model and related equipment

Abstract

The invention discloses an operation video analysis method based on a video large language model and related equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an operation video and a text question corresponding to the operation video; and inputting the operation video and the text question into a video big language model, and obtaining a text answer corresponding to a video clip in the operation video output by the video big language model. By implementing the technical scheme provided by the invention, the practical value of the operation video as teaching and reference resources can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a surgical video analysis method based on a video large language model and related equipment. Background Art

[0002] In the field of surgery, accurate understanding and analysis of surgical videos is of great significance for medical teaching, surgical evaluation and surgical skills improvement. Doctors need to quickly obtain key information from a large number of surgical videos, including professional content such as surgical steps, instrument use, and surgical skills. This information is crucial for improving surgical quality and training new doctors.

[0003] At present, the analysis of surgical videos mainly relies on manual viewing and interpretation by experienced doctors, or the use of traditional computer vision technology to process the videos. Some medical institutions also use deep learning-based video analysis systems to identify specific scenes and actions in surgical videos through preset rules and models.

[0004] However, existing surgical video analysis methods have difficulty accurately understanding and answering specific questions about the video content. Traditional video analysis systems can often only recognize preset scenes and cannot perform flexible question-and-answer interactions, which seriously limits the practical value of surgical videos as teaching and reference resources. Summary of the invention

[0005] The present invention provides a surgical video analysis method and related equipment based on a video large language model, which can improve the practical value of surgical videos as teaching and reference resources.

[0006] In a first aspect of the present invention, a surgical video analysis method based on a video large language model is provided, comprising: Obtaining a surgical video and text questions corresponding to the surgical video; The surgical video and the text question are input into a video language model, and a text answer corresponding to the video clip in the surgical video output by the video language model is obtained.

[0007] In a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a surgical video analysis method based on a large video language model as described above is implemented.

[0008] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the surgical video analysis method based on the video large language model as described in any one of the above is implemented.

[0009] In a fourth aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method for analyzing surgical videos based on a large video language model as described above is implemented.

[0010] In summary, one or more technical solutions provided in the present invention have at least the following technical effects or advantages: By inputting surgical videos and corresponding text questions into the video language model, and obtaining the text answers output by the model for the video clips, the technical problem of the lack of flexible question-answering interactive capabilities in the surgical video analysis system in the prior art is solved. Compared with traditional computer vision systems that only rely on manual viewing and interpretation or can only recognize preset scenes, the present invention realizes intelligent understanding and accurate question-answering of surgical video content, allowing doctors to quickly obtain the required key professional information such as surgical steps, instrument use and surgical skills, thereby improving the practical value of surgical videos as teaching and reference resources. The present invention breaks through the limitation that existing video analysis methods can only recognize preset scenes. Through the intelligent processing of the video language model, it realizes natural interaction with the content of surgical videos, helping doctors to more efficiently acquire professional knowledge from a large number of surgical videos, and providing more advanced technical support for medical teaching, surgical evaluation and surgical skills improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 It is a flowchart of a surgical video analysis method based on a video large language model provided in an embodiment of the present invention.

[0013] Figure 2 It is a schematic diagram of a video encoder training process provided by an embodiment of the present invention.

[0014] Figure 3 It is a schematic diagram of a large video language model training process provided by an embodiment of the present invention.

[0015] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Please refer to Figure 1 , Figure 1 The flowchart of the surgical video analysis method based on the video large language model provided by the embodiment of the present invention can be implemented by a computer program, can be implemented by a single-chip microcomputer, and can also be run on a surgical video analysis system based on the video large language model based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the method may include the following steps: S101. Obtain a surgical video and text questions corresponding to the surgical video.

[0018] Among them, surgical video refers to a sequence of video data recorded during the operation that contains information such as surgical operations, instrument use, and changes in surgical sites. Specifically, surgical video can be understood as time-series data composed of continuous image frames, each of which contains visual information of the surgical site, especially the position and movement trajectory of surgical instruments and their interaction with the surgical site. These video data can be real-time collected data from the operating room or historical surgical records stored in a medical database, which are used to provide basic visual information for subsequent intelligent analysis.

[0019] Correspondingly, text questions refer to specific analysis requirements or query instructions for the content of surgical videos. In an embodiment of the present invention, text questions can be understood as query statements described in natural language, such as asking specific questions such as the surgical status, instrument position, and surgical step progress at a specific time point. Text questions can be real-time inquiries raised by surgeons during or after surgery, or they can be a set of standard questions preset for the purpose of teaching, training, surgical review, etc., to guide the video large language model to perform targeted video understanding and information extraction.

[0020] The combination of surgical videos and text questions constitutes the input data basis of the video analysis method of the present invention, which is used to support the subsequent intelligent analysis process. By processing video data and text queries simultaneously, the system can achieve accurate understanding of surgical videos and targeted information extraction, ultimately providing decision support and auxiliary analysis for medical practice.

[0021] S102: Input the surgical video and text questions into the video big language model, and obtain the text answers corresponding to the video clips in the surgical video output by the video big language model.

[0022] Among them, the video big language model refers to an artificial intelligence model that can process both video and text modal information at the same time and has the ability of video understanding and natural language interaction. In an embodiment of the present invention, the video big language model can be understood as a multimodal intelligent system composed of a video encoding module, a text encoding module and a big language module, wherein the video encoding module is specifically used to process video data and extract video features, especially focusing on the position and action features of surgical instruments; the text encoding module is responsible for converting questions in natural language form into feature representations that can be processed by the model; the big language module integrates the information of these two modalities, understands the video content and generates corresponding text answers.

[0023] Furthermore, in the large language module, the system first intelligently segments the video to ensure that each video segment contains complete semantic information, and then extracts the required information from the relevant video segments based on the specific needs of the text question.

[0024] For example, when receiving a question such as "Where were the surgical instruments at the 3rd minute of the operation?", the system will automatically locate the video clip at the 3rd minute, analyze the specific position and operating status of the surgical instruments at that time point, and generate an accurate text answer, such as "At the 3rd minute, the surgical instruments were located in the upper right quadrant of the patient's abdomen, and tissue separation operations were being performed."

[0025] The advantage of this processing method is that it can achieve accurate understanding and targeted analysis of surgical video content, helping doctors quickly obtain the specific surgical information they need. This intelligent analysis method of the video large language model significantly improves the efficiency and accuracy of surgical video analysis, providing strong technical support for medical practice, and also providing an important reference for surgical teaching and surgical quality assessment.

[0026] Based on the above embodiment, as an optional embodiment, in S102: inputting the surgical video and the text question into the video big language model, and obtaining the text answer corresponding to the video clip in the surgical video output by the video big language model, this step may further include the following steps: S201. Extract video features of the surgical video and text features of the text questions through a video big language model.

[0027] Specifically, the video language model extracts features from the input surgical video through the video encoding module. This process includes frame-level analysis of the video data, focusing on identifying and extracting key visual features such as the location information, movement trajectory, and operation status of surgical instruments.

[0028] During the feature extraction process, the video encoding module will focus on the area where the surgical instruments are located, and accurately capture the interaction details between the instruments and the surgical site through a pre-trained mask mechanism. At the same time, the text encoding module processes the input text questions and converts the questions in natural language form into text feature vectors that the model can process.

[0029] For example, when processing a text question such as "identify the location of surgical instruments at the 5th minute in a video", the text encoding module extracts the key features of the time information "5th minute" and the query target "location of surgical instruments". Through this feature extraction method, the visual information in the surgical video and the semantic information in the text question are converted into standardized feature representations.

[0030] S202. Perform segment analysis on the surgical video based on video features and text features using a video language model to obtain text answers corresponding to the video clips in the surgical video.

[0031] Specifically, the video big language model first segments the extracted video features according to the time sequence, and each video clip contains continuous surgical operation information. During the segmentation process, the system will combine the time information contained in the text features, such as the specific time point such as "the 5th minute" or the description of the surgical stage such as "the suturing stage", to intelligently segment the video. Subsequently, the big language module will convert the video features into video tokens, the text features into text tokens, and combine the two tokens according to the time sequence relationship to form a complete input sequence.

[0032] For example, when the system receives a question about the location of surgical instruments at a specific time point, it will first locate the corresponding video clip, then analyze the instrument location, movement trajectory and other key information in the clip, and finally generate an accurate text description, such as "At the 5th minute, the surgical instrument is located in the upper right quadrant of the abdominal cavity, and tissue separation is being performed." This processing method based on segmented analysis can ensure that the system understands the surgical video more accurately and the generated answers are more in line with the actual needs of doctors.

[0033] Through the above segmented analysis method, the system can not only accurately answer time-related questions, but also understand and describe the specific operation details during the operation, providing doctors with valuable surgical information support. This processing method significantly improves the accuracy and practicality of surgical video analysis, enabling the system to provide intelligent support for a variety of medical scenarios such as surgical teaching and surgical quality assessment.

[0034] The above embodiment illustrates the actual application process of the video large language model. Based on the above embodiment, the training process of the video large language model will be described below. Specifically, the process may further include the following steps: S301, obtaining sample surgical videos, sample text questions corresponding to the sample surgical videos, and an initial video large language model, wherein the initial video large language model includes a video encoding module, a text encoding module, and a large language module.

[0035] Specifically, in order for the video language model to accurately understand and analyze surgical videos, it is necessary to first optimize the model performance through the training process. Before training begins, it is necessary to obtain the basic data and model components required for training.

[0036] Specifically, we first collected a large number of real surgical videos as sample surgical videos, which included different types of surgical operations, the use scenarios of various surgical instruments, and complete surgical process information. At the same time, we collected sample text questions annotated by professional doctors for these sample surgical videos, which covered multiple query requirements such as key time points, instrument operations, and surgical steps during the operation.

[0037] In addition, it is necessary to build an initial video language model, which consists of three core modules: the video encoding module is responsible for processing video data and extracting video features, the text encoding module is used to understand text questions and extract text features, and the language module is responsible for integrating these two features and generating answers. The basic data and model components required for these training are obtained.

[0038] S302: Train the video encoding module using sample surgical videos to obtain sample video features.

[0039] After obtaining the basic training data and model components, the video encoding module needs to be trained first to improve its ability to extract surgical video features. Specifically, the training of the video encoding module adopts a feature learning strategy centered on surgical instruments. First, the sample surgical video is cut into continuous video clips, and the location area of ​​the surgical instrument is identified and marked in each clip.

[0040] During the training process, the encoder Me of the video encoding module receives the video input with the mask and predicts the mask area, and the decoder Md reconstructs the original video content based on the output of the encoder. The parameters of the video encoding module are optimized by calculating the mean square error loss between the reconstruction result and the original video.

[0041] Based on the above embodiment, as an optional embodiment, the video encoding module includes a video encoder and a video decoder. In S302, the step of training the video encoding module through the sample surgical video to obtain the sample video features may further include the following steps: S401, cutting the sample surgical video into sample video segments, and adding a mask to the target position in the sample video segment to obtain a masked area and a non-masked area.

[0042] Among them, the target position refers to the area position that needs to be focused on and analyzed in the surgical video. In an embodiment of the present invention, the target position can be understood as the spatial area where the surgical instrument is located, including the instrument body and the operating area around it. These areas usually contain the position information, morphological characteristics and interaction status of the surgical instrument with the surrounding tissue. For example, when the surgical instrument is a surgical forceps, the target position includes not only the spatial area occupied by the surgical forceps itself, but also the area where its clamping part contacts the surgical part; when an electric knife is used for cutting, the target position includes the electric knife and its action area.

[0043] Furthermore, by adding a mask to the target position, the video encoding module can be guided to focus on learning the feature representation of surgical instruments; secondly, by predicting the target position information blocked by the mask, the model's ability to understand the operation of surgical instruments is improved; the target position information is also used to help the model establish the relationship between the instrument position, movement trajectory and operation behavior during the operation, thereby improving the model's overall understanding of the surgical video. In this way, the model can better capture key information during the operation and provide more accurate feature representation for subsequent video analysis.

[0044] Please refer to Figure 2 , Figure 2 A schematic diagram of a video encoder training process provided for an embodiment of the present application. Specifically, the complete sample surgical video is cut into multiple continuous sample video clips at fixed time intervals, and each clip needs to contain complete semantic information of the surgical operation. After obtaining the sample video clips, the system will automatically identify the target position in each clip, that is, the spatial area where the surgical instrument is located, including the instrument body and the operating area around it. For example, when the video clip shows surgical forceps clamping tissue, the target position includes the surgical forceps body and the area where its clamping part contacts the surgical site. Subsequently, masks are added to these identified target positions, that is, these areas are covered with preset occlusion marks to form masked areas. Other areas in the video that are not blocked constitute non-masked areas, which usually contain auxiliary information such as surgical background environment and anatomical structure.

[0045] S402: Optimize the video encoder based on the masked area and the non-masked area.

[0046] Specifically, the video encoding module contains two key components: encoder Me and decoder Md. The encoder Me is responsible for converting masked video clips into feature representations, and the decoder Md is responsible for reconstructing the original video content based on these features. During the training process, the masked video clips are first input into the encoder Me. The encoder will process the video content containing masked areas and non-masked areas and generate corresponding feature codes. Subsequently, these feature codes are input into the decoder Md, and the decoder will try to reconstruct the areas obscured by the mask, that is, predict the position and morphological features of the surgical instruments. By calculating the mean square error loss between the reconstructed result and the original video, the formula is expressed as follows: ; In the formula, ; In the formula, represents the mask area, represents the non-masked area, represents the encoder, Represents a decoder.

[0047] For example, when the model processes a video clip containing a surgical forceps operation, even if the surgical forceps area is blocked by a mask, the model can accurately infer and reconstruct the position and operating state of the surgical forceps through the surrounding non-masked area information. This mask reconstruction-based training method enables the video encoder to learn richer visual feature representations, especially a deeper understanding of the spatial position, morphological characteristics, and operating modes of surgical instruments. Through this optimization process, the video encoder can not only accurately capture key information in surgical videos, but also adapt to the instrument recognition and tracking requirements in different surgical scenarios, providing high-quality feature representations for subsequent video analysis tasks, significantly improving the performance and reliability of the model in practical applications.

[0048] S303: Train the text encoding module through sample text questions to obtain sample text features.

[0049] Specifically, the text encoding module receives sample text questions as input, which include various query requirements during the operation, such as "Where are the surgical instruments located at the 3rd minute of the video?", "Is there any bleeding during the operation?" and other professional questions. During the training process, the sample text questions are first preprocessed, including operations such as word segmentation and tokenization, to convert the questions in natural language form into input sequences that can be processed by the model. Subsequently, the text encoding module processes these input sequences through the attention mechanism, focusing on key information in the questions, such as time points, query targets, operation types, etc. For example, when dealing with a question like "Identify the operating status of surgical forceps at the 5th minute", the text encoding module will pay special attention to the time information of "the 5th minute" and the query target of "the operating status of surgical forceps".

[0050] In this way, the text encoding module gradually learns to map different types of surgery-related questions into standardized sample text features. These features not only contain the semantic information of the questions, but also retain key attribute information such as time and space, providing a reliable text representation for subsequent multimodal analysis.

[0051] Based on the above embodiment, as an optional embodiment, the video encoding module and the text encoding module can also be comparatively learned based on sample video features and sample text features to align the output features of the video encoder and the text encoding module in time.

[0052] Specifically, paired sample video features and sample text features are first selected from the training samples, which come from the video content and the corresponding text description at the same time point or the same surgical stage, respectively.

[0053] For example, when the sample text feature describes "the surgical forceps separate the tissue at the 5th minute", it needs to be paired with the surgical operation clip feature at the 5th minute in the video. In the contrastive learning process, the contrastive loss function is used to measure the similarity between the two modal features. For positive sample pairs (i.e., video features and text features at the same time point), the system will maximize their similarity in the feature space; for negative sample pairs (i.e., video features and text features at different time points), their similarity is minimized.

[0054] In this way, the system gradually optimizes the parameters of the video encoding module and the text encoding module so that the features they output form a temporally consistent representation in the semantic space. For example, when the video shows surgical instruments performing suturing operations, the suturing-related features mentioned in the text description will have a high degree of similarity with the video features at the corresponding time point. This temporal alignment strategy based on contrastive learning significantly improves the model's ability to understand the temporal relationship between surgical video content and text descriptions, allowing the system to more accurately locate and analyze surgical operations at specific time points.

[0055] By adopting the above technical solutions, not only the performance of the model in the time series analysis task is enhanced, but also the accuracy of the subsequent question-answering system is improved, providing a more reliable feature basis for the intelligent analysis of surgical videos. This time-aligned feature representation enables the model to better understand and answer surgical-related questions involving time series information.

[0056] S304: Train the large language module through sample video features and sample text features to obtain sample text answers.

[0057] Please refer to Figure 3 , Figure 3A schematic diagram of a video large language model training process is shown. Based on the above embodiment, as an optional embodiment, the video large language model also includes a classifier. In S304: the large language module is trained by sample video features and sample text features to obtain a sample text answer. Specifically, the following steps may also be included: S501: Train a classifier using sample text features to obtain classification results of the sample text features.

[0058] During the training of the large language module, the classifier needs to be specially trained in order to better understand and classify different types of surgical problems.

[0059] Specifically, the classifier receives sample text features processed by the text encoding module as input, which contain various types of surgical question information. During the training process, the sample text features are first classified and labeled, and the questions are divided into different categories, such as time query (asking about the surgical status at a specific time point), operation type query (asking about the specific operation of the surgical instrument), location query (asking about the spatial location of the surgical instrument), etc.

[0060] For example, when dealing with a question like "Where is the position of the surgical forceps at the 5th minute?", the classifier will identify it as a location query question. By classifying and learning the sample text features, the classifier gradually establishes a mapping relationship between the question type and the feature representation. During the training process, the cross entropy loss function is used to evaluate the classification accuracy, and the classifier parameters are optimized through back propagation. This classification training method enables the system to accurately identify different types of surgical questions, thereby providing more targeted processing strategies for subsequent answer generation.

[0061] Furthermore, when the system identifies a time query type question, it will pay special attention to the timing information in the video features; when it identifies an operation type query, it will focus on analyzing the motion features of surgical instruments. The classifier trained in this way can accurately understand the type of question and the query intent, providing important support for generating more accurate answers. This classification-based problem understanding method significantly improves the accuracy and efficiency of the system in dealing with different types of surgical problems, and provides a more detailed feature classification basis for the intelligent analysis of surgical videos.

[0062] S502: Convert sample video features into video tokens, and convert sample text features into text tokens.

[0063] After obtaining the sample video features and sample text features, the features of these two different modalities need to be converted into a unified token representation so that the large language module can perform subsequent multimodal analysis.

[0064] Specifically, the sample video features are first processed, and the continuous video features are mapped to discrete video token space through linear projection and position encoding. This process retains the temporal and spatial information in the video features. For example, when the video shows surgical forceps performing a suturing operation, the corresponding video token will contain key information such as the instrument position and movement trajectory. At the same time, the sample text features are processed and converted into a standard text token sequence.

[0065] During the conversion process, the system considers key elements such as professional terms, time information, and query targets in the text to ensure that the converted text tokens can accurately express the semantic content of the question. For example, when dealing with a question like "Identify the operating status of surgical forceps at the 5th minute", the system converts it into a token sequence containing time tags, instrument tags, and operation tags. Through this feature-to-token conversion process, the system establishes a unified representation between video content and text questions, allowing the large language module to process information from these two modalities in the same feature space.

[0066] Through the above conversion, not only the key information of the original features is retained, but also the processing efficiency of the model is improved through standardized token representation. This unified token-based representation method provides a reliable data foundation for subsequent multimodal analysis and answer generation, and significantly improves the performance of the system in surgical video analysis tasks.

[0067] S503: Input the video token and the text token into the large language module, so that the large language module generates sample text answers of the video token and the text token based on the classification results.

[0068] Specifically, the large language module first receives the converted video token and text token sequences, and combines them with the question classification results output by the classifier. During the processing, the large language module uses a multi-head attention mechanism to interactively analyze the two token sequences, and adjusts the processing strategies for different types of questions based on the classification results.

[0069] For example, when the classification result indicates that this is a time query question, such as "the operating status of the surgical forceps at the 5th minute", the system will focus on the information of the corresponding time point in the video token and highlight the relevant visual features through the attention weight. In the process of generating answers, the large language module gradually generates text answers through the decoder, while considering the specific needs of the question type. For location query questions, the system will generate a detailed description containing spatial location information; for operation type queries, it will focus on describing the specific operating status of the instrument. The model parameters are optimized by calculating the cross entropy loss between the generated sample text answers and the standard answers, prompting the system to generate more accurate and professional answers.

[0070] Through the above-mentioned multimodal analysis method based on classification perception, the large language module can generate more targeted answers according to different types of questions. For example, for the same video showing the operation of surgical instruments, the system can generate targeted professional descriptions such as "At the 5th minute, the surgical forceps are located in the upper right quadrant of the abdomen" or "The surgical forceps are performing tissue separation operations with moderate force" according to different question types. This training method significantly improves the system's performance in surgical video analysis and question-answering tasks, making the generated answers more accurate, professional and in line with medical practice needs.

[0071] S305. Fine-tune the initial video large language model based on the sample text answer to obtain a video large language model.

[0072] After completing the basic training of each module, the entire video language model needs to be fine-tuned end-to-end to improve the overall performance and generalization ability of the model.

[0073] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the surgical video analysis method based on the video large language model.

[0074] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical video analysis method based on the video large language model provided by the above-mentioned methods.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the surgical video analysis method based on the video large language model provided by the above-mentioned methods.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical video analysis method based on a video large language model, characterized in that: include: Obtaining a surgical video and text questions corresponding to the surgical video; The surgical video and the text question are input into a video language model, and a text answer corresponding to the video clip in the surgical video output by the video language model is obtained.

2. The surgical video analysis method based on the video large language model according to claim 1 is characterized in that: Inputting the surgical video and the text question into a video language model, and obtaining a text answer corresponding to a video clip in the surgical video output by the video language model, including: Extracting video features of the surgical video and text features of the text question through the video big language model; The surgical video is segmented and analyzed based on the video features and the text features by the video big language model to obtain text answers corresponding to the video segments in the surgical video.

3. The surgical video analysis method based on the video large language model according to claim 1 or 2, characterized in that: The training process of the video large language model includes: Obtaining a sample surgical video, a sample text question corresponding to the sample surgical video, and an initial video large language model, wherein the initial video large language model includes a video encoding module, a text encoding module, and a large language module; The video encoding module is trained by using the sample surgical video to obtain sample video features; The text encoding module is trained by the sample text problem to obtain sample text features; The large language module is trained by using the sample video features and the sample text features to obtain a sample text answer; The initial video large language model is fine-tuned based on the sample text answer to obtain the video large language model.

4. The surgical video analysis method based on the video large language model according to claim 3 is characterized in that: The video encoding module includes a video encoder, The video encoding module is trained by the sample surgical video to obtain sample video features, including: Cutting the sample surgical video into sample video segments, and adding a mask to the target position in the sample video segment to obtain a masked area and a non-masked area; The video encoder is optimized based on the masked region and the unmasked region.

5. The surgical video analysis method based on the video large language model according to claim 4 is characterized in that: The target location includes the location of the instrument.

6. The surgical video analysis method based on the video large language model according to claim 3 is characterized in that: The method further comprises: Based on the sample video features and the sample text features, comparative learning is performed on the video encoding module and the text encoding module to align output features of the video encoder and the text encoding module in time.

7. The surgical video analysis method based on the video large language model according to claim 3 is characterized in that: The video large language model also includes a classifier; The step of training the large language module by using the sample video features and the sample text features to obtain a sample text answer includes: The classifier is trained by the sample text feature to obtain a classification result of the sample text feature; Convert the sample video features into video tokens, and convert the sample text features into text tokens; The video token and the text token are input into a large language module, so that the large language module generates sample text answers of the video token and the text token based on the classification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the surgical video analysis method based on the video large language model as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the surgical video analysis method based on the video large language model as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the surgical video analysis method based on the video large language model as described in any one of claims 1 to 7 is implemented.

Images