Surgical video processing method and device based on multi-modal large model

Abstract

The application provides a surgery video processing method and device based on a multimodal large model, wherein the method comprises the following steps: determining a surgery video and an original question related to the surgery video; splitting the surgery video to obtain a plurality of video paragraphs with a fixed frame number; performing coding processing on each video paragraph in the plurality of video paragraphs by using a pre-trained video encoder to obtain abstract features; converting the spatial dimension of the abstract features to be consistent with the spatial dimension of a preset multimodal large model by using a preset multimodal converter to obtain processed abstract features; performing cross embedding on the processed abstract features and a text auxiliary description to obtain mixed abstract features; and inputting the mixed abstract features and the original question into the preset multimodal large model to obtain text answer content output by the preset multimodal large model. The application can enhance the interactivity and flexibility of the model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a surgical video processing method and apparatus based on a multimodal large model. Background Technology

[0002] In today's medical field, with the rapid development of computer technology, the analysis and understanding of surgical videos has become a key technology for improving surgical precision and patient safety.

[0003] However, although existing surgical video understanding algorithms have assisted the surgical process to some extent, these algorithms are mainly designed for single tasks, such as stage recognition or tool detection, and often provide only limited outputs, lacking interactivity with doctors.

[0004] This shows that the surgical video understanding methods in related technologies have technical problems such as poor interactivity and low practicality. Summary of the Invention

[0005] This invention provides a surgical video processing method and apparatus based on a multimodal large model to address the shortcomings of existing surgical video understanding methods, such as poor interactivity and low practicality, so as to enable multi-task question answering based on natural language while parsing surgical videos.

[0006] This invention provides a surgical video processing method based on a multimodal large model, comprising the following steps: determining the surgical video and an original question associated with the surgical video; splitting the surgical video into multiple video segments with a fixed number of frames; encoding each video segment using a pre-trained video encoder to obtain abstract features; converting the spatial dimension of the abstract features to match the spatial dimension of a preset multimodal large model using a preset multimodal converter to obtain processed abstract features; cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual response output by the preset multimodal large model.

[0007] According to a surgical video processing method based on a multimodal large model provided by the present invention, before determining the surgical video and the original problem associated with the surgical video, the method further includes: training a preset video encoder to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align the video features of surgical instruments with the text features of surgical instruments.

[0008] According to a surgical video processing method based on a multimodal large model provided by the present invention, the step of training a preset video encoder to obtain a pre-trained video encoder includes: acquiring surgical video training samples, wherein the surgical video training samples include at least one surgical instrument marker and surgical instrument text features corresponding to the surgical instrument marker; masking the surgical video training samples using the preset video encoder to obtain masked video training samples; encoding the masked video training samples using the encoder of the preset video encoder to obtain encoded video training samples; reshaping the encoded video training samples using the decoder of the preset video encoder to obtain surgical instrument video features; and updating the preset video encoder based on a preset loss function between the surgical instrument video features and the surgical instrument text features to obtain a pre-trained video encoder.

[0009] According to a surgical video processing method based on a multimodal large model provided by the present invention, the step of splitting the surgical video into multiple video segments with a fixed number of frames includes: converting the surgical video into continuous image frames; and converting the continuous image frames into multiple video segments according to a preset target number of frames, wherein the number of frames in each video segment is the preset target number of frames.

[0010] According to a surgical video processing method based on a multimodal large model provided by the present invention, the method for cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features includes: determining the timeline segment of the video segment corresponding to the processed abstract features; generating textual auxiliary descriptions based on the timeline segments according to a preset textual description template; converting the textual auxiliary descriptions through a preset text token processor to obtain textual abstract features; and cross-embedding the processed abstract features with the textual abstract features to obtain hybrid abstract features.

[0011] According to a surgical video processing method based on a multimodal large model provided by the present invention, the step of inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the text response content output by the preset multimodal large model includes: determining the question type of the original question; calling the preset multimodal large model to determine a target pre-trained low-rank adaptation module based on the question type; and outputting the text response content based on the target pre-trained low-rank adaptation module according to the hybrid abstract features and the original question.

[0012] This invention also provides a surgical video processing device based on a multimodal large model, comprising the following modules: a determination module for determining the surgical video and an original question related to the surgical video; a splitting module for splitting the surgical video into multiple video segments with a fixed number of frames; an encoding module for encoding each video segment in the multiple video segments using a pre-trained video encoder to obtain abstract features; a conversion module for converting the spatial dimension of the abstract features to be consistent with the spatial dimension of a preset multimodal large model using a preset multimodal converter to obtain processed abstract features; an embedding module for cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and an output module for inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual answer content output by the preset multimodal large model.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the surgical video processing method based on a multimodal large model as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the surgical video processing method based on a multimodal large model as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical video processing method based on a multimodal large model as described above.

[0016] The present invention provides a surgical video processing method and apparatus based on a multimodal large model. This method involves identifying the surgical video and related original questions to facilitate the association of corresponding video segments with the questions; splitting the surgical video into multiple video segments with a fixed number of frames; encoding each video segment using a pre-trained video encoder to obtain abstract features; transforming the spatial dimension of the abstract features to match the spatial dimension of the pre-set multimodal large model using a pre-defined multimodal converter to obtain processed abstract features; cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and inputting the hybrid abstract features and the original questions into the pre-defined multimodal large model to obtain the textual response output by the model. Thus, by combining a video encoder and a multimodal large model, this method supports multi-task question answering based on natural language while parsing the surgical video. In this way, the model can be queried directly using natural language to obtain real-time feedback and detailed analysis of the surgical progress, greatly enhancing the model's interactivity and versatility. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the surgical video processing method based on a multimodal large model provided by the present invention.

[0019] Figure 2 This is a flowchart of surgical video understanding based on a multimodal large model provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the process for Phase 1: Surgical Instrument Mask Pre-training provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the process for aligning the text description of the surgical video in Stage Two, provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the process for Phase 3: multimodal large model training provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the surgical video processing device based on a multimodal large model provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] In today's medical field, with the rapid development of computer technology, the analysis and understanding of surgical videos has become a key technology for improving surgical precision and patient safety. Multimodal large models, by combining video analytics with natural language processing, offer unprecedented new approaches to surgical decision support. These models can not only analyze surgical videos but also optimize surgical outcomes and improve operational efficiency by understanding the multidimensional information of the surgical scenario.

[0027] However, while existing surgical video understanding algorithms have assisted the surgical process to some extent, these algorithms are primarily designed for single tasks, such as stage identification or tool detection, and often provide only limited output, lacking interactivity with the surgeon. Surgeons cannot effectively communicate with these models using natural language to gain in-depth insights and answers regarding specific clinical situations. Furthermore, these algorithms typically perform poorly when handling combined multi-task tasks, such as simultaneously performing surgical tool detection and stage identification, further limiting their practicality and flexibility.

[0028] To address these challenges, this invention proposes a surgical video processing method based on a multimodal large model. This method combines a video encoder and a large language model, enabling it to support multi-task question answering based on natural language while parsing surgical videos. In this way, doctors can directly query the model using natural language to obtain real-time feedback and detailed analysis of surgical progress, greatly enhancing the model's interactivity and versatility. Furthermore, this invention considers the complexity and dynamic changes of surgical scenarios, using deep learning and pattern recognition technologies to accurately identify and respond to doctors' query needs in varied surgical environments. This invention not only improves the accuracy of surgical video understanding but also provides surgical teams with a powerful decision support tool, thereby achieving higher surgical efficiency and better patient treatment outcomes.

[0029] The purpose of this invention is to address the problems of limited parsing and understanding functions and poor interactivity in existing surgical videos, so as to enhance the interactivity and versatility of the model, improve surgical efficiency, and achieve better patient treatment outcomes.

[0030] To achieve the above objectives, this invention provides a surgical video processing method based on a multimodal large model, which utilizes a video encoder and a language large model to perform in-depth understanding and analysis of surgical videos.

[0031] refer to Figure 1 , Figure 1 This is a flowchart illustrating the surgical video processing method based on a multimodal large model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0032] Step 101: Identify the surgical video and the original questions associated with the surgical video.

[0033] In this embodiment of the invention, the surgical video can be real-time transmitted video data or pre-stored surgical video data; and the original questions related to the surgical video can be multi-task question-and-answer based on natural language by the user (doctor), such as what surgical instruments should be used for the current stage of the surgery, or what drugs should be selected for the current patient condition, etc. This invention does not limit these questions.

[0034] In this embodiment of the invention, after obtaining the original questions related to the surgical video, the original questions are preprocessed to obtain text vectors that can be recognized by a preset multimodal large model.

[0035] For example, the original problem text is segmented into multiple words or tags (removing common words such as interjections that do not significantly help in understanding the meaning of the text); multiple words are embedded using a pre-set word embedding model (e.g., Word2Vec) to obtain a text sequence (positional encoding is added to preserve the order information of words in the sequence); the processed word embedding sequence (including other types of embedding, such as paragraph embedding, positional embedding, etc.) is passed to a pre-trained video encoder and a pre-set multimodal large model.

[0036] Step 102: The surgical video is split into multiple video segments with a fixed number of frames.

[0037] In this embodiment of the invention, the surgical video is converted into continuous image frames; according to a preset target number of frames, the continuous image frames are converted into multiple video segments, wherein the number of frames in the video segments is the preset target number of frames.

[0038] For example, the surgical video V can be divided into multiple shorter video segments {v1, v2, v3…, v...} based on a specific number of frames. n}, where n represents the sequence number of the video segment (the total number of video segments).

[0039] Step 103: Encode each video segment from multiple video segments using a pre-trained video encoder to obtain abstract features.

[0040] In this embodiment of the invention, based on a specific number of frames, consecutive image frames are converted into shorter video segments containing fewer image frames, and each shorter video segment is processed by a pre-trained video encoder to obtain abstract features.

[0041] Here, the pre-trained video encoder is a model trained on a large-scale dataset to learn a general representation of video content, thereby enabling it to handle various video understanding tasks. In this embodiment of the invention, the pre-trained video encoder is used to solve a wide range of video understanding tasks, including classification, localization, retrieval, caption generation, and question answering.

[0042] In this embodiment of the invention, the pre-trained video encoder is pre-trained using a two-stage training method. The first stage is comparative learning, and the second stage is predicting occluded video blocks.

[0043] In the contrastive learning process, all video-text pairs are used to align the video encoder with the text encoder. Symmetric cross-entropy loss is minimized by minimizing the similarity scores of all video-text pairs in the batch.

[0044] In predicting masked video patches, a first-stage video-level global embedding and token-based embedding are predicted based on the unmasked input video patches. The encoder's output tokens are randomly shuffled before being passed to the decoder to avoid learning shortcuts. Global and token-based semantic embeddings are refined using extensive plain video data to improve masked video modeling.

[0045] Step 104: The spatial dimension of the abstract feature is converted to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter, and the processed abstract feature is obtained.

[0046] In this embodiment of the invention, the preset multimodal converter can be Q-Former (QueryingTransformer), which is used for efficient interaction and fusion between visual and language models.

[0047] The abstract features are processed by Q-Former to obtain Q = {q1, q2, q3, ..., q n This ensures that its spatial dimension is consistent with that of a large multimodal model. Specifically, Q-Former uses a learnable set of query vectors (denoted as Q={q1, q2,..., q...}). n}, where n is the vector index, extracts visual features from the visual model and transforms these features into a representation consistent with the spatial dimensions of the pre-defined multimodal large model.

[0048] Through the steps described above, Q-Former can transform abstract visual features (represented by query vectors) into representations consistent with the spatial dimensions of large multimodal models. This representation not only helps large language models understand and generate text related to visual content, but also improves the overall performance and generalization ability of multimodal learning models.

[0049] Step 105: Cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features.

[0050] In this embodiment of the invention, the textual auxiliary description is used to describe the start and end times of the video segment corresponding to the processed abstract features. For example, this is a video segment from second 0 to second d.

[0051] In this embodiment of the invention, a timeline segment of the video segment (in the surgical video) corresponding to the processed abstract feature is determined; a text auxiliary description is generated based on the timeline segment according to a preset text description template; the text auxiliary description is converted by a preset text token processor to obtain the text abstract feature; and the processed abstract feature and the text abstract feature are cross-embedded to obtain the hybrid abstract feature.

[0052] Step 106: Input the hybrid abstract features and the original question into the preset multimodal large model to obtain the text answer output by the preset multimodal large model.

[0053] In this embodiment of the invention, a preset multimodal approach obtains hybrid abstract features and the original question. For different types of questions, the preset multimodal approach selects the corresponding LoRA module based on its type, making the response to the question more targeted.

[0054] Here, the LoRA module (Low-Rank Adaptation of Large Language Models) is a low-rank parameter fine-tuning technique for large language models (LLMs). By training a low-rank matrix and injecting its parameters into the large language model, customized adjustments can be made to the model's generative style or task-specific capabilities.

[0055] Through the steps described above in this embodiment of the invention, a surgical video and an original question related to the surgical video are determined; the surgical video is split into multiple video segments with a fixed number of frames; each video segment is encoded using a pre-trained video encoder to obtain abstract features; the spatial dimension of the abstract features is transformed to match the spatial dimension of a pre-defined multimodal large model using a preset multimodal converter to obtain processed abstract features; the processed abstract features are cross-embedded with textual auxiliary descriptions to obtain hybrid abstract features; the hybrid abstract features and the original question are input into a preset multimodal large model to obtain the textual response output by the preset multimodal large model; thus solving the technical problems of poor interactivity and low practicality in surgical video understanding methods in related technologies.

[0056] According to the surgical video processing method based on a multimodal large model provided by the present invention, before determining the surgical video and the original problems related to the surgical video, the method further includes:

[0057] The preset video encoder is trained to obtain a pre-trained video encoder, which is used to align the video features of surgical instruments with the text features of surgical instruments.

[0058] In this embodiment of the invention, the pre-trained video encoder V e The process involves generating a mask for the instrument's location, reading the video content into the encoder via an autoencoder, and then reshaping the mask using a decoder. Afterward, the autoencoder is aligned with the corresponding text description.

[0059] According to the present invention, a surgical video processing method based on a multimodal large model is provided, which trains a preset video encoder to obtain a pre-trained video encoder, including:

[0060] Obtain surgical video training samples, wherein the surgical video training samples include at least one surgical instrument mark and surgical instrument text features corresponding to the surgical instrument mark;

[0061] The surgical video training samples are masked by an instrument using a preset video encoder to obtain masked video training samples.

[0062] The masked video training samples are encoded by the encoder of the preset video encoder to obtain the encoded video training samples.

[0063] The surgical instrument video features are obtained by masking and reshaping the encoded video training samples using the decoder of the preset video encoder.

[0064] Based on a preset loss function between the video features and text features of surgical instruments, the preset video encoder is updated to obtain a pre-trained video encoder.

[0065] In this embodiment of the invention, a pre-trained video encoder is used to identify specific objects (such as the location of medical devices) in the video, and generates a mask for the specific objects through an autoencoder structure, and further aligns the video content with the text description.

[0066] In this embodiment of the invention, video data containing the target medical device is collected. The video content is ensured to be diverse, covering different devices, locations, and angles. Each frame in the video data is labeled to clearly mark the location of the medical device (using bounding boxes or pixel-level masks). Detailed textual descriptions are provided for each video segment or keyframe within the video, including the device's name, location, and actions, to obtain surgical video training samples.

[0067] The preset video encoder includes an encoder part and a decoder part. The pre-trained video encoder is used as the encoder part of the autoencoder to extract features from the input video. The decoder part should have a structure that matches the encoder but is in the opposite direction. It is used to reconstruct video frames or generate a mask for the position of the device from the encoded features.

[0068] Based on a pre-defined loss function (e.g., cross-entropy loss function) between the video features and text features of surgical instruments, an autoencoder is trained using a labeled video dataset. The parameters of the encoder and decoder are then optimized using a backpropagation algorithm to obtain a pre-trained video encoder.

[0069] According to the present invention, a surgical video processing method based on a multimodal large model is provided, which splits the surgical video to obtain multiple video segments with a fixed number of frames, including:

[0070] Convert surgical videos into continuous image frames;

[0071] Based on a preset target number of frames, consecutive image frames are converted into multiple video segments, where the number of frames in each video segment is the preset target number of frames.

[0072] In this embodiment of the invention, the surgical video V is divided into multiple shorter video segments {v1, v2, v3…, v...} according to a specific number of frames. n}, where n represents the total number of video segments, and v represents the number of video segments in the surgical video V. n Contains a series of consecutive image frames {x1,x2,x3,…,x} m}, where m represents the total number of frames (the preset target number of frames), x t∈ R H×W×M Each x t This represents a frame in the video. H, W, and M represent the height, width, and number of color channels of the image, respectively (M=3 for RGB images).

[0073] Based on a specific frame number d, consecutive image frames {x1,x2,x3,…,x} are selected. m} Convert into a shorter video segment v1={x1,x2,x3,…x d},v2={x d+1 ,x d+2 ,x d+3 ,…x d+d-1},…vn={x (n-1)*d+1 , x (n-1)*d+2 ,x (n-1)*d+3, …, x n*d-1}

[0074] The video segments of a surgical video can be represented by the following formula (1):

[0075] (1)

[0076] in, Indicates surgical video The first in A video segment, Represents a video frame. Indicates the sequence number of the video segment. Indicates the sequence number of the video frame. This indicates the preset target number of frames.

[0077] After obtaining multiple video segments with a fixed number of frames, the above method also includes:

[0078] Each shorter video segment After video encoder The abstract feature E = {e1,e2,e3,…,e} is then obtained. n}

[0079] Each feature in the abstract features can be represented by the following formula (2):

[0080] (2)

[0081] in, Representing the first abstract feature One characteristic, Indicates surgical video The first in One video segment; Indicates the sequence number of the video segment.

[0082] The abstract features are processed by Q-Former to obtain Q = {q1, q2, q3, ..., q n}, so that its spatial dimension is consistent with the spatial dimension of the large model (multimodal large model), can be expressed by the following formula (3):

[0083] (3)

[0084] in, This represents the abstract features after processing. The first abstract feature after processing One characteristic, This indicates a preset multimode converter. Representing the first abstract feature One characteristic.

[0085] According to the present invention, a surgical video processing method based on a multimodal large model is provided, which obtains hybrid abstract features by cross-embedding the processed abstract features with textual auxiliary descriptions, including:

[0086] Determine the timeline segments of the video segments corresponding to the processed abstract features;

[0087] Based on the timeline paragraphs, text auxiliary descriptions are generated according to preset text description templates;

[0088] The textual auxiliary description is transformed by a preset text token processor to obtain the textual abstract features;

[0089] Hybrid abstract features are obtained by cross-embedding the processed abstract features with textual abstract features.

[0090] In this embodiment of the invention, the processed abstract features are combined with the built-in text auxiliary description segment T. default Cross-embedding yields a hybrid abstract feature M, which serves as the input to the larger model. The specific process is as follows:

[0091] First, add matching text before each short video segment, for example: {"This is a video segment from second 0 to second d", q1; "This is a video segment from second d+1 to second 2d-1", q2; "This is a video segment from second 2d to second 3d-1", q3; ...; "This is a video segment from second (n-1)*d to second n*d-1", q n}

[0092] Here, each character is transformed into a text abstract feature by a text token processor, and finally mixed to obtain a mixed abstract feature M={M1,M2,M3,…,M…} nSpecifically, it can be expressed by the following formula (4):

[0093] (4)

[0094] in, Represents the first in the mixed abstract features One characteristic, This indicates that the preset text token processor is used to cross-embed the processed abstract features with text abstract features. Indicates the first Abstract features of a text, Indicates the first A processed abstract feature, Indicates an index. Indicates the total number of indexes.

[0095] According to the present invention, a surgical video processing method based on a multimodal large model is provided, which inputs hybrid abstract features and the original question into a preset multimodal large model to obtain the text response output by the preset multimodal large model, including:

[0096] Determine the problem type of the original problem;

[0097] The pre-defined multimodal large model is invoked based on the problem type to determine the target and pre-train a low-rank adaptive module;

[0098] The target-pretrained low-rank adaptive module outputs textual answers based on hybrid abstract features and the original question.

[0099] In this embodiment of the invention, a preset multimodal large model is used to obtain hybrid abstract features M={M1,M2,M3,…,M…} n The model receives the mixed abstract features M and the original question T as input, and outputs the text answer content. For different types of questions, the large model will select the corresponding LoRA module according to its type, so that the answer to the question is more targeted. The preset multimodal large model receives the mixed abstract features M and the original question T raised by the user as input.

[0100] When dealing with different types of problems, the pre-defined multimodal large model can select corresponding pre-trained LoRA modules based on the problem type or domain. Each LoRA module is trained for a specific type or domain of problem, thus capturing the language patterns and knowledge unique to that type of problem, thereby generating more accurate and targeted responses. By training multiple LoRA modules for different tasks or domains, the pre-defined multimodal large model can easily expand its capabilities without retraining the entire model. This allows the model to quickly adapt to new scenarios and requirements.

[0101] refer to Figure 2 , Figure 2 This is a flowchart of surgical video understanding based on a multimodal large model provided by the present invention, which specifically includes the following steps.

[0102] Step 201, pre-trained video encoder.

[0103] Step 202: Receive video and text commands.

[0104] Step 203: The video is cut into shorter sets of segments and processed by a pre-trained video encoder.

[0105] Step 204: Extract abstract video features using Q-Former.

[0106] Step 205: The extracted video abstract features are mixed and embedded with textual auxiliary descriptions.

[0107] Step 206: Input the text commands of the model and the hybrid embedded features into the large model.

[0108] Step 207: The large model generates an answer.

[0109] The embodiments provided by this invention combine a multimodal large model with video coding technology to enhance the depth and breadth of surgical video understanding. By segmenting continuous surgical videos and transforming them into abstract features, and leveraging the deep learning capabilities of the large model, comprehensive analysis and real-time interaction of the surgical process are achieved. The surgical instrument mask pre-training proposed in this invention ensures that the extraction of abstract features from the surgical video primarily focuses on changes and movements of the instruments. Furthermore, the cross-embedding of built-in text descriptions and video features enhances the model's semantic understanding of surgical videos, enabling the model not only to answer queries about surgical stages but also to analyze surgical tools and operational details.

[0110] The surgical video processing method based on a multimodal large model provided by this invention comprises three main network modules: a video encoder, a Q-Former, and a multi-expert hybrid fine-tuning large language model. The first module, the video encoder, takes the surgical video as input and compresses and extracts its visual content. The second module, the Q-Former, aligns the input space of the video encoder with the input space of the large model. The third module, the multi-expert hybrid fine-tuning large language model, answers the doctor's questions based on the video content, providing responses in natural language.

[0111] refer to Figure 3 , Figure 3This is a flowchart of the first stage of the present invention: pre-training of surgical instrument masks, including inputting surgical video (a series of image frames), mask (instrument mask) processing, encoder encoding, obtaining encoded video training samples, decoder decoding, and obtaining surgical instrument video features.

[0112] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second stage of the present invention: alignment of surgical video text descriptions, including text input, a pre-trained language representation model (BERT) that receives the text input, video input, an encoder that receives the video input, and contrast loss.

[0113] refer to Figure 5 , Figure 5 This is a flowchart of the third stage of the present invention: multimodal large model training, which includes segmenting the video input into multiple segments (including segment 1 and segment 2), a pre-trained video encoder, the original question, a gated unit network, Qformer, abstract feature 1, abstract feature 2, text-assisted embedding, LoRA modules (LoRA1, LoRA2, LoRA3, LoRA4), a language large model, and text output.

[0114] In practical applications, taking laparoscopic cholecystectomy endoscopic video as an example, the embodiments of the present invention specifically include the following steps:

[0115] Step S0, for laparoscopic cholecystectomy, the endoscopic video is compared with the video encoder V. e Pre-training is performed to generate a mask for the instrument's location. Video content is then read into the autoencoder, and the decoder reconstructs the mask. Subsequently, the autoencoder is aligned with the corresponding text descriptions, and the model parameters are updated by calculating the cross-entropy loss function between video and text features.

[0116] Step S1: Receive the endoscopic video (V) of the laparoscopic cholecystectomy and the doctor's specific questions (T).

[0117] Step S2: Divide the endoscopic video V of the laparoscopic cholecystectomy into multiple shorter video segments {v1, v2, v3…, v3} based on a specific number of frames. n Each shorter video segment is processed by the video encoder V. e We obtain the abstract features {e1,e2,e3,…,e n The specific process is as follows:

[0118] First, the endoscopic video V of laparoscopic cholecystectomy contains a series of consecutive image frames {x1,x2,x3,…,x…}m}, where m represents the total number of frames (the preset target number of frames), x t ∈ R H×W×M Each x t This represents a frame in the video. H, W, and M represent the height, width, and number of color channels of the image, respectively (M=3 for RGB images).

[0119] Secondly, based on a specific frame number d, consecutive image frames {x1,x2,x3,…,x} are... m} Convert into a shorter video segment v1={x1,x2,x3,…x d},v2={x d+1 ,x d+2 ,x d+3 ,…x d+d-1},…vn={x (n-1)*d+1 ,x (n-1)*d+2 , x (n-1)*d+3, …, x n*d-1}

[0120] The video segments of the surgical video can be represented by the following formula (5):

[0121] (5)

[0122] Finally, each shorter video segment v is processed by the video encoder V. e The abstract feature E = {e1,e2,e3,…,e} is then obtained. n}

[0123] Each feature in the abstract features can be represented by the following formula (6):

[0124] (6)

[0125] in, Representing the first abstract feature One characteristic, Indicates surgical video The first in One video segment; Indicates the sequence number of the video segment.

[0126] Step S3: Pass the abstract features through Q-Former to obtain Q={q1,q2,q3,…,q n}, so that its spatial dimension is consistent with the spatial dimension of the large model can be expressed by the following formula (7):

[0127] (7)

[0128] in, This represents the abstract features after processing. The first abstract feature after processing One characteristic, This indicates a preset multimode converter. Representing the first abstract feature One characteristic.

[0129] Step S4: Combine the processed abstract features with the built-in text-assisted description segment T default Cross-embedding yields a hybrid abstract feature M, which serves as the input to the larger model. The specific process is as follows:

[0130] First, add matching text before each short video segment, for example: {"This is a video segment from second 0 to second d", q1; "This is a video segment from second d+1 to second 2d-1", q2; "This is a video segment from second 2d to second 3d-1", q3; ...; "This is a video segment from second (n-1)*d to second n*d-1", q n}

[0131] Each character is converted into a text abstract feature by the text token processor, and finally mixed to obtain a mixed abstract feature M={M1,M2,M3,…,M…} n Specifically, it can be expressed by the following formula (8):

[0132] (8)

[0133] in, Represents the first in the mixed abstract features One characteristic, This indicates that the preset text token processor is used to cross-embed the processed abstract features with text abstract features. Indicates the first Abstract features of a text, Indicates the first A processed abstract feature, Indicates an index. Indicates the total number of indexes.

[0134] Step S5: The large model obtains the mixed abstract features M={M1,M2,M3,…,M…} n} and the original problem T.

[0135] In step S6, for different types of questions, the large model will select the corresponding LoRA module based on its type, making the response to the question more targeted.

[0136] In step S6, the large model receives M and the doctor's question T as input and outputs a text answer.

[0137] The surgical video processing device based on a multimodal large model provided by the present invention will be described below. The surgical video processing device based on a multimodal large model described below and the surgical video processing method based on a multimodal large model described above can be referred to in correspondence with each other.

[0138] refer to Figure 6 , Figure 6 This is a schematic diagram of the surgical video processing device based on a multimodal large model provided by the present invention, including a determination module 601, a splitting module 602, an encoding module 603, a conversion module 604, an embedding module 605, and an output module 606.

[0139] The system comprises the following modules: a determination module 601, used to determine the surgical video and the original question related to the surgical video; a splitting module 602, used to split the surgical video into multiple video segments with a fixed number of frames; an encoding module 603, used to encode each video segment using a pre-trained video encoder to obtain abstract features; a transformation module 604, used to transform the spatial dimension of the abstract features to be consistent with the spatial dimension of a preset multimodal large model using a preset multimodal converter to obtain processed abstract features; an embedding module 605, used to cross-embed the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and an output module 606, used to input the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual answer content output by the preset multimodal large model.

[0140] Specifically, the surgical video processing device based on a multimodal large model provided by the present invention can implement all the method steps implemented in the above-described surgical video processing method embodiment based on a multimodal large model, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0141] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a surgical video processing method based on a multimodal large model. This method includes: determining the surgical video and the original question associated with the surgical video; splitting the surgical video into multiple video segments with a fixed number of frames; encoding each video segment using a pre-trained video encoder to obtain abstract features; transforming the spatial dimension of the abstract features to match the spatial dimension of the preset multimodal large model using a preset multimodal converter to obtain processed abstract features; cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual response output by the preset multimodal large model.

[0142] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical video processing method based on a multimodal large model provided by the above methods. The method includes: determining the surgical video and the original question related to the surgical video; splitting the surgical video to obtain multiple video segments with a fixed number of frames; encoding each video segment in the multiple video segments using a pre-trained video encoder to obtain abstract features; converting the spatial dimension of the abstract features to be consistent with the spatial dimension of the preset multimodal large model using a preset multimodal converter to obtain processed abstract features; cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual answer content output by the preset multimodal large model.

[0144] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the surgical video processing method based on a multimodal large model provided by the above methods. This method includes: determining a surgical video and an original question related to the surgical video; splitting the surgical video into multiple video segments with a fixed number of frames; encoding each video segment using a pre-trained video encoder to obtain abstract features; converting the spatial dimension of the abstract features to be consistent with the spatial dimension of a preset multimodal large model using a preset multimodal converter to obtain processed abstract features; cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; and inputting the hybrid abstract features and the original question into a preset multimodal large model to obtain the textual response output by the preset multimodal large model.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical video processing method based on a multimodal large model, characterized in that, include: Identify the surgical video and the original questions associated with the surgical video; The surgical video is split into multiple video segments with a fixed number of frames; Abstract features are obtained by encoding each of the multiple video segments using a pre-trained video encoder. The spatial dimension of the abstract feature is transformed to be consistent with the spatial dimension of the preset multimodal large model by a preset multimodal converter, and the processed abstract feature is obtained. Based on the cross-embedding of the processed abstract features and textual auxiliary descriptions, hybrid abstract features are obtained; The hybrid abstract features and the original question are input into the preset multimodal large model to obtain the text answer output by the preset multimodal large model; Prior to determining the surgical video and the original issues associated with the surgical video, the method further includes: A preset video encoder is trained to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align the video features of surgical instruments with the text features of surgical instruments. The step of training a preset video encoder to obtain a pre-trained video encoder includes: Obtain surgical video training samples, wherein the surgical video training samples include at least one surgical instrument mark and surgical instrument text features corresponding to the surgical instrument mark; The surgical video training samples are masked by a preset video encoder to obtain masked video training samples. The masked video training samples are encoded by the encoder of the preset video encoder to obtain encoded video training samples; The encoded video training samples are masked and reconstructed using the decoder of the preset video encoder to obtain the video features of surgical instruments. Based on a preset loss function between the video features and text features of the surgical instruments, the preset video encoder is updated to obtain a pre-trained video encoder.

2. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that, The process of splitting the surgical video into multiple video segments with a fixed number of frames includes: The surgical video is converted into a series of image frames; According to a preset target number of frames, the continuous image frames are converted into multiple video segments, wherein the number of frames in each video segment is the preset target number of frames.

3. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that, The process of cross-embedding the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features includes: Determine the timeline segment of the video segment corresponding to the processed abstract features; Based on the timeline segments, auxiliary text descriptions are generated according to preset text description templates; The textual auxiliary description is converted by a preset text token processor to obtain textual abstract features; By cross-embedding the processed abstract features with the textual abstract features, a hybrid abstract feature is obtained.

4. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that, The process of inputting the hybrid abstract features and the original question into the preset multimodal large model to obtain the textual response output by the preset multimodal large model includes: Determine the problem type of the original problem; The preset multimodal large model is invoked to determine the target pre-trained low-rank adaptive module based on the problem type; Based on the target, the pre-trained low-rank adaptation module outputs a textual answer according to the hybrid abstract features and the original question.

5. A surgical video processing device based on a multimodal large model, characterized in that, include: A determination module is used to determine the surgical video and the original questions associated with the surgical video; The splitting module is used to split the surgical video into multiple video segments with a fixed number of frames; The encoding module is used to encode each of the multiple video segments using a pre-trained video encoder to obtain abstract features; The conversion module is used to convert the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter, so as to obtain the processed abstract feature; An embedding module is used to cross-embed the processed abstract features with textual auxiliary descriptions to obtain hybrid abstract features; The output module is used to input the hybrid abstract features and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model; Before determining the surgical video and the original questions associated with the surgical video, the device is further configured to: A preset video encoder is trained to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align the video features of surgical instruments with the text features of surgical instruments. The step of training a preset video encoder to obtain a pre-trained video encoder includes: Obtain surgical video training samples, wherein the surgical video training samples include at least one surgical instrument mark and surgical instrument text features corresponding to the surgical instrument mark; The surgical video training samples are masked by a preset video encoder to obtain masked video training samples. The masked video training samples are encoded by the encoder of the preset video encoder to obtain encoded video training samples; The encoded video training samples are masked and reconstructed using the decoder of the preset video encoder to obtain the video features of surgical instruments. Based on a preset loss function between the video features and text features of the surgical instruments, the preset video encoder is updated to obtain a pre-trained video encoder.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the surgical video processing method based on a multimodal large model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the surgical video processing method based on a multimodal large model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the surgical video processing method based on a multimodal large model as described in any one of claims 1 to 4.

Images